How to Select the Right Load Model

Motivation

One of the decisions you have to make before running a load test is the selection of the right load model. A load model describes what basic characteristics are influenced to reach a certain load and performance behavior.

Let’s just clarify some terminology before we talk about the models, because the definitions used in the following paragraph are not unique across the industry:

Transactions: A transaction is an execution of exactly one test case or test scenario. In order to perform the scenario, the page flow is modeled in code. The test scenario is implemented as a test case, which itself executes a sequence of one or more actions.

Actions: An action can be defined as one irreducible step within a test case. An action interacts with the current page and – as a result – loads the next page. That page is associated with this action and becomes the current page for the next action in the test scenario. Generally, an action triggers one or more requests. This also applies to REST style test cases.

Requests: This level is equivalent to the HTTP request level used in web browsers or in any other application that relies on HTTP communication. You do not have to deal with requests directly because they are automatically generated by the underlying HtmlUnit framework when performing actions on HTML elements.

Available Load Models

Let’s assume that, before running the next marketing campaign, your company’s online shop will have to be tested to make sure it won’t break under the expected traffic. There are two different approaches to define the load your system should be able to handle:

• you know how many concurrent users you are expecting, or
• you know how many visits or orders per hour you are expecting, e.g. during a campaign.

These requirements require load models to correctly test them.

User Count Model

For the User Count Model, you define a certain number of concurrent users the system will have to handle. At any given time during the test, the target system has to handle 2000 concurrent users, no more, no less. The number of transactions that can be achieved depends on the time the target system needs to respond.

With the user count model, you can make sure your application will be stable with the defined number of concurrent users.

Arrival Rate Model

With the Arrival Rate Model, your criteria is the number of transactions (or here: visits and orders) per hour. The order scenario should be performed 500 times, equally distributed across a period of one hour. As many concurrent users as necessary to fulfill the given arrival rate are used.

Response Time as Influencing Factor

When using the user count model simply less transactions will be finished if the response time increases. In comparison the arrival rate model will increase the number of concurrent users to make sure, the planned number of orders will be achieved.

This means that the arrival rate model works feedback based and will react to response time changes during the test period. This way, the generated load is somewhat unpredictable, but this is more along the lines of how the real world behaves.

But another thing to keep in mind is the potential aggressive behavior of a test using the arrival rate model: if the response time increases, more concurrent users are used, which probably means that the response time will increase even more due to heavier load (which will cause the system to use even more concurrent users, which is where the recursion kicks in).

To avoid a complete breakdown in this scenario, you should define an upper limit to the number of concurrent users used in the arrival model. This restricts the total load on the system, in order to avoid a total overload as a result of the feedback loop. That is not reflecting reality of course, but if the users feel a certain pain in reality, they might hold back as well.

Summary

The arrival rate model is better fit to model real world load, so it might be the right choice for most load tests. But there are still some scenarios in which the user count model could be a good choice, so here are some guidelines which model fits what purpose best:

The user count model is well-suited for:

• a simple base line test (single-user test) to assess the base performance of the system under almost no load,
• a real load or performance test to assess the performance under a high, but predictable aka stable load,
• a test that should be easily repeatable, i.e. its load factor should not be influenced by the system under test.

The arrival rate model is best used:

• if the load test should prove that a system is indeed able to handle a certain number of transactions per hour.

Since the primary purpose of load and performance tests is the simulation of a certain number of transactions per hour, the arrival rate load model is the best choice for most of your testing tasks.

So get testing and give all models a try. Load tests often start with a fixed user rate and once this runs fine, you move over to the more challenging arrival rate model.