How to Select the Right Load Model

Motivation

One of the decisions you must make before running a load test is selecting the right load model. A load model describes which basic characteristics are influenced to achieve a certain load and performance behavior.

Let’s clarify some terminology before discussing the models, as the definitions used in the following paragraphs are not unique across the industry:

Transactions: A transaction is the execution of exactly one test case or test scenario. To perform the scenario, the page flow is modeled in code. The test scenario is implemented as a test case, which executes a sequence of one or more actions.

Actions: An action can be defined as one irreducible step within a test case. An action interacts with the current page and—as a result—loads the next page. That page is associated with this action and becomes the current page for the next action in the test scenario. Generally, an action triggers one or more requests. This also applies to REST-style test cases.

Requests: This level is equivalent to the HTTP request level used in web browsers or any other application that relies on HTTP communication. You do not have to deal with requests directly, as they are automatically generated by the underlying HtmlUnit framework when performing actions on HTML elements.

Available Load Models

Let’s assume that before running the next marketing campaign, your company’s online shop must be tested to ensure it won’t break under the expected traffic. There are two different approaches to defining the load your system should be able to handle:

• You know how many concurrent users you expect, or
• You know how many visits or orders per hour you expect (e.g., during a campaign).

These requirements necessitate load models to test them correctly.

User Count Model

For the User Count Model, you define a specific number of concurrent users the system must handle. At any given time during the test, the target system must handle 2000 concurrent users—no more, no less. The number of transactions that can be achieved depends on the time the target system needs to respond.

With the user count model, you can ensure your application will be stable with the defined number of concurrent users.

Arrival Rate Model

With the Arrival Rate Model, your criterion is the number of transactions (or, in this case, visits and orders) per hour. The order scenario should be performed 500 times, equally distributed across one hour. As many concurrent users as necessary to fulfill the given arrival rate are used.

Response Time as Influencing Factor

When using the user count model, fewer transactions will simply be finished if the response time increases. In comparison, the arrival rate model will increase the number of concurrent users to ensure the planned number of orders is achieved.

This means the arrival rate model works on a feedback basis and reacts to response time changes during the test period. This way, the generated load is somewhat unpredictable, but this aligns more closely with real-world behavior.

However, another thing to keep in mind is the potentially aggressive behavior of a test using the arrival rate model: if the response time increases, more concurrent users are used. This probably means the response time will increase further due to heavier load, causing the system to use even more concurrent users (which is where recursion kicks in).

To avoid a complete breakdown in this scenario, define an upper limit for the number of concurrent users in the arrival model. This restricts the total load on the system to avoid a total overload resulting from the feedback loop. Of course, that does not reflect reality, but if users experience a certain pain point in reality, they might also hold back.

Summary

The arrival rate model is better suited to model real-world load, so it might be the right choice for most load tests. However, there are still some scenarios where the user count model could be a good choice. Here are some guidelines on which model best fits what purpose:

The user count model is well-suited for:

• A simple baseline test (single-user test) to assess the base performance of the system under almost no load.
• A real load or performance test to assess performance under a high but predictable (i.e., stable) load.
• A test that should be easily repeatable (i.e., its load factor should not be influenced by the system under test).

The arrival rate model is best used:

• If the load test should prove that a system can indeed handle a specific number of transactions per hour.

Since the primary purpose of load and performance tests is the simulation of a specific number of transactions per hour, the arrival rate load model is the best choice for most of your testing tasks.

So, get testing and try all models. Load tests often start with a fixed user rate; once this runs fine, you can move to the more challenging arrival rate model.