Choosing a Load Test Model

When running load tests, success depends on the concept behind how you set up and execute your tests. While we’ve previously touched on gathering metrics, today we’ll dive into the different load models you can use to structure your runs.

Choosing the right load model is a fundamental step. While it might sound technical, the concept is straightforward: a load model describes the primary characteristics you influence to reach a specific performance behavior or load level.

A quick note on terminology before we dig in: definitions in this field aren’t always standardized. Different vendors may use these terms slightly differently, so here is how we define them in the context of XLT:

• Transactions: “A transaction is an execution of exactly one test case or test scenario. In order to perform the scenario, the page flow is modeled in code. The test scenario is implemented as a test case, which itself executes a sequence of one or more actions.”

• Action: “An action can be defined as one irreducible step within a test case. So an action interacts with the current page and – as a result – loads the next page. That page is associated with this action and becomes the current page for the next action in the test scenario. Generally, an action triggers one or more requests.”

• Requests: “This level is equivalent to the HTTP request level used in web browsers or in any other application that relies on HTTP communication. You do not have to deal with requests directly because they are automatically generated by the underlying HtmlUnit framework when performing actions on HTML elements.”

Example Scenario

Let’s look at a practical example. Before a major marketing campaign, your company’s online shop needs thorough testing to ensure it can handle the anticipated traffic surge. Two of your stakeholders are discussing the requirements:

• Head of IT: “The shop will have to handle 2,000 concurrent users.”
• Head of Marketing: “With our campaign, the shop will have to handle 500 orders per hour.”

These two perspectives lead to different load models, depending on your primary goal:

• User Count Model: From a technical perspective, this means defining a fixed number of concurrent users. At any given time during the test, the system handles exactly 2,000 users. The total number of transactions achieved then depends entirely on how fast the system responds.
• Arrival Rate Model: The business perspective often focuses on outcome, such as the number of orders per hour. In this model, the scenario is performed 500 times, evenly distributed over an hour. The test will automatically scale the number of concurrent users to meet this target arrival rate.

Comparing the Models

So, what happens if the system’s response time increases during the test?

In the User Count Model, the system will simply process fewer transactions. In contrast, the Arrival Rate Model will increase the number of concurrent users to ensure the target number of orders is still met, even if things slow down.

See the difference? The Arrival Rate Model is feedback-based and reacts to performance changes in real-time. This makes the generated load dynamic and unpredictable—just like the real world.

Think of it like a supermarket: people continue to enter the store even if there are long lines at the checkouts. That’s exactly what happens with your online shop under the Arrival Rate Model—new users keep arriving, unaware of any slowdown experienced by those already there.

While the Arrival Rate Model is often the better choice for realism, technical teams sometimes worry about its “aggressive” behavior. Imagine this scenario:

• Response time increases
• More concurrent users are used to keep up
• Response time increases even further due to the increased load
• Even more concurrent users are added
• A feedback loop begins

Illustration: The Feedback Loop

There is a solution: to prevent a complete system breakdown, you can set an upper limit on the number of concurrent users in the Arrival Rate Model. This restricts the total load and prevents the feedback loop from spiraling out of control. While this slightly sacrifices perfect realism, it mirrors how real users might eventually stop trying if the experience becomes too painful.

Guidelines for Choosing a Model

So, when should you choose one over the other? While the Arrival Rate Model is powerful, the User Count Model still has its place.

The User Count Model is well-suited for:

• Baseline tests: A single-user or low-concurrency test to assess the base performance of the system under almost no load.
• Stability tests: Assessing performance under a high, predictable, and stable load.
• Repeatability: Tests that need to be easily repeatable where the load factor is not influenced by the system’s own performance.

Conclusion

The Arrival Rate Model is ideal for proving that a system can handle a specific number of transactions per hour. Since this is often the core goal of performance testing, it is usually the best choice for most scenarios.

Start testing and experiment with both models. A common approach is to begin with a fixed user rate to ensure stability, then transition to the more challenging and realistic Arrival Rate Model.