4.7.x

XLT 4.7.3

Script Developer

Support Firefox 50 (#2809)

Script Developer has been made compatible with Firefox 50, which will be released on November 15, 2016. Script Developer runs on Firefox 38/ESR up to 50 now.

Selection bar disappears when collapsing script packages or deleting items (#2806, #2811)

Navigating through the script package tree on the left hand side is also possible with the cursor keys. However, when collapsing an expanded package subtree with (or double-clicking it with the mouse), the selection bar disappeared making it impossible to continue keyboard navigation.

Similarly, when deleting one or more items - scripts or script packages

  • by either pressing the keyboard shortcut Delete or by using the appropriate context-menu item, the selection also disappeared afterwards.

Both issues are fixed now.

Bullet lists not transformed from Markdown into HTML (#2794)

When generating HTML documentation for a script test suite with <xlt>/bin/create_scriptdoc.sh, bullet lists in Markdown-enriched descriptions were sometimes not transformed as expected. This happened if a bullet list was not explicitly separated from the preceding text by an empty line. Fixed.

Load Testing

User data for AWS instances not recognized (#2801)

If you use our public Amazon machine image for running XLT 4.7.2, you may have noticed that the host name mappings you can pass to the machine instances as user data might sometimes not become effective. Fixed.

Scrolling issues on report pages (#2803, #2812)

On load test report pages with chart tab groups, double-clicking a tab of a tab group causes all the other tab groups to show the same chart tab. When doing so with Chrome, the page performed some unexpected vertical scrolling. When viewing such a page with Firefox and clicking a link in the data table to jump to the corresponding chart group, the page might scroll horizontally. Both scrolling issues have been fixed.

Framework

Store incomplete responses (#2808)

XLT stores the content of any response to a file for later display in the result browser. However, sometimes the file was empty. This happened if the response was delivered in compressed form and the server ended the content transmission prematurely. In such a case, XLT now reads and stores as much of the response as it is able to.

I/O exceptions swallowed when reading response content (#2810)

Once the response headers have all been received for a request, the download of the response content starts. If an exception occurred during content download (read timeout, etc.), that exception was only logged, but not propagated. Hence, XLT did not mark the request as failed. Fixed.

XLT 4.7.2

Script Developer

Support Firefox 49 (#2780)

Script Developer has been made compatible with Firefox 49, which was released on September 20, 2016. Script Developer runs on Firefox 38/ESR up to 49 now.

Script content disappears when closing other tabs or deleting scripts on disk (#2767, #2783)

Multiple editor tabs can be closed at once by clicking one of the two context-menu items ‘Close All Tabs’ or ‘Closing Other Tabs’. In the latter case, it might happen that the content of the left-over script disappeared from the UI. Once the left-over tab was clicked, the content re-appeared.

The same effect could be observed when deleting one or more scripts, which are currently opened in tabs, outside of Script Developer. When reloading the entire test suite, any other still opened script might be shown with no content.

Both issues are fixed now. Please note that the actual content of the scripts was not affected, but only its representation in the UI.

Not all selected script items are extracted to a new module (#2774)

If you have selected several commands and module calls in a script and you try to extract them into an own module, the newly created module might not contain all of the selected script items, but only the first few items. This happened if a selected module call was expanded and an item inside the module was selected too. In this case, the extraction process stopped after extracting this module. Fixed now.

Formatting of JavaScript code gets lost (#2784)

Since XLT 4.7.0, Script Developer supports multi-line JavaScript expressions as target value for *Eval commands. Unfortunately, all line formatting got lost when the enclosing script was reloaded from disk. Hence, when opening the ‘Edit Command’ dialog for such an *Eval command, the entered expression was reduced to a single line as all new-line and extra whitespace characters were removed from it. This is fixed now and (re)loading a script maintains all multi-line expressions as they were entered.

Data sets file not copied when exporting a script to Java (#2785)

As of XLT 4.7.0, Script Developer supports data driven testing and managing the underlying data sets files. However, when exporting a test script to Java, its data sets file was not copied next to the generated class, unlike all the other data files. Fixed.

Framework

Automatic screenshots for WebDriver-based test cases (#1988)

Up to now, interpreted XML script test cases were the only kind of test cases for which screenshots were taken automatically after each action. Now this works also for script test cases that have been exported to Java. Likewise, it will work for test cases that inherit from AbstractWebDriverTestCase, but make sure that you have structured your test cases into actions by calling Session.getCurrent().startAction("...") as needed. In any case, open the result browser for your test case to view the series of screenshots that have been taken during test execution.

If the used Web driver does not support taking screenshots, the resulting screenshots will be empty. XltDriver is a special case. It does not support screenshots either, but the XLT framework will store the visited pages as viewable HTML pages. However, this is not a new feature.

Page for the last action not available in result browser (#2786)

When running Java test cases, which have been exported from script test cases, with XltDriver as the underlying Web driver, the page for the last action was missing in the result browser. Fixed.

Load test data as resource (#2776)

When a script test case is exported to Java, its data file will be copied to the same directory as the generated Java class. Furthermore, all script packages in the package tree hierarchy will be requested to store their test data to the corresponding Java package directory. All of these test data files will then be read in by XLT when executing the test as plain JUnit test or as load test. This said, XLT expected all the test data to reside in local files. But this is not always the case. For example, a Java class might also be loaded from a Java archive (JAR). All attempts to locate and load the data files will obviously fail then.

Starting with XLT 4.7.2, all test data files (except of global_testdata.properties which is still required to reside in the root directory of your test suite) are treated as a resource instead of a local file which allows us to load them from JARs too.

Load Testing

Add AWS region to agent controller names (#2787)

When using ec2_admin to print the corresponding master controller configuration for a set of AWS machines, the generated agent controller names will now also contain the region in which the respective machine is running:

com.xceptance.xlt.mastercontroller.agentcontrollers.ac001_ap-southeast-2.url = https://52.62.14.48:8500  
com.xceptance.xlt.mastercontroller.agentcontrollers.ac002_us-east-1.url = https://54.82.197.127:8500

If your load test is driven from multiple locations in the world, it is much easier now to tell which agent controller runs in which AWS region.

Show an AMI’s name tag instead of its description (#2758)

When starting new AWS machine instances via ec2_admin, you have to choose the machine image (AMI) to use from the list of available AMIs. In this list, ec2_admin used to display the AMI’s description, but displays now the AMI’s name tag and shows its description only if no name tag is present. The advantage of preferring the name tag is that the name tag can easily be modified in case of mistakes (for instance, via the AWS Web Console), whereas the description cannot be changed at all.

Unwanted log lines printed to the console (#2764)

Since XLT 4.7.1, several log messages were printed to the console when starting the master controller in embedded mode. This happened due to a missing 3rd-party library slf4j-api that was excluded from shipment by accident. Fixed.

Improved error message for request merge rule indices with leading zeros (#2782)

Request merge rules are processed in ascending order specified by their index. In case this index contains one or more leading zero digits, report generation will be aborted as leading zero digits are not allowed in request merge rule indices. The error message shown to the user in this situation is much more helpful now.

XLT Jenkins Plug-In

State not passed to subsequent build actions (#2744)

In order to fix a vulnerability in Jenkins, plug-ins are not allowed to add arbitrary parameters to a build anymore, except those that have been explicitly defined in a job’s configuration. Since our XLT plug-in for Jenkins defines several environment variables passing state to subsequent build actions, it was affected by this fix and didn’t work correctly with Jenkins (v1.651.2/v2.3 or newer). As a temporary workaround, you had to add -Dhudson.model.ParametersAction.keepUndefinedParameters=true on the command line of Jenkins which effectively disabled the security fix.

The current version 1.2.1 of the XLT plug-in re-enables the feature of passing state to subsequent build actions as environment variables - without having to specify the command line property. If you have used this workaround to get our plug-in working, don’t forget to remove this setting now.

XLT 4.7.1

Script Developer

Cannot apply changes to test suite details (#2755)

Unfortunately, pressing the OK button to accept the changes made in the Test Suite Details dialog had not effect. Hence, neither the test suite’s name, nor its description could be edited. This is fixed now.

Refactoring a module does not update calling scripts (#2759)

Although refactoring a module - renaming it or moving it to another package - did work as expected, one essential part was missing, namely the update of the calling scripts. As a consequence, all of them were in an erroneous state as name or package of the called module was not adjusted accordingly. This has been fixed and refactoring a module does not lead to broken scripts anymore.

Framework

Authorization header is unnecessarily sent (#2751)

In case the same WebClient is used to send requests to several hosts and one of them requires authentication, the Authorization request header was sent to all of the other hosts too, even if the user doesn’t need to authenticate with them. What makes this even worse: this happened also if different hosts require different credentials, so requests to the first host were successful, but requests to the other hosts used the wrong Authorization header, causing an authentication failure.

This issue is caused by a bug in HtmlUnit, reported in late 2015. Fortunately, we were able to fix this issue and provided a patch that will be part of the upcoming release of HtmlUnit (v2.24).

Request header ‘Authorization’ missing in result browser (#2752)

In case the site under test requires the user to authenticate, the Authorization header that is sent with every request to this site was not shown in XLT’s result browser although it has been written to wire. Fixed.

XLT 4.7.0

Release Overview

XLT 4.7.0 comes with many new and improved functionalities. This section summarizes the highlights of this release.

Script Developer ships with two new major features. The first is the support of “post steps” at the end of your script test cases. Post steps will always be executed no matter if the test case (or a previous post step) failed during execution. A post step is the perfect candidate to implement cleanup tasks to be run after the actual test case. The second feature is the full support of data sets for data-driven tests. Now data sets can be maintained and used right in Script Developer. This enables you to run a test case multiple times with different test data. Script Developer has also been reworked to improve usability. Again, it supports the latest Firefox.

We created a new script command. The echo command is especially useful when debugging your scripts. As usual, this command is supported not only in Script Developer, but also in the XLT framework.

Load testing is much more fun now. If you are an XLT power user and use variable load profiles a lot, you will be happy to hear that XLT supports load factor functions now. Simply define a single load factor function to vary the load for all scenarios together.

The load test report has been improved as well. The data table and the charts in the Custom Values section are equipped with many more details now. The Errors section features separate charts for each error type, making it easier for you to identify when certain errors occurred in the course of a load test. The Network Timing tab for requests now lists the time taken to look up an IP address for a host name to help you identify issues with your DNS servers. If you need to focus on one or a few test scenarios only, you can create a load test report for those scenarios only.

The XLT framework also comes with some improvements. When you use Chrome via XltChromeDriver, you will now get a much more detailed result browser with almost the same request and response details that you already know from HtmlUnit / XltDriver. XLT can be enabled to store the HTTP method as well as any POST data to the timers.csv files. Now the data in these files is sufficient to even replay the recorded requests. If you prefer coding script test cases in Java or need to export them to Java, you will enjoy more detailed logs and more descriptive exception messages. Last but not least, all core libraries have been updated, especially HtmlUnit and WebDriver/Selenium. This ensures a much better browser emulation with significant performance improvements and brings support for the latest native browsers.

Sounds interesting? Continue reading and get all the details.

Script Developer

Post-Steps

In case your test suite contains tests that apply persistent changes to the application under test, you might consider to define a “Post-Steps” section. This section of the test is somewhat special as each command you define there will be executed in each and every case. This allows you to roll back any persistent change made by the test. More information about this feature can be found in the legacy documenation.

Script test case with post-steps

Data Sets

So far, data-driven tests had to be executed as JUnit tests using Eclipse, ANT, Maven or similar. Furthermore, all the data files had to be maintained by an external editor. Starting with XLT 4.7.0, Script Developer is capable of data-driven testing which means that data sets defined by XML files can now be maintained and executed in Script Developer too.

Manage data sets in Script Developer

Data-driven tests in Script Developer

For more information, see the legacy documentation.

Usability Improvements

  • The Save All functionality is available not only as keyboard shortcut Ctrl+Shift+S, but also as separate toolbar item.
  • There is a new Auto-Save option in the Settings dialog. If checked, any modified file will be saved automatically when running a test case.
  • A test suite can now be removed much more intuitively via the folder menu next to the test suite dropdown. It’s also possible to remove the very last test suite.
  • If you have a huge test suite, save valuable time by refreshing only those scripts you’re interested in: select them in the explorer view and press F5.
  • To ease editing longer JavaScript expressions, the Target input field for eval commands has been turned into a multi-line text field.
  • Since tests take usually more than one second to complete, all test durations shown in the batch test report use seconds instead of milliseconds now.
  • New commands can be inserted at the end of a script also by pressing Enter or double-clicking on the always-empty last line.

Supported Firefox Versions

Script Developer has been made compatible with the latest available Firefox version, while outdated versions are not supported any longer. Thus, Script Developer runs on Firefox 38/ESR up to 48 now.

Scripting

New Commands

Script Developer and XLT support a new command: echo. Use this command to write a custom message to the log for debugging purposes or simply for information. Specify the message to be logged as the value of this command. As usual, any placeholder (${…} or @{…}) present in the log message will be resolved right before executing this command.

Load Testing

Variable Load Factors

Back in XLT 4.4.0, we introduced a new load profile setting, the load factor. Using this factor, it is very convenient to scale the applied load up or down - for a single test scenario or for all scenarios together. Now the load factor is not bound to be a double value any longer but can also be a variable load function, which you may already know from users or arrival rates. This means that the load factor may change in the course of the load test, and so will the applied load.

Being able to define the load factor as a function is a huge advantage if you want to vary the applied load for all scenarios at once, at the same time and to the same extend. This is true in most cases. You just need to define one function, the one for the load factor, and this single function is applied to the configured user counts and arrival rates of all scenarios together. No need to define separate load functions for each scenario any longer. Want to adapt the load profile? Just edit one function only.

An example configuration would look like this:

com.xceptance.xlt.loadtests.default.loadFactor = 15m/1, 60m/1, 65m/2, 120m/1.5  
com.xceptance.xlt.loadtests.TBrowse.arrivalRate = 2000  
com.xceptance.xlt.loadtests.TOrder.arrivalRate = 100  
com.xceptance.xlt.loadtests.TSearch.arrivalRate = 2000

In this example, the load is scaled such that it is linearly ramped up to 100% within the first 15 minutes, is then kept stable at 100% for another 45 minutes, before it will be doubled within the next 5 minutes, and finally be lowered to 150% over the next 55 minutes. The given arrival rate values denote the load at 100% for each scenario and will be adjusted according to the load factor.

No Load Test in Case of Configuration Errors

Some of the load profile parameters are mutually exclusive, for instance the settings for arrival rate and iterations. In such a case, the master controller logged a warning (to the log file) and silently continued with one of the parameters. This way, the resulting load might not always have been what the user wanted. Since specifying contradicting parameters is actually a configuration error, the master controller will now print an appropriate error message to the console and won’t start the load test in such a case.

AWS Data Center in Mumbai

Recently, Amazon opened a new data center in Mumbai, India (ap-south-1 ). This new data center is now fully supported in ec2_admin, our tool to manage your AWS EC2 machine instances.

Load Test Report

Report for only a Subset of the Scenarios

Load test reports for larger load tests can become quite lengthy. Sometimes you may need to focus on one or a few test scenarios only to better illustrate a problem or to remove unwanted details. To this end, you can now specify the scenarios to be included (or excluded) when generating the load test report. See below:

create_report.sh ../results/20160405-174602 -i “TBrowse,TOrder”  
create_report.sh ../results/20160405-174602 -e “TBrowse,T.\*?Order”

In the first example, we simply list the scenarios to be included (-i). Consequently, only TBrowse and TOrder will be part of the report. But the syntax allows for regular expressions as well. In the second example, we combine the list of expressions with -e to exclude the matching scenarios, so any scenario except TBrowse, TOrder, and also TGuestOrder will be part of the report.

More Details for Custom Values

Custom values are used to record arbitrary values during a load test, either directly from your test case or asynchronously from dedicated custom samplers. However, the level of detail for custom values in the load test report was not comparable with the one for runtime values. Now the custom values are not only presented on their own main page, but there is also much more information on this page:

  • The data table shows the count per second/hour/day and the configured percentiles.
  • The Overview charts are drawn not only with lines, but also with dots to visualize the clustering of values. Of course, they also feature the small histogram to show the value distribution.
  • There is a new chart visualizing the average values.
  • The entries in the data table and the chart list can be filtered.

Custom Values

Separate Error Charts

When analyzing and communicating the results of a load test, the Errors section is always of special interest. It does not only list the different types of errors, but also how often they occurred. However, this information is sometimes not sufficient, especially if you need to know when a certain error has occurred.

For example, there might have been a couple of technical errors (Connection Refused errors, for instance) in your report and you want to know if they occurred sporadically over the course of a load test or if they are clustered at certain times. The general Errors chart might not show this information, especially if there were many other errors. You simply could not tell one from the other. That’s why each type of error can now have its own chart. No more digging through the timers.csv files to get this information!

Request Errors

Transaction Errors

The report generator can create different error charts:

  • Request error charts - A separate chart for each encountered HTTP server error status code (>= 500) and another one for code 0, i.e. there was no response at all.
  • Transaction error overview charts - A separate chart per transaction error type, over all transactions.
  • Transaction error detail charts - A separate chart per transaction error type, grouped by transaction/action name.

Note that the error message text defines the “type of error”. All errors with the same message are grouped together. If you have dynamic parts in your error messages, e.g. a SKU, you may end up with thousands of different error types. Consequently, the report generator would have to create as many charts. To avoid this situation, the report generator will generate charts for the topmost 50 errors only by default.

However, you can tailor this limit selectively in reportgenerator.properties. See the following example:

com.xceptance.xlt.reportgenerator.errors.requestErrorOverviewChartsLimit = 0  
com.xceptance.xlt.reportgenerator.errors.transactionErrorOverviewChartsLimit = –1  
com.xceptance.xlt.reportgenerator.errors.transactionErrorDetailChartsLimit = 10

With these settings the report generator will create as many transaction error overview charts as needed, but only 10 transaction error detail charts and no request error charts at all.

Separate Pages for Errors and Events

Both the Errors and the Events section in the load test report can become quite lengthy, depending on the number of error types and events logged during a load test. For the sake of clarity and a shorter page loading time, the event information has been moved to its own main page.

DNS Timings

For each request type there is not only a total request runtime, but also a broken-down view (on the Network Timings tab). This gives a deeper insight as to whether the time was spent connecting to the server, waiting for the server to respond, or receiving the response. However, one important piece has been missing in the picture: the time it takes to look up the IP address for a host name from a DNS server. These servers work fast and reliably most of the time, but sometimes even DNS servers may have the hiccups.

In order to guide you in identifying such issues, the time spent in IP address lookups is now available as a separate value on the Network Timings tab. This indicates whether a longer request runtime was caused by the system under test or the DNS server.

DNS timings

Note that the DNS time value (as any other network timing value) is only available when using the browser emulation via HtmlUnit / XltDriver or when using a real browser for which an XLT timer recorder plug-in is available (currently Chrome and Firefox, via XltChromeDriver and XltFirefoxDriver, respectively).

Sticky Table Header

Sometimes your data tables can become really long. This is especially true for the table on the Requests page. When scrolling such a long table, it is hard to remember which value is shown in what column. The table header therefore is now sticky when scrolling.

Runtime Segmentation is Optional

Ever since XLT 4.6.0, you can have both configurable percentiles and a configurable runtime segmentation on the Requests page. Either of them allows to assess the distribution of runtime values, but from slightly different perspectives. You may prefer to have only one of them or even none in your report.

Runtime percentiles have always been completely optional, but the runtime segmentation was required. Now it is optional as well. To switch it off, either comment out the respective setting in file reportgenerator.properties or leave it empty:

#com.xceptance.xlt.reportgenerator.runtimeIntervalBoundaries = 1000, 3000, 5000  
com.xceptance.xlt.reportgenerator.runtimeIntervalBoundaries =

Framework

Result Browser with Request/Response Details also for Chrome

When using plain HtmlUnit or XltDriver for your test cases, you surely have come across our result browser. It lists all the visited pages as well as all important request and response details, such as headers and cookies. The result browser is also available if your test cases use real browsers via the WebDriver API. However, it shows the visited pages only.

In this release we have improved our plug-in for Chrome to provide not only request timing data, but also other request and response details. The response body is the only piece not available yet. All this data is shown in the result browser the same way as you know it from HtmlUnit or XltDriver. There is nothing special to do to get this data, just make sure you use XltChromeDriver instead of ChromeDriver when running tests with Chrome.

More Detailed Logs and Exception Messages for Exported Test Cases

When running XML script test cases, XLT logs any executed command and reports when a module was entered or left. Now this kind of logging is available for exported test cases as well.

Furthermore, the message of exceptions thrown from exported test cases will now always carry the current command as context information and will provide more detailed failure information. For instance, instead of a simple Timed out waiting for condition message for a failed waitFor... command, you will now get:

com.xceptance.xlt.engine.TimeoutException: Command failed:
waitForText target=“id=title” value=“Hello” -> Timed out while waiting
for condition: Element not found

or

com.xceptance.xlt.engine.TimeoutException: Command failed:
waitForText target=“id=title” value=“Hello” -> Timed out while waiting
for condition: Element text ‘Hullo’ does not match

Record HTTP Method and POST Data

XLT records a lot of data for each request it makes and stores that data to a timers.csv file. Now XLT can be enabled to store the HTTP request method to this file and, for POST requests, the request content type as well as any form data. This new information is not generally needed for load testing and the test report, but comes in handy if you need to reconstruct the requests that have been made. In other words, the data in timers.csv is sufficient now to replay the recorded requests.

For example, do not run a test scenario repeatedly as a load test, but only once as a functional test. The resulting timers.csv file can then easily be transformed to an input file for our NoCoding test suite.

But replaying the recorded requests is a very special use case and POST data can become significantly large. Therefore, recording method, POST data and its content type is disabled by default. To enable this feature, set the following property in your project configuration:

com.xceptance.xlt.results.data.request.collectAdditionalRequestInfo = true

Path to Browser Binary Configurable

The WebDriver factory built into XLT allows to configure the path to the driver server binary in the XLT properties. What has been missing until now was a way to configure the path to the actual browser binary. This is useful in case multiple browser versions are installed and the default one is not the one to use.

xlt.webDriver.chrome.pathToDriverServer = /path/to/chromedriver  
xlt.webDriver.chrome.pathToBrowser = /path/to/chrome

Note that some WebDrivers do not support setting the browser binary, e.g. InternetExplorerDriver and EdgeDriver.

Browser History Limit

If you use plain HtmlUnit or XltDriver, be aware that the implementation of the browser history in HtmlUnit has changed. Now the history also keeps strong references to the visited pages, up to the configured number of history entries. This can cause memory issues. Therefore, XLT limits the maximum number of entries in the history to 1 by default. If your test scenarios don’t make use of the browser history at all, you can turn it off completely with the following setting in your project configuration:

com.xceptance.xlt.browser.history.size = 0

Likewise, if your test scenarios require more than one entry in the history, increase the setting as needed.

XLT Jenkins Plug-In

In order to fix a vulnerability in Jenkins, plug-ins are not allowed to add arbitrary parameters to a build anymore, except those that have been explicitly defined in a job’s configuration. Since our XLT plug-in for Jenkins defines several environment variables passing state to subsequent build actions, it is affected by this fix and won’t work if you use Jenkins v1.651.2 and v2.3 or newer.

We are heavily working on a fix for this. In the meantime, Jenkins has to be started with an additional command line option to work around this issue:

java -Dhudson.model.ParametersAction.keepUndefinedParameters=true -jar jenkins.war

XLT Proxy Recorder

Recording of HTTPS Requests

In order to be able to record HTTPS requests using XLT Proxy Recorder, the recorder’s server certificate must be one that your browser will accept. To make this happen, you need to add the recorder’s root CA certificate to your browser’s trust store. Unfortunately, there is no easier way. However, since it is a root CA certificate, this may raise security concerns. We are aware of that. Instead of shipping a predefined root CA certificate that would be shared among all XLT users, we have equipped XLT Proxy Recorder with an easy-to-use tool to create a fresh root CA certificate that is all your own and completely under your control.

In the Settings dialog, click Create Certificate and specify the destination for the certificate file, the corresponding private key file as well as the password and for how long the certificate will be valid:

How to create a certificate

When done, XLT Proxy Recorder will automatically be reconfigured to use the certificate you just created.

Now that you have created your very own root CA certificate, you need to import it into your browser’s trust store. The details of this procedure vary from browser to browser. Please consult the Internet to get specific instructions for your browser on your OS.

Last modified May 25, 2022