Performance Testing with Apache jMeter

Designing and implementing distributed systems, both customer-faced or just datacrunching farms, one is soon required to determine performance impacts and possible bottlenecks.

In this specific case, I wanted to gain information regarding the limits and scalability aspects of a customer-facing web application which also manages a high number of connected devices.

Why I chose Apache jMeter

The performance testing case I was working on made me opt for jMeter in the end, for the following reasons:

• Developed in Java, supporting plugins in Java or Beanshell. It is unlikely to have a metering case not which cannot be met with Java. In this case, Java became a killer feature, as most modules were implemented in Java, so it was possible to integrate jMeter into the given scenario without writing gluecode.
• Distributed by design. It is unlikely for a single machine to stress the system under test (SUT) enough to gain any useable information. Testing and stressing a distributed system requires controlled distributed load generators.
• Easy to get load generators on demand. jMeter is used by many engineers, there are a lot of services that accept jMeter recipies and play them on their machines for cash.
• jMeter is able to take jUnit tests and their performance metrics into account, too. This makes it possible to share tests between jMeter and the main testbench.
• jMeter brings a sophisticated GUI for testplan generation and debugging and also supports headless operation.
• Very flexible to configure.
• It is easy to package a jMeter node inside a docker image, so it can also run on a cloud computing provider which allows the execution of docker.
• Many built-in graph and reporting components, data export to 3rd party analysis tools is available.
• Open-Source, Apache Project
• • Finally: A wise man once said to me: “Long live the standards!”. jMeter can be considered as a defacto-standard swiss army knife for performance testing.

Tools and Terminae

jMeter uses a set of functional units for performing tests. After learning the vocabulary, the system is quite straightforward. The table below gives an overview on the jMeter modules and their purpose.

Designing a Test Plan

jMeter manages its test plan in a tree structure, which favours the XML data format used on the filesystem. So, the whole testplan meets a structure of ordered nodes with 0<n<inf children. For example, a parallel execution of a certain set of nodes would be represented by a parent node with the functionality of a thread controller, same applies on loop controllers or conditional controllers.

As an example, a login on a standard user/password form would be represented in jMeter as follows:

• ConfigurationManager.CSV> get Mock users from CSV file, read into variables
• CookieManager
  • Sampler.Http> Retrieve Login page, fail on HTTP error
    • Assert.XML> Check if site was delivered successfully
    • Postprocessor.XML> Extract CSRF token and save to variable
  • Sampler.Http> Post to login form with POSTdata composed from previous requests
    • Assert.Headers> Check that Session ID is present

Analysis and Reporting

After running the testplan and listening for the metrics delivered by the samplers, jMeter compiles a set of prebuilt reports which gather a lot of information, in most cases every information required to derive the next actions. For instance, it is possible to graph the respone times, the error ratio and the response times in relation to the quantity of parallel accesses. It is also possible to export the data into csv/xml files or use the generated reportfiles for further analysis. An interesting approach is to pass the data into R and use R’s graphing and reporting tools for further analysis.

Automation

Even though jMeter brings a really impressive GUI, it can be fully operated from the commandline. So, it is no problem to script it and, for example, integrate it into a CI/CD pipeline and let a build fail if it does not meet the performance expectations.

Distribution

In a distributed jMeter installation, workers are called “servers” and masters “clients”. Both are connected via old-fashioned Java RMI, so, after setting up an appropiate communication foundation between servers and client(s), triggering a load/performance testing job on the master suffices to start the slaves and collect their metrics.

Test plan creation

The jMeter files (.jmx) are pure XML, so it is theoretically possible to write them manually or, more probably, generate programatically. In most cases, one would use the GUI to click a test and customize it with config files, environment variables or own tiny scripting, depending on the system under test.

Plugin development

If jMeter does not deliver a functionality out of the box, it is possible to add the functionality by scripting or plugins. This means, it is possible to divide any implementation of a Sampler, Pre/Postprocessor, Assertion or Listener into three classes:

Class A: Available out of the box Class B: Possible by jMeter/Beanshell scripting Class C: Only possible by developing an own plugin

Developing a Java Sampler

A test-case classified as C needs to be implemented as a plugin. Basically, every aspect of jMeter can be delegated to a Java plugin, so it would also be possible to use a Java class to implement a custom assertion. Nevertheless, I think that the most common case is implementing a custom sampler to wrap a test around a functionality which either is not available through a public API or has asynchronity/concurrency requirements jMeter itself cannot meet.

An easy way of implementing a custom sampler is fetching the dependencies via maven and providing an implementation to the jMeter Sampler API.

A very minimal Maven POM just includes a dependency expression to the ApacheJMeter_java artifact, in a real use-case one might want to add maven-assembly to create a fat bundle including all further dependencies, so a downloadable package can be built on a buildserver.

 1<?xml version="1.0" encoding="UTF-8"?>
 2<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
 3  <modelVersion>4.0.0</modelVersion>
 4  <groupId>...</groupId>
 5  <artifactId>...</artifactId>
 6  <version>1.0-SNAPSHOT</version>
 7  <packaging>jar</packaging>
 8  <properties>
 9    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
10    <maven.compiler.source>1.8</maven.compiler.source>
11    <maven.compiler.target>1.8</maven.compiler.target>
12  </properties>
13
14  <dependencies>
15    <dependency>
16      <groupId>org.apache.jmeter</groupId>
17      <artifactId>ApacheJMeter_java</artifactId>
18      <version>2.7</version>
19      <type>jar</type>
20    </dependency>  
21  </dependencies>
22</project>

The JavaRequest sampler main class needs to extend the AbstractJavaSamplerClient class and provide a tree (1 <= n <= inf) of SampleResults:

 1import org.apache.jmeter.config.Argument;
 2import org.apache.jmeter.config.Arguments;
 3import org.apache.jmeter.protocol.java.sampler.AbstractJavaSamplerClient;
 4import org.apache.jmeter.protocol.java.sampler.JavaSamplerContext;
 5import org.apache.jmeter.samplers.SampleResult;
 6//...
 7
 8public class MyCustomRequestSampler extends AbstractJavaSamplerClient implements Serializable {
 9  private static final Logger LOG = Logger.getLogger(MyCustomRequestSampler.class.getName());
10
11  @Override
12  public Arguments getDefaultParameters() {
13    Arguments a = new Arguments();
14
15    a.setArguments(
16            Arrays.stream(JMeterCallParameters.values())
17                    .map(item -> new Argument(item.name(), ""))
18                    .collect(Collectors.toList())
19    );
20    return a;
21  }
22
23  /**
24   * Our actual entrypoint   
25   * (you want to do this to be able to unittest the sampler!)
26   */
27  public SampleResult runTest(SamplerConfiguration config) {
28    SampleResult result = new SampleResult();
29
30    result.sampleStart();
31    try (...) {          
32      result.addSubResult(anyAdditionalResult);    
33      result.setResponseCodeOK();
34      result.setResponseMessageOK();
35      result.sampleEnd();
36    } catch (Exception e) {
37      LOG.log(Level.SEVERE, "exception={0} stackTrace={1}", new Object[]{e, e.getStackTrace().toString()});      
38      result.setSuccessful(false);
39      result.setResponseData(e.getLocalizedMessage(), "UTF-8");
40    }
41
42    return result;
43  }
44
45  /**
46   * Entry point for jMeter Request
47   *
48   * Calls our internal function with the configuration gathered from jMeter
49   * Context and returns its SamplerResult
50   *
51   * @param jsc -  from jMeter
52   * @return Sample Result for jMeter
53   */
54  @Override
55  public SampleResult runTest(JavaSamplerContext jsc) {
56    SamplerConfiguration config = new MyCfgBuilder(jsc).build();
57    return runTest(config);
58  }
59}

After deploying the build artifact to $JMETER/lib/ext and restarting jMeter, it is available and can be integrated into a testplan using the GUI.

Dockerizing jMeter

An easy way to deploy a distributed jMeter installation is providing a docker set which consists of a master and n slaves. In this setup, it is advisable to create a base image and derive both master and server. If the setup is supposed to run in distributed environments which do not neccessarily provide a registry, it is advisable to use an own lightweight mechanism, such as a list in Redis which is populated by the slaves as soon as they get invoked.

The base image:

 1FROM m9d/debian-withSsh:jessieLatest
 2MAINTAINER manfred dreese <>
 3
 4RUN apt-get update
 5
 6# Install basics
 7RUN apt-get install -y wget unzip tar software-properties-common
 8
 9# Install Java
10RUN add-apt-repository ppa:webupd8team/java && \
11    apt-get update &&\
12    echo oracle-java8-installer shared/accepted-oracle-license-v1-1 select true | debconf-set-selections && \
13    apt-get install -y oracle-java8-installer
14
15# Install Download jMeter
16ENV JMETER_ROOT /opt/jmeter/
17ENV JMETER_BIN /opt/jmeter/bin/jmeter
18RUN wget -O /tmp/jmeter.tgz \
19  https://liam.aah.m9d.de/demo/binary-assets/third-party/apache-jmeter/3.1/apache-jmeter-3.1.tgz
20
21RUN mkdir $JMETER_ROOT && \
22  tar --directory /opt/jmeter --strip 1 -xzvf /tmp/jmeter.tgz
23
24# Pull jMeter Plugin
25RUN wget -O /tmp/InfraredPatternReactionPlugin.tgz \
26    https://jenkins.aah.m9d.de/InfraredPatternReactionPlugin-1.0-RELEASE-bundle.tar.gz && \
27  tar xzvf /tmp/InfraredPatternReactionPlugin.tgz -C /opt/jmeter/lib/ext

The master:

 1FROM m9d/jmeter-base
 2MAINTAINER manfred dreese <>
 3
 4# Install nginx (for retrieving reports)
 5RUN apt-get -y install nginx  && \
 6  rm /var/www/html/index.nginx-debian.html
 7COPY files/etc/nginx/sites-available/default /etc/nginx/sites-available/default
 8
 9# Install redis (for client registration)
10RUN apt-get -y install redis-server
11COPY files/etc/redis/redis.conf /etc/redis/redis.conf
12
13# Startscript
14COPY files/jmeter/run-testplan.sh /jmeter/run-testplan.sh
15COPY files/tmp/start.sh /tmp/start.sh
16COPY files/opt/jmeter/bin/user.properties /opt/jmeter/bin/user.properties
17
18ENTRYPOINT /tmp/start.sh
19
20EXPOSE 22
21EXPOSE 80

The server:

 1FROM m9d/jmeter-base
 2MAINTAINER manfred dreese <>
 3
 4# Install dependencies for client registration
 5RUN apt-get install -y redis-tools iproute2
 6
 7# Startscript
 8COPY files/tmp/start.sh /tmp/start.sh
 9COPY files/opt/jmeter/bin/user.properties /opt/jmeter/bin/user.properties
10
11ENTRYPOINT /tmp/start.sh