Is JMeter URL encoding everything through the HTTP Proxy? This is how to fix it!

I was having problems with JMeter URL encoding everything through the HTTP proxy which was causing me serious grief. One application I’m performance testing sends XML over HTTP, which when URL encoded turns <” symbols into %3Es and other weird things that make servers panic and fail.

I made sure that encoding was switched off when the sampler was run, but that didn’t fix the problem. I checked data flowing through the JMeter HTTP Proxy and encoding was happening there compared to the normal application. Wireshark was great to figure this out.

The problem? The sampler HTTP Request HTTPClient. This is the Apache sampler and is generally better than HTTP Request in most situations I’ve been told. This is one situation where it wasn’t. I switched to HTTP Request, and all my dramas went away.

Extracting variables using Regular Expressions in JMeter

I’m trying to web performance test an application that regularly changes URLs. We’ll use regular expressions to figure out what that URL should be, store it as a JMeter variable, and use it in a sampler.

So first of all, we do a search for a book at a known page at the following address /cgi-bin/.exe/ENQ/CAT/BIBENQ. This address and the screenshots have been edited to remove the application name and the host address. From the result of this we’ve received some data as you can see in the second screenshot.

This data comes in the format of a URL: /cgi-bin/.exe/FULL/CAT/BIBENQ/XXXXX/410986,1, where XXXXX is a five digit number that changes constantly. This means we can’t set up a sampler with a static address, as the address won’t always be correct.

So for the sampler with the result data, we’ll right click and add Post Processor > Regular Expression Extractor. The fields in details are:

  • Reference Name – This is the variable name, in my case FLUTEURL. Elsewhere in JMeter, we’ll reference this variabvle as ${FLUTEURL}
  • Regular Expression – This is the regular expression that will extract the data we’re looking for. Brackets are put around the value we want to extract. So in our example we have /cgi-bin/.exe/FULL/CAT/BIBENQ/(.+?)/410986,1 or, we’re looking for the value that sits between the slash after BIBENQ and the slash before 410986,1. This will return in our case a five digit number, XXXXX
  • Template – Your regular expression could have multiple values extracted, in our case we have just one, and so we’re using the first template, or $1$
  • Match No. – If there are multiple matches, which one should be extracted? In our case, we want the first match, or 1
  • Default value – If a value cannot be extracted, what should we put in place instead? You could have ERROR, but I just leave it blank

So we now have a JMeter variable called ${FLUTEURL} with a value of say, 12345. How would we use this? In my case, for the path for the next sampler is:

  • /cgi-bin/.exe/FULL/CAT/BIBENQ/${FLUTEURL}/410986,1

So in execution the URL the sampler requests is /cgi-bin/.exe/FULL/CAT/BIBENQ/12345/410986,1 and the sampler works correctly.

In summary, we’ve extracted a value from a page, stored it as a JMeter variable, and then used that variable in another sampler.

Using the HTTP Proxy in JMeter to record web interactions

I’ve been trying to figure out the performance of an application that runs over HTTP. Sounds like a perfect candidate for JMeter!

In the past, I’ve had to deal with native HTTPS applications which means I couldn’t use the HTTP Proxy to record sessions. However, since this application functions in both HTTP and HTTPS modes, this was my chance to understand and use the HTTP Proxy, so let’s go through how to set up and do this.

First, right click on the WorkBench and add the Non Test Element HTTP Proxy Server. Non Test Elements only appear when you right click on the WorkBench, so if you’ve never seen them before, now you know why.

Here are the options I recommend:

  • Port – I use 8000 just because I might be running an Application Server on port 8080. Any port over 1024 will see you right.
  • Target Controller – Select Use Recording Controller. We’ll add a Recording Controller just after this.
  • Grouping – I use Store 1st sampler of each group only just because I don’t want to save javascript or image files, just the HTML file being served. You may wish to pick a different option if you want to know the total amount of time to transfer all files to the end user for your performance test.
  • Capture HTTP Headers – I ticked it.
  • Add Assertions – I ticked it. Adds a blank assertion for each sampler. Since you’ll be using assertions later on to determine how requests are going, save time by adding a blank one now.
  • HTTP Sampler Type: HTTP Request HTTPClient. This is using the Apache HTTP Request library over the standard Java library.
  • Follow Redirects – I ticked it. This depends on the behaviour of your web application.
  • Redirect Automatically – I left blank. This also depends on the behaviour of your web application.

There are content filters you can add to stop sampling specific file types such as GIF and JPG, but since I just store the first sampler anyway, I normally leave these blank. Same with URL Pattersm to Exclude.

Right click on WorkBench and add Logic Controller > Recording Controller. All samplers recorded will be stored under here.

My useful tip – right click on the HTTP Proxy Server and add Listener > View Results Tree. This way you can analyse the requests being recorded by the HTTP Proxy.

Finally go back to the HTTP Proxy and click Start. All requests going through the HTTP Proxy on port 8000 are now being recorded.

Now head to your favourite web browser and set the HTTP proxy settings for localhost on port 8000. Anything you do now in that browser will be recorded by JMeter.

As you can see in the View Results Tree listener, you can see all the requests being made by the browser, such as:

  • the Sampler name
  • the Sampler result
  • the HTTP response code
  • Request information showing the actual request, and headers associated
  • And the actual response data, rendered as HTML

When you expand the Recording Controller, you’ll see the HTTP Request with the:

  • Server name
  • Protocol (HTTP, HTTPS)
  • Method (GET or POST)
  • Parameters (variables sent in the request)

All filled in for you. You’ll even see a HTTP Header Manager all filled in for you as well. Some web applications pay close attention to this, so it’s good to see all the information a normal browser sends.

So in summary, we’ve set up an HTTP Proxy to record sessions going through JMeter, and we’ve highlighted some of the options and what they do.