There are so many choices for doing a system loading test on backend side. Let’s see the list below:
You can read this post to get more detail information. Choosing Artillery is because it can provide the test scenario for developers to design a test context. After doing some researches and adapting to this tool, I found some advantages for using it to do the web benchmark.
Let’s start from scratch. The first step would be installing it.
npm install -g artillery
After the installation, you can write some tests for your APIs. Artillery supports the scenarios testing for your APIs, which means you can create different test scenarios for your API endpoints and make sure everything is ok. Let’s check the primary setting of Artillery:
This hello.yml file runs a test that uses the
GET method to test the API path
/docs on the target server
https://artillery.io. It’s pretty self-explained. One of the most significant benefits of using Artillery is that the learning curve is pretty easy for developers. You can execute the yaml file in this way:
artillery run hello.yml
You might see the result like the following:
All virtual users finished
There are several terms you might not understand. It’s ok and let’s go through all of them.
Scenarios launched is the number of virtual users created in the preceding 10 seconds (or in total)
Scenarios completed is the number of virtual users that completed their scenarios in the preceding 10 seconds (or in the whole test). Note: this is the number of completed sessions, not the number of sessions started and completed in a 10-second interval.
Requests completed is the number of HTTP requests and responses or WebSocket messages sent
RPS sent is the average number of requests per second completed in the preceding 10 seconds (or throughout the test)
Request latency is in milliseconds, and p95 and p99 values are the 95th and 99th percentile values (a request latency p99 value of 500ms means that 99 out of 100 requests took 500ms or less to complete).
Codes provides the breakdown of HTTP response codes received. If you see NaN (“not a number”) reported as a value, that means not enough responses have been received to calculate the percentile.
If there are any errors (such as socket timeouts), those would be printed under Errors in the report as well.
So, I think it’s quite clear if you check the report. Some people might like the Wrk style report:
Running 30s test @ http://127.0.0.1:8080/index.html
But I think that’s a matter of debate, totally depends on what kind of format you like.
Let’s check the two different parts of
hello.yml file — the first one, config. The config part is where you specify the target (such as the address of the API server under test), the load progression (telling Artillery, for example, to create 20 virtual users every second for 10 minutes), and can set a variety of other options such as HTTP timeout settings, or TLS settings.
The second part, scenarios. This section is where you define what virtual users created by Artillery would be doing. A scenario is a description of a typical user session in the application. For example, in an e-commerce application, a common scenario is to search for a product, add it to cart, and checkout. In a chat application, it may be to connect to the server, send a few messages, lurk for a while, and then disconnect. Remember what I mentioned before; you can create different scenarios for different test groups. Note: you are testing a behavior logic in a scenario. Here is an example of testing a shopping cart checkout scenario:
This scenario describes a context that I want to search for something with the keyword variable, which is loaded by the payload path. After I get a specific product, I want to see the product detail information. If I like the product, I’d like to put it into the shopping cart. This context is the so-called
scenario testing. You set up a scenario(it’s primarily a use-case) that covers several APIs you want to test.
Here are some tips I want to share with you:
- Tip: you might have several environments, for instance, staging, production, and other pre-defined environments. What we want to test APIs load in different environments? Artillery gives you this ability, don’t worry.
You can define an
environments tag here, and use the following command:
artillery run -e staging your-test-load.yaml
You can choose different environments in this way.
- Tip: do you want to cache something from an API call? Try this.
You might try to capture a field called id from the response. Remember, each JSON should have a paired
as keyword. Then you can directly use this id in other APIs.