The Venture of San Francisco

System loading test by using Artillery

Posted on Edited on In Disqus:

artillery-logo.png?w=2924

There are so many choices for doing a system loading test on backend side. Let’s see the list below:

You can read this post to get more detail information. Choosing Artillery is because it can provide the test scenario for developers to design a test context. After doing some researches and adapting to this tool, I found some advantages for using it to do the web benchmark.

Let’s start from scratch. The first step would be installing it.

1
npm install -g artillery

After the installation, you can write some tests for your APIs. Artillery supports the scenarios testing for your APIs, which means you can create different test scenarios for your API endpoints and make sure everything is ok. Let’s check the primary setting of Artillery:

Hello.yml

1
2
3
4
5
6
7
8
9
10
11
12
config:
  target: 'https://artillery.io'
  phases:
    - duration: 60
      arrivalRate: 20
  defaults:
    headers:
      x-my-service-auth: '987401838271002188298567'
scenarios:
  - flow:
    - get:
        url: "/docs"

This hello.yml file runs a test that uses the GET method to test the API path /docs on the target server https://artillery.io. It’s pretty self-explained. One of the most significant benefits of using Artillery is that the learning curve is pretty easy for developers. You can execute the yaml file in this way:

1
artillery run hello.yml

You might see the result like the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
All virtual users finished
Summary report @ 15:02:32(-0700) 2019-09-11
  Scenarios launched:  1200
  Scenarios completed: 1200
  Requests completed:  2400
  RPS sent: 39.72
  Request latency:
    min: 0.4
    max: 9.7
    median: 1.5
    p95: 2.6
    p99: 4
  Scenario counts:
    Get Doc: 1200 (100%)
  Codes:
    200: 2400

There are several terms you might not understand. It’s ok and let’s go through all of them.

  • Scenarios launched is the number of virtual users created in the preceding 10 seconds (or in total)

  • Scenarios completed is the number of virtual users that completed their scenarios in the preceding 10 seconds (or in the whole test). Note: this is the number of completed sessions, not the number of sessions started and completed in a 10-second interval.

  • Requests completed is the number of HTTP requests and responses or WebSocket messages sent

  • RPS sent is the average number of requests per second completed in the preceding 10 seconds (or throughout the test)

  • Request latency is in milliseconds, and p95 and p99 values are the 95th and 99th percentile values (a request latency p99 value of 500ms means that 99 out of 100 requests took 500ms or less to complete).

  • Codes provides the breakdown of HTTP response codes received. If you see NaN (“not a number”) reported as a value, that means not enough responses have been received to calculate the percentile.

If there are any errors (such as socket timeouts), those would be printed under Errors in the report as well.

So, I think it’s quite clear if you check the report. Some people might like the Wrk style report:

1
2
3
4
5
6
7
8
Running 30s test @ http://127.0.0.1:8080/index.html
  12 threads and 400 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   635.91us    0.89ms  12.92ms   93.69%
    Req/Sec    56.20k     8.07k   62.00k    86.54%
  22464657 requests in 30.00s, 17.76GB read
Requests/sec: 748868.53
Transfer/sec:    606.33MB

But I think that’s a matter of debate, totally depends on what kind of format you like.

Let’s check the two different parts of hello.yml file — the first one, config. The config part is where you specify the target (such as the address of the API server under test), the load progression (telling Artillery, for example, to create 20 virtual users every second for 10 minutes), and can set a variety of other options such as HTTP timeout settings, or TLS settings.

The second part, scenarios. This section is where you define what virtual users created by Artillery would be doing. A scenario is a description of a typical user session in the application. For example, in an e-commerce application, a common scenario is to search for a product, add it to cart, and checkout. In a chat application, it may be to connect to the server, send a few messages, lurk for a while, and then disconnect. Remember what I mentioned before; you can create different scenarios for different test groups. Note: you are testing a behavior logic in a scenario. Here is an example of testing a shopping cart checkout scenario:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
config:
  target: "https://staging1.local"
  phases:
    - duration: 60
      arrivalRate: 5
    - duration: 120
      arrivalRate: 5
      rampTo: 50
    - duration: 600
      arrivalRate: 50
  payload:
    path: "keywords.csv"
    fields:
      - "keywords"
scenarios:
  - name: "Search and buy"
    flow:
      - post:
          url: "/search"
          body: "kw={{ keywords }}"
          capture:
            json: "$.results[0].id"
            as: "id"
      - get:
          url: "/details/{{ id }}"
      - think: 3
      - post:
          url: "/cart"
          json:
            productId: "{{ id }}"

This scenario describes a context that I want to search for something with the keyword variable, which is loaded by the payload path. After I get a specific product, I want to see the product detail information. If I like the product, I’d like to put it into the shopping cart. This context is the so-called scenario testing. You set up a scenario(it’s primarily a use-case) that covers several APIs you want to test.

Here are some tips I want to share with you:

  • Tip: you might have several environments, for instance, staging, production, and other pre-defined environments. What we want to test APIs load in different environments? Artillery gives you this ability, don’t worry.

your-test-load.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
config:
  environments: 
    staging: 
      target: 'https://your.staging.env'
    production: 
      target: 'https://your.production.env'
  phases:
    - duration: 60
      arrivalRate: 20
  defaults:
    headers:
      x-my-service-auth: '987401838271002188298567'
scenarios:
  - flow:
    - get:
        url: "/docs"

You can define an environments tag here, and use the following command:

1
artillery run -e staging your-test-load.yaml

You can choose different environments in this way.

  • Tip: do you want to cache something from an API call? Try this.

your-test-load.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
config:
  environments: 
    staging: 
      target: 'https://your.staging.env'
    production: 
      target: 'https://your.production.env'
  phases:
    - duration: 60
      arrivalRate: 20
  defaults:
    headers:
      x-my-service-auth: '987401838271002188298567'
scenarios:
  - flow:
    - get:
        url: "/docs"
        capture:
          json: "$.id"
          as: "id"
    - get: 
        url: "/docs/{{ id }}"

You might try to capture a field called id from the response. Remember, each JSON should have a paired as keyword. Then you can directly use this id in other APIs.