Load Performance Testing (deprecated) (PREMIUM)
Introduced in GitLab 13.2.
WARNING: This feature was deprecated in GitLab 15.9 and is planned for removal in 17.0. This change is a breaking change.
With Load Performance Testing, you can test the impact of any pending code changes to your application's backend in GitLab CI/CD.
GitLab uses k6, a free and open source tool, for measuring the system performance of applications under load.
Unlike Browser Performance Testing, which is used to measure how web sites perform in client browsers, Load Performance Testing can be used to perform various types of load tests against application endpoints such as APIs, Web Controllers, and so on. This can be used to test how the backend or the server performs at scale.
For example, you can use Load Performance Testing to perform many concurrent GET calls to a popular API endpoint in your application to see how it performs.
How Load Performance Testing works
First, define a job in your
.gitlab-ci.yml file that generates the
Load Performance report artifact.
GitLab checks this report, compares key load performance metrics
between the source and target branches, and then shows the information in a merge request widget:
Next, you need to configure the test environment and write the k6 test.
The key performance metrics that the merge request widget shows after the test completes are:
- Checks: The percentage pass rate of the checks configured in the k6 test.
- TTFB P90: The 90th percentile of how long it took to start receiving responses, aka the Time to First Byte (TTFB).
- TTFB P95: The 95th percentile for TTFB.
- RPS: The average requests per second (RPS) rate the test was able to achieve.
If the Load Performance report has no data to compare, such as when you add the
Load Performance job in your
.gitlab-ci.yml for the very first time,
the Load Performance report widget doesn't display. It must have run at least
once on the target branch (
main, for example), before it displays in a
merge request targeting that branch.
Configure the Load Performance Testing job
Configuring your Load Performance Testing job can be broken down into several distinct parts:
- Determine the test parameters such as throughput, and so on.
- Set up the target test environment for load performance testing.
- Design and write the k6 test.
Determine the test parameters
The first thing you need to do is determine the type of load test you want to run, and how you want it to run (for example, the number of users, throughput, and so on).
Refer to the k6 docs, especially the k6 testing guides, for guidance on the above and more.
Test Environment setup
A large part of the effort around load performance testing is to prepare the target test environment for high loads. You should ensure it's able to handle the throughput it is tested with.
It's also typically required to have representative test data in the target environment for the load performance test to use.
We strongly recommend not running these tests against a production environment.
Write the load performance test
After the environment is prepared, you can write the k6 test itself. k6 is a flexible tool and can be used to run many kinds of performance tests. Refer to the k6 documentation for detailed information on how to write tests.
Configure the test in GitLab CI/CD
When your k6 test is ready, the next step is to configure the load performance
testing job in GitLab CI/CD. The easiest way to do this is to use the
template that is included with GitLab.
NOTE: For large scale k6 tests you need to ensure the GitLab Runner instance performing the actual test is able to handle running the test. Refer to k6's guidance for spec details. The default shared GitLab.com runners likely have insufficient specs to handle most large k6 tests.
This template runs the k6 Docker container in the job and provides several ways to customize the job.
An example configuration workflow:
Set up GitLab Runner to run Docker containers, like the Docker-in-Docker workflow.
Configure the default Load Performance Testing CI/CD job in your
.gitlab-ci.ymlfile. You need to include the template and configure it with CI/CD variables:
include: template: Verify/Load-Performance-Testing.gitlab-ci.yml load_performance: variables: K6_TEST_FILE: <PATH TO K6 TEST FILE IN PROJECT>
The above example creates a
load_performance job in your CI/CD pipeline that runs
the k6 test.
For Kubernetes setups a different template should be used:
k6 has various options to configure how it runs the tests, such as what throughput (RPS) to run with,
how long the test should run, and so on. Almost all options can be configured in the test itself, but as
you can also pass command line options via the
For example, you can override the duration of the test with a CLI option:
include: template: Verify/Load-Performance-Testing.gitlab-ci.yml load_performance: variables: K6_TEST_FILE: <PATH TO K6 TEST FILE IN PROJECT> K6_OPTIONS: '--duration 30s'
GitLab only displays the key performance metrics in the MR widget if k6's results are saved via summary export as a Load Performance report artifact. The latest Load Performance artifact available is always used, using the summary values from the test.
If GitLab Pages is enabled, you can view the report directly in your browser.
Load Performance testing in Review Apps
The CI/CD YAML configuration example above works for testing against static environments, but it can be extended to work with review apps or dynamic environments with a few extra steps.
The best approach is to capture the dynamic URL in a
as a job artifact to be shared, then use a custom CI/CD variable we've provided named
to configure the k6 Docker container to use the file. With this, k6 can then use any
environment variables from the
- In the
- Capture the dynamic URL and save it into a
.envfile, for example,
echo "ENVIRONMENT_URL=$CI_ENVIRONMENT_URL" >> review.env.
- Set the
.envfile to be a job artifact.
- Capture the dynamic URL and save it into a
- In the
- Set it to depend on the review job, so it inherits the environment file.
- Set the
K6_DOCKER_OPTIONSvariable with the Docker CLI option for environment files, for example
- Configure the k6 test script to use the environment variable in it's steps.
.gitlab-ci.yml file might be similar to:
stages: - deploy - performance include: template: Verify/Load-Performance-Testing.gitlab-ci.yml review: stage: deploy environment: name: review/$CI_COMMIT_REF_SLUG url: http://$CI_ENVIRONMENT_SLUG.example.com script: - run_deploy_script - echo "ENVIRONMENT_URL=$CI_ENVIRONMENT_URL" >> review.env artifacts: paths: - review.env rules: - if: $CI_COMMIT_BRANCH # Modify to match your pipeline rules, or use `only/except` if needed. load_performance: dependencies: - review variables: K6_DOCKER_OPTIONS: '--env-file review.env' rules: - if: $CI_COMMIT_BRANCH # Modify to match your pipeline rules, or use `only/except` if needed.