Consider a scenario where your website (or web application) works perfectly fine when there is a lower adoption. But the same experience fizzles out when there is higher load on the server. This experience can dampen your growth plans, which in turn will impact the revenue numbers 🙁
Speed and Quality must go hand-in-hand. Shipping new features at a rapid pace should not come at the cost of quality. No longer are teams using the traditional waterfall model for software development. Agile model has completely taken over, as testing and development can be done in a continuous manner.
DevOps (Developer Operations) has become more important than ever before. Continuous Delivery and Continuous Integration (CI/CD) processes help teams to build, integrate, test, and release code at a faster pace. Automation tests running in a CI/CD pipeline help in effectively managing the frequent code changes, irrespective of whether the code is deployed on prod/stage/or any other environment.
The major benefit of CI is that no part of the code goes untested, thereby improving the product quality. Performance tests that verifies the scalability, reliability, stability, and the responsiveness of the product can also be part of the CI pipeline. Doing so will ensure that the product features are built to work at a massive scale.
In this blog, we would look into performance testing from the lens of continuous integration (CI). Many developers and enterprises who want to scale performance testing efforts leverage the expertise offered by performance testing company since it impacts the overall TTM (time to market).
Quick Recap : Continuous Integration and Benefits
As mentioned earlier, Continuous Integration (CI) is a software process where the source code changes are pushed into the shared code repository at frequent time periods. The duration in which the integrated code in the shared repository is tested depends on the project requirements.
Automated performance testing is an integral part of the CI/CD pipeline since a large number of tests can be run as a part of the pipeline. Apache JMeter, LoadRunner, LoadView, and LoadNinja are some of the popular performance testing tools that let you run performance tests with CI.
Here are some of the major steps that are a part of the CI process:
- Developers push the code changes to the respective branch
- The code changes are reviewed by the reviewers
- Once the changes are approved by the reviewer, automated tests (of different types like UI, performance, etc.) that are a part of the pipeline are automatically triggered
- Depending on the test results of step (3), the build is marked as passed (or failed)
- If the result of step (4) is Pass, changes are deployed to the staging (or QA) environment
- If the result of step (5) is Pass, changes are selectively deployed (i.e. based on the features that must be a part of the build) to the Prod environment
Also Read: [Guide] Best Practices in Performance Testing
CI/CD tools like Jenkins, Circle CI, Travis CI, etc. can be used to run tests in a continuous manner. Here are some of the major benefits of Continuous Integration (CI):
- Improved Product Quality
- Faster Time to Market
- Cost Effectiveness
- Timely identification of bugs
A performance testing services company like KiwiQA that has deep-rooted experience working with a number of clients can help in building a top-notch performance testing strategy that does wonders with CI.
How to setup Performance Testing with Continuous Integration?
Now that we have touched upon the essentials of performance testing and continuous integration, let’s look at how you can weave the magic with them 🙂
Here are some of the major steps that you have to follow to integrate performance tests as a part of the CI pipeline:
Setting up the test environment
Like any other form of testing, the first step is to build the testing environment. You could opt for local infrastructure for running performance tests in a continuous manner but the major downside is scalability and maintainability.
Instead, it is recommended to opt for cloud-based performance testing tools as StormForge, WebLoad, NeoLoad, among opthers. The advantage of using cloud performance testing with CI is that you are able to scale up performance tests without being worried about the infrastructure.
In case you are using performance testing on the cloud, you need to ensure that the performance testers and developers are able to use the required tools on the cloud. Almost all the cloud-based performance testing tools have detailed reporting mechanism, thereby providing the team adequate information about the performance test results.
Since performance testing is also about testing the back-end interactions at scale, you need to evaluate if the cloud infrastructure is not acting a spoilt-sport in the process.
Setting up the test data
There are three categories of data that can be used for performance testing:
- Reusable data
- Non-reusable data that is retained even after test execution
- Non-reusable data that is reset after test execution
There could be usecases that require the state of the data to be retained across test scenarios. The updated data is saved in the back-end and retrieved for use at a later stage in the tests.
Before integrating performance tests in the CI pipeline, you need to setup the test data so that you can make the most of performance testing and CI.
Also Read: Load Testing vs. Performance Testing vs. Stress Testing
Choose adequate performance testing tool
As discussed in the earlier points, there are options to run performance tests on local infrastructure as well as on the cloud. There are a number of performance testing tools to choose from. Some of the popular ones are LoadNinja, Kobiton, WebLOAD, and more.
When choosing the performance testing tool, you should check the friendliness of the tool when used with CI/CD tools (e.g. Jenkins, Circle CI, etc.). Performance testing on the cloud has significant advantages that testing on the local infrastructure.
You should look at load balancers, version control systems, internal skillsets, and other such factors when shortlisting performance testing tools.
Prioritize Application Performance Monitoring
Performance monitoring should be done on a continuous basis when it is used with a CI pipeline. CPU usage, memory usage, individual web requests, and more such KPIs must be trackable via the performance monitoring tools.
You should be able to run the monitoring tests in a CI pipeline. DataDog, New Relic, App Dynamics, DynaTrace, and OpsView are some of the popular APM tools. The popular performance monitoring tools that we listed in the earlier point support integration with most of the APM tools.
For example – JMeter tests results can be monitored using DataDog through JMeter-DataDog integrations[1]. Integration between Application Performance Monitoring (APM) tools and Performance Testing (PT) tools is one of the major factors that must be considered when looking out for tools for performance testing.
Execute tests in the Pipeline
Once you have the PM and APM tools handy, the next step is to run the performance tests as a part of the pipeline. The frequency of the test execution purely depends on the project type.
Periodic monitoring of error rates and invalidation of tests (in error cases) are some of the best practices that must be followed when running performance tests in a CI pipeline.
Tests must be run more frequently in cases where the code has to be tested on Prod. This is not a thumb rule and you could decide the frequency depending on the product complexity, product features, target audience, and more.
Analyze test results
Test reporting is one of the integral parts of any form of automation testing. Performance monitoring is no different! Results derived by using Performance Testing tools and Application Monitoring tools (APM) must be presented in a format that can be understood by the managers, engineers, and other technical members involved in the team.
Many performance testing tools also provide integrations with popular test reporting tools, thereby helping them with more in-depth test results.
Purge the test environment
Finally, once the test execution is completed; it is recommended to purge the test envinroment. Running tests in a cleaned environment is important else the traces of the earlier test runs could result in false-postitives in the current runs.
Also Read: How To Plan A Foolproof Performance Testing Plan?
It’s A Wrap
Performance testing has become an absolute must for websites and applications that would be used by a large user base. In order to accelerate quality releases, performance tests have to be run on a frequent basis in a CI (Continuous Integration) pipeline.
As seen in this blog, it is recommended to opt for cloud-based performance testing tools in order to make the most of the tests when run in the CI pipeline. You should also look at the integration portfolio of the cloud performance testing tool.
In case you do not have in-house expertise with performance testing, you should onboard a performance testing services company so that your team can reap the benefits offered by performance testing and continuous integration. The immense performance testing experience of the outsourced QA vendor will be helpful in testing product features for their scalability, reliability, usability, and beyond.