Technologies like AI (Artificial Intelligence), ML (Machine Learning), etc. can be easily considered as one of the most influential technologies that are having a monumental impact on the technology ecosystem. Under the hood, AI-language models use complex algorithms and learning techniques through which it understands and generates Natural Language response.
ChatGPT by OpenAI is one of the most significant advancements in the field of NLP (Natural Language Processing). Such was the excitement around the product that close to a million users signed up on ChatGPT in just five days after the launch. As a user, you can literally ask anything in a natural language to ChatGPT and the engine would provide human-like answers based on the data trained on a massive dataset from the internet.
ChatGPT is super-powerful since it has the potential to write poems, construct emails, craft social media content, and much more. Not to forget, ChatGPT can also be used in various aspects related to automated testing, It can be used for test case generation, test case management, base test code creation, etc.
The fascinating thing about ChatGPT is its potential to craft human-readable, accurate, working boilerplate code that can build the base for your test suite(s). It can leverage popular test automation frameworks (e.g. Selenium, Cypress, Playwright, etc.) and programming languages (e.g. Python, Java, etc.) to come up with the test code!
The icing on the cake is that ChatGPT can also create sample YAML files for running the automation tests in a CI pipeline (e.g. Jenkins, CircleCI, etc.). In this blog, we deep dive into the endless possibilities of harnessing ChatGPT for various activities related to test automation. Let’s get started…
What is ChatGPT?
Built by OpenAI, ChatGPT is a language model that is based on the GPT (Generative Pre-trained Transformer)3.5 architecture. It has been fine-tuned using supervised and reinforcement learning techniques (Source). Since ChatGPT is a dialogue-based model, it has the capability to understand human-like questions and provide answers that can be very-well understood by humans.
Since ChatGPT is pre-trained on a large corpus of data from the internet, it generates responses that are ideally accurate and grammatically correct! In fact, OpenAI also offers ChatGPT & Whisper APIs (Application Programming Interfaces), using which developers can integrate ChatGPT & Whisper models in their applications (Source).
A number of developers who signed up for the OpenAI API Beta Program managed to create some useful applications using the APIs. For instance, a user created a fully-functional AMA (Ask Me Anything) search engine using GPT3.
The possibilities with ChatGPT and APIs offered by OpenAPI are endless but we would restrict the discussion to ChatGPT and test automation. You can find more such useful applications here.
How does ChatGPT work?
As shown in the diagram below, the ChatGPT architecture consists of two major blocks – Training and Answer.
How ChatGPT works
The training block in turn comprises of the following stages:
- Pre-training stage
- Fine-tuning stage
Reinforcement learning (or PPO optimization) is applied in the fine-tuning stage to fine-tune the model so that ChatGPT provides a more accurate answer to the user(s).
As the name suggests, the response (or answer) is generated in the ‘Answer a prompt’ stage where the generated answer is shown to the user once it passes the safety and moderation phase. We would recommend you to read How ChatGPT works to get more granular information about the internal working of this amazing product!
ChatGPT and Test Automation
Now that we have covered the essentials of ChatGPT, let’s look at some of the cases where it can give a huge head start! Let’s look at the different types of testing (e.g. UI testing, API testing, etc.) where it can be used.
ChatGPT and UI Automation Testing
Consider a scenario where you are planning to use popular test automation frameworks like Selenium, Cypress, Playwright, etc. for testing interactions with elements in the DOM (Document Object Model). Well, ChatGPT can be instrumental in generating the source code that can be a starter for your detailed test suite.
Here are some of the questions that we tried asking to ChatGPT, the answers were bang on target!
Q. Write a test using Selenium framework and Java to search “KiwiQA” on Google. Click the first result after the search.
Shown below is the code generated by ChatGPT. The code is accurate and works when we executed the same on our system.
As seen above, the code uses the CSS Selector for locating the elements in the DOM. Though CSS Selector is a better selector than XPath, we still asked ChatGPT to provide us with the code that uses the XPath selector for locating the WebElement.
And voila, it actually worked!
Next we tried leveraging ChatGPT for creating an automation test that makes use of a BDD (Behaviour-Driven Development) framework to perform the same task i.e. Search “KiwiQA” on Google
Q. Can you provide the SpecFlow code which searches for “KiwiQA” on Google.
Shown below is the .feature file generated by ChatGPT. As seen in the snapshot, the file is tagged as an SQL file which seems to be an error from the tool 🙁
Feature File
Here is the step definition file where the tool has created steps corresponding to the scenarios mentioned in the .feature file:
Barring aside some minor issues that we listed earlier, the code looks perfectly fine. The best part is that it also mentioned the dependencies that have to be installed as a part of executing the above code.
Note that in order to use SpecFlow with C#, you will need to install the appropriate packages – SpecFlow, SpecFlow.NUnit, and SpecFlow.Tools.MsBuild.Generation. You will also need to download and configure the ChromeDriver executable file as well, similar to the Java example.
We tried this after creating a project in Visual Studio and it worked like a charm! It also gave the following note in the answer that acts as a “Word Of Caution”.
Many times, teams are confused on whether they should opt for BDD or TDD? Though this opinion largely depends on the team’s expertise, ChatGPT can still be used to get some inputs that can aid in making an informed decision.
Q. When to choose BDD framework over TDD?
Here is the answer from ChatGPT that very well summarizes the advantages of BDD over TDD.
In a nutshell, ChatGPT can provide you with code that can act as a good starting point for kickstarting the activities in your automation project. Though the code needs to be tweaked as per your project (or product) requirements, the boilerplate code provided by ChatGPT can come handy in saving those extra hours that you (or your team) will normally spend in creating the base code!
Since ChatGPT uses the information being inputted by the users to train their models, it is recommended not to upload/ask any confidential information to the platform! You can always partner with an experienced software testing consulting services company so that you are able to accelerate the project delivery using the combination of AI and human intelligence!
ChatGPT and DevOps
Software projects (irrespective of size or scale) make use of DevOps like Jenkins, CircleCI, Travis CI, etc. to run automation tests in a continuous manner. There is extensive usage of YAML files for defining the pipelines with a specific DSL (Domain Specific Language).
We asked ChatGPT to create a Jenkins pipeline for running Selenium tests in parallel.
Q. Create a Jenkins pipeline for running Selenium tests in parallel
ChatGPT provided a Jenkinsfile with four stages: Checkout, Build, Run tests, and Publish test results. It also gave details of every stage in the pipeline.
Here is the complete Jenkinsfile generated by ChatGPT:
We upped the ante a bit and asked ChatGPT to port an existing Jenkins pipeline file to CircleCI.
Q. Can you help me convert a Jenkins pipeline script to CircleCI script?
In this case, we provided the earlier Jenkins script as an input to ChatGPT.
Here the configuration file (i.e. .circleci/config.yml) was correctly identified as a YAML file. The interesting part in the answer was ChatGPT recommending OpenJDK (instead of the commercial Java variant).
In a nutshell, ChatGPT can be used for creating pipelines for a range of CI tools like Jenkins, CircleCI, Bamboo, Bitbucket, and more.
ChatGPT and Microservices Testing
Microservices architecture is a type of software architecture where the application is developed as a collection of services that interact with each other using APIs. Here are the major building blocks of a microservices-based architecture[1]:
- Microservices
- Containers
- Service mesh
- Service discovery
- API gateway
Since most of the software products are using microservices, we decided to try out ChatGPT so that it can help us with microservices testing.
Q. We have three microservices running in our system. Can you please provide test code to verify them out. The first microservice comprises of Selenium tests, second test makes entries into the MySQL database, and the third one does API testing using SoapUI
ChatGPT provided us with a Python test suite that uses Selenium, MySQL, and SoapUI to test three microservices.
ChatGPT also goes on to mention about the details of the tests and instructs to run the tests using popular runners like Pytest or Nose2.
In case your project is making use of the Microservices architecture, ChatGPT can provide you with the base code for testing the respective microservice. It is important to reiterate that the generated code for microservices testing can be used for educational or reference purposes.
However, ChatGPT can be instrumental in providing you guidance on how to modernize an application by migrating from Monolithic to Microservices based architecture. We came across an interesting question where a Telecom executive asked ChatGPT regarding application modernization to Cloud-native Microservices architecture.
Now that we have looked into the broad usage of ChatGPT for test automation, let’s deep dive into the Do’s & Don’ts of using this powerful tool/platform!
Advantages of ChatGPT for test automation
We had great fun leveraging ChatGPT to generate test scripts, define Jenkins pipeline, create test code for microservices testing, and more. Though inputs provided by ChatGPT gives you the information on which you can further build your code, it needs to be used with caution. First, let’s look at some of the major advantages of ChatGPT:
- ChatGPT can be instrumental in minimizing the learning curve when it comes to automation testing, DevOps, etc. Consider an example where you (as an automation tester) is given the task to create a Jenkins pipeline to run those tests in a continuous manner. In such cases, you can leverage ChatGPT to get high-level information on creating Jenkins pipeline for running automation tests. The output can then be appropriately tweaked to suit your needs.
- Creating scalable and reliable tests is an important factor that can play a huge role in minimizing the efforts involved in test code maintenance. AS seen earlier in the UI automation testing example, ChatGPT could quickly generate code that used a more reliable locator like CSS Selector in the source code. This is where you can use ChatGPT to generate test code that is more resilient, maintainable, and reliable in the short as well as long run.
A word of caution – ChatGPT is a conversational AI tool that stores your conversations to improve its accuracy.
Accuracy of ChatGPT
The accuracy level of ChatGPT is ~85 percent, a factor that should be enough to imbibe confidence in this awesome tool! However, it is suggested not to upload any confidential information (or code) to the tool; else it might have serious implications on your project & organization.
Conclusion
ChatGPT has tremendous potential due to the depth in the answers provided by the platform. If you are someone who wants to transition to automation testing/DevOps, etc.; you can leverage ChatGPT to prepare a blueprint of your learning path.
ChatGPT is a boon to automation testers since it helps in providing the base/boilerplate code on which you can further build the test cases (or test suite). Those worried about AI taking away their QA jobs can relax, since an AI-tool like ChatGPT will help you in creating well-designed test code at an accelerated pace.
AI can never replace human creativity which is why you should onboard an experienced software QA consulting services company so that your make the best worlds – human intelligence and AI intelligence 🔥🔥