My Journey With GitHub Copilot For Automation Testing Projects

As you have already heard, GitHub Copilot is an AI assistant specifically designed for coding purposes. According to GitHub research it should accelerate coding for developers. You can start a conversation with Copilot about your codebase, you can ask for help, you can delegate it a task, it gives you live suggestions, you can use it for code review and many other things according to GitHub claims. All that is possible through seamless integrations with largest code editors. The main idea behind it was to create an ensemble coding partner. My company is using GitHub Copilot for several months now. I wanted to share how I use GitHub Copilot in automation testing and is it worth it.

Installation of GitHub Copilot

Installation is quite easy, at least for Visual Studio Code. You only need to install extension for VS Code and login to your GitHub account to link the Copilot to your account. After that you can start using the assistant.

Code suggestions

One of the most used features of Copilot is predicting code as you type. From my experience this really does accelerate coding. Copilot will make suggestions based on your previous work and it is really good at guessing what your next line would look like. During writing article for debugging Cypress tests, I realized that Copilot is suggesting me the correct CSS selectors. Upon checking these selectors on the web app under test, I noticed that they were spot on.

I was able to accept the suggestion by hitting tab button or I can accept only one word at a time for partial match. I can see all the possible suggestions for my work if I open the Completions panel.

In Completions panel I see all the available suggestions and I can accept the one that I find the most useful. However, the largest impact I get by just typing tab button. The suggestions come up almost instantly and I can hit the button and move to the next line. The suggestions show the entire line of code as you can see on the picture above, not just the next action like intellisense is doing. I can complete a lot of work in a matter of minutes like this. Of course, I keep an eye of correctness of my code and is the naming convention maintained or is the intent written in the code clear enough. You cannot expect high autonomy from Copilot and you have to stay alert all the time.

Chat support

When I get stuck with the code, especially in debugging I ask for help in the chatbox. Copilot will generate some answers so you can start investigating the problem. It will not offer to solve the problem in your tests like Postman Postbot does. The suggestions from chat on their own should be enough to give you an idea where to search for the cause of an error. Usually, this is enough for me to find out what went wrong. Here is a simple demonstration of what an answer could look like. I asked the question at the top of the screen and the list of suggestions and solutions is quite long.

I even asked how to convert my spec file to Page Object Model and got a valid answer on how to do it with examples. It even offered the suggestion on how should I organize my framework files.

This is a basic example and for more complex examples your involvement would be necessary. Copilot does offer some starting point so you don’t have to start from scratch. If you do need to start from scratch you can for sure move faster.

Delegating tasks

I haven’t used extensively this feature because I am used to working on my own. However it can be useful for some less demanding tasks. If I want to ask GitHub Copilot for automation testing of final price of products in the cart I would simply write a comment in the Javascript file and press enter. I have asked the Copilot to add all the products from Array in cart and assert the total price of the order.

As you can see it made a mistake in the suggestion I accepted. It didn’t anticipate that the user would have to return to home page after adding item to the cart. When I changed that manually, the test worked.

To conclude

GitHub Copilot used in automation testing certainly accelerates development of tests. Without a doubt I can say that it requires constant vigilance. It does not have high level of autonomy and the Copilot’s outputs have to be constantly checked. GitHub Copilot is a coding assistant as described at the official pages. If you have an idea of typing a series of prompts and expect to get an immaculate test, you have too high expectations. It is just a tool which can be really powerful in the right hands. If you don’t have enough experience with writing automation tests it will not help you a lot. You still have to be in the driver’s seat and after all provide the guidance on what should be asked and accepted.