AI Quality Assurance Tools Review: TestCraft

There is (no) AI in QA

I came to an idea to review some of the most popular AI Quality assurance tools I could find on the internet. One of my first picks would be TestCraft Chrome extension. Just to say I am not affiliated to TestCraft in any way. I just want to test how useful this tool is for Quality engineers. Could it help us with our day to day activities? I already shared my experience with GitHub Copilot and this feels like a sequel to AI series of articles.

What is TestCraft?

TestCraft is a simple Chrome extension which helps you with generating test cases for selected elements on the page. It is free for use and quite simple. The installation is as simple as installing any Chrome extension. Sign up is not required neither should you provide any personal information. The software is open source and distributed under MIT license. Their website is quite simple and offers just enough information for an easy start. You can find usage examples on GitHub pages. The extension is pretty straight forward to use and I believe the page offers just enough information.

Installation

As already mentioned installation is quite easy. You can download the extension from Chrome Web Store which you can access from your Chrome browser. At the moment of writing this article the extension has 3.5 stars. The negative comments are coming usually from inability to use the extension. I had similar issue when I installed it the first time, so I removed it when I couldn’t use it on specific web app. However, I installed it again and it worked fine afterwards even on websites it wouldn’t work the first time. One thing to keep an eye on is several extensions in web store which have the same name among which some of them have labels in comment section as malware. To be sure that you are installing the right thing you can click Install button from TestCraft website. It will redirect you to the correct extension page.

The functionality

It offers three features: generation of test ideas, automation and accessibility.

Finding elements

What you would usually do is navigate to the page you want to test and then click on Pick Element button. This will enable inspector for your mouse. All you need to do is to hover over element on the page and click on it. The element selection confirmation message appears and you can proceed with next actions. This feature has a limitation, it doesn’t work with iFrames. It does locate an iFrame but it does not recognize it as an element or any of the elements inside iFrame.

Generating test ideas

After you select an element on the page you can click Generate Test Ideas. You will get a list of test ideas what you can try with that element. In the following example I selected an Add to cart button and clicked Generate Test Ideas. I got a list of tests for positive and negative testing along with some creative testing ideas.

I can edit the ideas if I want to change something and I can select the ones I like and copy them or automate them. Which brings us to the next feature.

Automate tests

When I find an element and click automate button, I will get a Page Object Model created in a selected framework with programming language of my choice. The supported frameworks are Cypress, Playwright and Selenium with Javascript, Typescript, Java, C# and Python.

Generating scripts seems to be working fine in POM model. Asserts should be modified manually which is not a bad thing.

Here is the output for the tests in Selenium with C# copied as is with all the comments:

Generated Tests From Ideas
Copy to clipboard
// PageObjectModel.cs
using OpenQA.Selenium;
using OpenQA.Selenium.Support.UI;

public class PageObjectModel
{
    private readonly IWebDriver driver;
    private readonly WebDriverWait wait;

    public PageObjectModel(IWebDriver webDriver)
    {
        driver = webDriver;
        wait = new WebDriverWait(driver, TimeSpan.FromSeconds(10));
    }

    public IWebElement AddToCartButton => wait.Until(d => d.FindElement(By.CssSelector("a.btn.btn-success.btn-lg")));

    public void ClickAddToCart()
    {
        AddToCartButton.Click();
    }

    public string GetConfirmationMessage()
    {
        return wait.Until(d => d.FindElement(By.Id("confirm-message-id"))).Text; // Replace with actual ID of confirmation message
    }

    public void LoginIfNotLoggedIn()
    {
        // Logic to check if user is logged in and prompt to log in if not
    }
}

// Tests.cs
using NUnit.Framework;
using OpenQA.Selenium;
using OpenQA.Selenium.Chrome;

[TestFixture]
public class ShoppingCartTests
{
    private IWebDriver driver;
    private PageObjectModel page;

    [SetUp]
    public void Setup()
    {
        driver = new ChromeDriver();
        driver.Navigate().GoToUrl("https://www.demoblaze.com/prod.html?idp_=1#");
        page = new PageObjectModel(driver);
    }

    [Test]
    public void VerifyAddToCart()
    {
        page.ClickAddToCart();
        string confirmationMessage = page.GetConfirmationMessage();
        Assert.IsTrue(confirmationMessage.Contains("item added to cart"), "Confirmation message not displayed.");
    }

    [Test]
    public void CheckLoginPromptWhenNotLoggedIn()
    {
        page.LoginIfNotLoggedIn();
        // Assume a prompt appears; replace with actual logic to verify the prompt
        Assert.IsTrue(driver.PageSource.Contains("Please log in"), "Login prompt not displayed.");
    }

    [Test]
    public void TestAddToCartWithSlowConnection()
    {
        // Simulate slow connection
        // Assuming there's a way to simulate this in your testing environment

        page.ClickAddToCart();
        string confirmationMessage = page.GetConfirmationMessage();
        Assert.IsTrue(confirmationMessage.Contains("item added to cart"), "Confirmation message not displayed.");
    }

    [TearDown]
    public void TearDown()
    {
        driver.Quit();
    }
}

Check accessibility

The third feature is to check accessibility of a web page. I believe I haven’t ran into many AI quality assurance tools with option for accessibility testing. Upon clicking Check Accessibility button the plugin starts to check for compliance and displays all the issues on the web page. Here is the output for the product page of some item.

Issues

Conformance Level A –

• Issue: Missing alternative text for non-text content
• Criteria: 1.1.1 Non-text Content
• Solution: Ensure that any non-text content, if present, has a text alternative that serves the same purpose. In this case, since there is no non-text content, this is not applicable.

Conformance Level AA –

• Issue: No visible focus indicator for keyboard navigation
• Criteria: 2.4.7 Focus Visible
• Solution: Ensure that the link has a visible focus indicator when navigated to via keyboard. You can achieve this by adding appropriate CSS styles to the .btn class to show a focus outline.
• Issue: The link’s purpose is not clear from the context alone
• Criteria: 2.4.4 Link Purpose (In Context)
• Solution: Ensure the link’s purpose is clear. You could add more context or descriptive text around the button or use ARIA attributes to clarify the action.

Conformance Level AAA –

• Issue: No indication of the action being taken on the button
• Criteria: 3.3.3 Error Suggestion
• Solution: Provide additional context or feedback after the button is clicked to inform users what action has been taken (e.g., “Item added to cart”).

Suggested Tests

• Test: Check for visible focus indicator
• Criteria: 2.4.7 Focus Visible
• Test Details: Use keyboard navigation (Tab key) to navigate to the button and observe if there is a visible outline or style change indicating focus.
• Test: Verify link purpose clarity
• Criteria: 2.4.4 Link Purpose (In Context)
• Test Details: Review the surrounding content to ensure the purpose of the “Add to cart” button is clear without additional context. If necessary, check with users to confirm understanding.
• Test: Check for action feedback after button click
• Criteria: 3.3.3 Error Suggestion
• Test Details: Click the button and observe if any feedback is provided to indicate that the item has been added to the cart. This could be a message displayed on the page or a notification.

Conclusion

TestCraft is a really nice little addition to your QA toolbox. It can help you with ideas for testing when you feel stuck. It helps you with accessibility testing by checking the common compliance which can sure speed up your testing process. The automation feature is not something you should use without supervision and take it for granted. It is a tool for helping you with test automation, but some code modifications from your side are inevitable. To use this extension you still need to have a good understanding of the testing framework and programming skills. Your knowledge combined with TestCraft can improve your automation coverage. I believe it is a good choice in the realm of AI quality assurance tools.