ServiceNow Automated Test Framework – The Tool for UX Testing

Introduction to ATF

The ServiceNow Automated Test Framework (ATF) was introduced in the Istanbul release, and with each new major release it received some additional features, including Service Portal support. It is ServiceNow’s own solution for doing regression testing on user experience (UX) and user interface (UI) functionality. The basic idea is to create test suites for processes that do not change too often to keep test case maintenance minimal.

ATF is used for testing via the ServiceNow UI, and an open browser session is required for its Client Test Runner. All major browsers are supported – even Edge and, to some extent, Internet Explorer – but in my experience, these browsers sometimes have memory issues if used for longer tests.

ATF Structure Overview

The image below is only an example; the structure can be unique for each customer depending on their needs. The image shows a list of complex test suites grouped by processes, for example Incident Management, Change Management, etc. While it can be fairly simple to implement an Incident test suite, Change Management is an entirely different setup which should in most cases be broken down into smaller pieces.

If you would like to use smaller tests, check the details about planning test suites below for smaller versus longer tests.

Planning Test Suites

There are two approaches to creating tests:

  • Individual, smaller tests, such as the out-of-the box quick-start tests available starting from the Madrid release.
  • Longer, complex tests.

Smaller tests can be very handy to quickly test a specific part of the UI/process to pinpoint issues, and the maintenance of test cases is also much easier. Some tests will require pre-created records to be able to succeed which will have to remain to keep the test from failing. You could generate test data using scripts, but that might not produce the same result as generating test data manually would.

Longer tests can be more time-consuming to implement and have a higher chance of failing, so it is usually better to create smaller test cases. However, in some cases end-to-end process testing is the best choice to ensure that the whole process is working as expected, as this option gives the opportunity to follow and verify the workflow very thoroughly.

Scheduling

It is possible to schedule tests so that they become automated and not only an ordinary Test Framework. Scheduling is similar to scheduled jobs, so you can specify a scripted condition and add multiple test suites if needed. This would require a computer with an active browser session, with ATF tests running in ServiceNow with the Scheduled Test Runner open. ATF can eat up a lot of memory over time, so it might be a good idea to reload the browser from time to time.

Server-Side Scripting

While ATF is mainly used for UI (e.g. client side) testing, it can also be useful for testing server-side scripts, for example to find a record, evaluate some inputs from the records created, etc. The only way to reference a previous step in a script is to hardcode its sys_id into the script (which is bad practice, by the way, and I do not endorse it, but there seems to be no way around it) which you can see below:

(function(outputs, steps, stepResult, assertEqual) {

    var formSubmittedStep = "0aef45580fae0300b11cf68ce1050ea8";

    var changeTaskGr = new GlideRecord('change_task');
    changeTaskGr.addQuery("change_request",steps(formSubmittedStep).record_id);
    changeTaskGr.addActiveQuery();
    changeTaskGr.query();

    if (changeTaskGr.next()) {
        outputs.table = 'change_task';
        outputs.record_id = changeTaskGr.sys_id;
        stepResult.setOutputMessage("Successfully found change task record");
        return true; // pass the step
    } else { 
        stepResult.setOutputMessage("Failed to find change task record");
        return false; // fail the step
    }

})(outputs, steps, stepResult, assertEqual);

New in the Madrid Release

Quick-Start Tests

Most likely the best new ATF feature of all is the quick-start tests. These provide out-of-the-box test suites with a variety of predefined test scenarios that can be copied and customised or just left as is if they meet requirements. They can be used as a baseline for creating your own test suites in general and/or to do regression testing after upgrades to find misbehaving processes faster.

Using quick-start tests can especially be useful for Change Management, Incident Management, Customer Service Management and Knowledge Management, but there are several other smaller suites for various modules. See more on the ServiceNow Docs page.

Parameterised Testing

Until the Madrid release, you had to work with one dataset, but now you can specify parameters. They can be either shared so that other tests can reuse them or exclusive so they are only accessible for the current test. This could be used to intentionally fail some tests (meaning the test fails when it is completed successfully), to cover more scenarios or to simply run the same tests multiple times.

Custom UI Testing

In the latest release, it is now also possible to test UI pages and UI macros with the help of the Manual Page Inspector. This can be useful for legacy pages to ensure they work after each upgrade.

Challenges of ATF

Record Retention

Unfortunately, one of the biggest pain points of ATF is that it requires dummy test records on sub-production instances. Those need to be configured with data preservation rules to stay in place during a clone. Also, as test data generated by ATF is deleted after every test, investigating errors that interrupt or hang tests can be challenging.

The only part you can make sure will stay is the test results. Those can provide useful insights such as screenshots on failure and JavaScript logs to solve a problem, but not being able to check related records due to test record deletion can sometimes make the investigation of related defects more difficult.

Integration Testing

If there is a stable delay in communication between the two systems, you can use gs.sleep(), but that is not always recommended as it will not always provide a reliable result. Also, it would make the whole test run slower.

There are some assert type tests to check the response time and verify JSON/XML payload, and it is possible to do inbound REST testing, but if something changes on the other side you would not know based on that test.

Custom Service Portal Testing

Currently, custom widget testing requires you to insert code taken from out-of-the-box widgets. A portal form page and catalog page can be used to access a record form or catalog item or record producer that can work for some simpler implementations.

Complex Tests

Complex processes take a longer time to run and have a high chance of exiting with an error that is unrelated to the process itself. Maintenance of those complex test can also be time consuming. This might not be an issue for everyone, but if you can, it is recommended to keep the test below 50 test steps.

Future Potential

ServiceNow could improve the reliability of tests by having the option of manual timeouts or slowing down the tests between steps to ensure that all the client-side scripts are loaded and executed properly. This has improved in Madrid already, so the tests wait for pages to be fully loaded, but it would be a good idea to make this customisable depending on customer needs.

The option to have record retention on a test suite basis would be a tremendous help with debugging tests. This would also open up the possibility to allow one test to pass its outputs to the next test.

Should ATF Be Used?

The short answer is that in most cases, yes. Before an ATF implementation, I would suggest investigating the current opportunities with its capabilities in mind. If you plan to upgrade to Madrid or if you are already on Madrid, check the quick-start tests and see if you can make those work. Either way, I would break up conventional test spreadsheets into smaller, individual tests that can be run on their own and then create some records as a starting point for the tests. In essence, ATF is already useful, and because ServiceNow improves it with every release, I am confident that it will only become more useful in the future.

About the Author

Mate Lukacs is a ServiceNow Architect who has successfully completed over 25 ServiceNow projects in multinational teams for customers from numerous EU countries.

Updates

Blog