Test Run Re-execution (Individual Test Cases)
Smart allows users to easily re-execute Test Runs with only a few button clicks. In situations where a test has unexpectedly failed, or users would like to update data for a failed test, it is a simple process to re-execute a test case or a number of test cases to ensure they complete as expected.
Test Run Re-execution
Clicking the Related Actions "..." button beside the Test Run will present a list of the available actions that can be carried out against the Test Run. These actions change depending on the state of the test run. For further information on the Test Run screen, please refer to the Test Run States & Actions section of the user guide.
This section of the User Guide will focus on the Re -execute Related Action for Completed or Failed Test Runs.
Re-execute Test Run
After selecting to Re-execute the test, the user is prompted to select a Workday tenant to execute the Test Run; please see below. Any successfully configured Workday tenants will be available for selection. Users will also have the option to select from one of two options:
- All - Re-execute all tests in the Test Run, regardless of whether they passed or failed.
- Individual test cases - Choose which test cases to re-execute from any given Test Run. Please see the section below on Individual test case re-execution.
Users also have the choice to update their data templates before re-execution. Please see the section below on Re-execution with updated templates.
The ability to specify exactly which test cases are required for re-execution, combined with the ability to upload templates containing updated test data, as well as additional/fewer tests if the test plan/scope has changed, allows users to have complete control over the updating and execution of their Smart test packs.
As with the initial execution, users can specify whether to run the test Now or Later. Selecting Later will show a date and time selection option to allow users to specify exactly when to launch their tests.
Tags can also be added during the re-execution of a Test Run in the box highlighted below.
The Retrieve latest baseline results checkbox will show for Security Tests and allow the test to be re-executed with baselined test results.
Select the Confirm Re-Execution button to re-execute the Test Run now or schedule to run at a future date.
Individual test case re-execution
Smart provides users the ability to choose which test cases they would like to run on re-execution, regardless of their status. In addition to assisting in re-execution of tests that may have failed due to changes detected in Workday, it also gives more granular control over testing, assisting in the running of specific tests when it comes to change management procedures, or choosing to only run certain tests from a regression pack each week.
From the re-execution screen above, selecting the Individual test cases option will present users with a new selection menu. From this menu, users can now select which tests they would like to execute.
Users have a number of features available to them to assist in selecting the required test cases, highlighted in the image above:
- Radio Buttons - Beside each test case there is a check box. Check the box to include the test case in the next re-execution. Alternatively, there is a mass-select check box at the top of each test page, used to select all tests on the current page.
- Multiple Pages - Depending on the volume of test cases, there may be multiple pages to navigate between. To assist in this, users can specify how many test cases to view per page by changing the Rows per page value to either 10 (default), 20 or 50. Check box values persist while navigating between pages using the < or > buttons, allowing users to select any combination of tests to execute.
- Filters - A Filter test cases... field at the top of the menu will allow users to filter on values used by any of the test cases. Only one filter can be applied at any one time, however any check box selected while a filter is applied will still be selected when the filter is removed.
When using filters as part of individual test case re-execution, the filters field will look at every column across all available test cases. This allows the user freedom to apply basic filters such as "Passed" or "Failed", or to only show specific business processes by filtering on the business process name.
Once users have specified their required filter, either pressing the enter key or clicking APPLY will enable the filter. After the required test cases are selected, a new filter can be specified and applied (note that the filters do not accumulate on eachother) by simply entering it into the field and pressing enter or APPLY. To remove the last filter applied, select the Clear filter option.
Re-execution with updated templates
Users are also able to make changes to their data templates as part of a re-execution.
If the data that drives test scenarios changes or the scope of the test is revised, users can upload a new template when re-executing a test run. In capturing these changes during the re-execution process, users can avail of the execution history and test case comparison functionality.
When re-executing a Test Run using an updated template, there are currently some restrictions in place:
- Number of Test Runs - The number of Test Runs in a template must remain the same. Adding or removing tests from a template and attempting to upload it to a re-executed Test Run will cause an error informing users that the number of test cases cannot be changed.
- Test Case IDs - Currently, the Test Case IDs in a template must remain the same. Changing the Test Case ID in a template will cause an error informing users that the Test Case IDs do not match the existing test data.
Future enhancements to the re-execute functionality will allow the addition/removal of Test Cases when re-executing.
Re-executed Test Run example
The following Edit Service Dates Test Run has completed execution, the test results have been set, and there are examples of failing tests:
Checking the individual Test Case view, the error message thrown shows that there is a data issue with the template:
Checking the data template itself, the issue is clear. A small data entry issue with the Continuous Service Date field has caused the failure:
The template is amended to fix the issue with the date (in this example, changing 20101/20001 to 2010/2000 respectively) and the template is now ready to be uploaded into the Test Run. Click the RE-EXECUTE button in the Test Run view to initiate the Re-execution screen:
After selecting to execute Individual test cases, select the tests that require re-execution using the radio buttons. Finally, selecting Yes under 'Do you want to update your data templates?' will prompt users to change the data template as required:
On selecting Change the updated data template can then be chosen and uploaded to the Test Run. If for any reason the change needs to be undone, there is a Revert button available which will roll the data template back to before it was changed:
The remaining options available for re-execution can then be set and the Test Run can be re-executed. Checking the Test Run results during execution shows only the failed tests being re-executed:
When users are uploading updated templates to a Test Run re-execution, the template must not contain any blank test name fields. If a template contains data in a column, however there is no name/description for the test (this is entered in the field above the Test Case ID), Smart will not allow this to be uploaded to the Test Run. If Smart detects an empty name/description field, the following validation will be presented:
Additionally, Test Case IDs must be unique within a Test Run. For more information on this, see the section on Unique Test Case IDs.
Updating Data - Flowchart
The following flowchart depicts the typical steps taken when re-executing a Test Run with an updated template to resolve failed tests.
Test Status & Viewing Progress
Once the test has been executed, the user will be returned to the Test Runs screen which will display the status of the test. At this point the user can drill into the test run and view detailed information for the test cases. Click on the name of the test run to view the Test Run results page. This page shows the following information:
- Test Run name
- The user who executed the Test Run
- The status of the Test Run
- The Start date and Execution time (the Execution time is test execution duration value which is populated once the Test Run has finished)
- The tenant that the tests ran against
- Execution view, displaying the current status of all test cases within the Test Run
- Results view, displaying the results of all test cases
- Previous test run, which links to the previous execution of the test run selected
Test Runs can also be Re-executed from within the Test Status view, via a RE-EXECUTE button located above your test status summary.