Results can be viewed in one of two modes. Interim results can be viewed in the Execution view and completed results in the Results view.
Viewing Test Run Results
Within the Test Runs screen, clicking on an individual test run name, the test case results screen shown below is displayed.
Security test run and case results differ slightly from business process testing. To view Security results, please refer to the Security Test Runs section of this user guide.
The Execution status displayed in the test information row gives a high level overview of the status of the test. See the table below for an explanation of the various potential test statuses.
Setting Test Results
When hovering over a test case row in the test run results screen, users can change the Result Status of each individual test by clicking the appropriate button highlighted below. This functionality allows test results to be quickly changed at a glance. Drilling further into the test by clicking the Test Case ID (also highlighted below) will show the detailed test case view.
Note that if the Default Test Result in your tenant configuration is set to 'Execution Status', the tests will automatically be set according to whether a test Completed (Passed) or Failed to complete (Failed).
Filtering Test Results
Both the Execution Status and Result Status can be filtered on in the test run results screen, allowing users to only show the specific tests they wish to look at. Clicking the arrow buttons beside each of these headings will present users with a number of options to select.
Either or both of these filters can be used at same time to filter out only the tests runs that users wish to see in their test run results screen. Additionally the filter selections will persist, even when navigating to different areas in the Smart tenant (e.g. viewing a different test case to verify the results).
Please note that any filters will be cleared when logging out of the Smart tenant.
Detailed Test Case View
Selecting the business process link as highlighted above will display the detailed test case view shown below. This will allow the user to see each step in the business process that was encountered by Smart as the test was executed. Each step has a time stamp to display when it completed and which Workday user completed the step in the Person column. In addition to setting the test results in the test case results screen, the result for an individual test can be set on this screen.
When Smart encounters a non-fatal warning message in Workday, it will take a screenshot and make this available for viewing in the Screenshot column. Similarly, when a critical error in Workday is encountered, this causes the test to fail and provide a screenshot of the error encountered to the user. In addition to this visual feedback on issues encountered by your test, clicking the appropriate icon in the Status column (highlighted below) will provide further information regarding the error/warning. See the image below for a sample Hire that encountered a non-fatal warning in Workday.
In this case, the Hire failed to complete due to an error with an integrations that launches as part of the Hire Business Process in this particular tenant, but also encountered warning messages during the Initiation and Change Benefits for Life Event steps. The combination of three different types of visual feedback when it comes to error handling, enables users to easily troubleshoot their tests from within Smart, without needing to access their Workday tenant.
Test Case Comparison
Test case comparison is a powerful way to track change in your configuration. On re-execution, Smart will compare the Business Process steps from the previous execution with the current one. The comparison highlights any differences found making it easy to spot potential issues.
When you view a re-executed test case there will be a comparison result in the 'Test Case Details' section. The comparison will result in a:
- 'Match' when there are no differences found
- 'Non Match' when there are differences found
You can view the comparison in the tab labelled 'Steps Comparison'. This shows the step flows from the previous and current executions side by side. Smart makes the comparison easy by highlighting the steps that differ. These differences can be due to a change in:
- Process/step order
- Execution Status
- Error Message
Smart also identifies when there was no corresponding step in the flow. Adding a missing step marker labelled 'No step' makes it easy to spot differences in the flow.
Smart provides the user with the ability to view a summary of one or more test runs. From the Test Runs screen, click GENERATE SUMMARY REPORT. Users can then select their tests via the available radio buttons to the left of the test runs and click GENERATE REPORT. Doing this will bring the user to a page showing a summary of their selected test runs.
This feature can be used in conjunction with filters placed on the Test Runs page, allowing users to easily filter and then summarize only the test runs they wish to report on.
By clicking on one of the individual Business Process links, users can see a detailed explanation of the test results for the chosen process. The ability to export the test results as a CSV or PDF file is available by clicking the relevant hyper links at the bottom of the page.