Viewing Results

Results can be viewed in one of two modes. Interim results can be viewed in the Execution view and completed results in the Results view.

Viewing Test Run Results

Within the Test Runs screen, clicking on an individual test run name, the test case results screen shown below is displayed.

Security test run and case results differ slightly from business process testing. To view Security results, please refer to the Security Test Runs section of this user guide.

Execution Status

The Execution status displayed in the test information row gives a high level overview of the status of the test. See the table below for an explanation of the various potential test statuses.

Execution Status

Setting Test Results

When hovering over a test case row in the test run results screen, users can change the Result Status of each individual test by clicking the appropriate button highlighted below. This functionality allows test results to be quickly changed at a glance. Drilling further into the test by clicking the Test Case ID (also highlighted below) will show the detailed test case view.

Note that if the Default Test Result in your tenant configuration is set to 'Execution Status', the tests will automatically be set according to whether a test Completed (Passed) or Failed to complete (Failed).

Filtering Test Results

Both the Execution Status and Result Status can be filtered on in the test run results screen, allowing users to only show the specific tests they wish to look at. Clicking the arrow buttons beside each of these headings will present users with a number of options to select.

Either or both of these filters can be used at same time to filter out only the tests runs that users wish to see in their test run results screen. Additionally the filter selections will persist, even when navigating to different areas in the Smart tenant (e.g. viewing a different test case to verify the results).

Please note that any filters will be cleared when logging out of the Smart tenant.

Detailed Test Case View

Selecting the business process link as highlighted above will display the detailed test case view shown below. This will allow the user to see each step in the business process that was encountered by Smart as the test was executed. Each step has a time stamp to display when it completed and which Workday user completed the step in the Person column. In addition to setting the test results in the test case results screen, the result for an individual test can be set on this screen.

When Smart encounters a non-fatal warning message in Workday, it will take a screenshot and make this available for viewing in the Screenshot column. Similarly, when a critical error in Workday is encountered, this causes the test to fail and provide a screenshot of the error encountered to the user. In addition to this visual feedback on issues encountered by your test, clicking the appropriate icon in the Status column (highlighted below) will provide further information regarding the error/warning. See the image below for a sample Hire that encountered a non-fatal warning in Workday.

In this case, the Hire failed to complete due to an error with an integrations that launches as part of the Hire Business Process in this particular tenant, but also encountered warning messages during the Initiation and Change Benefits for Life Event steps. The combination of three different types of visual feedback when it comes to error handling, enables users to easily troubleshoot their tests from within Smart, without needing to access their Workday tenant.

Test Case Comparison

Smart provides the user with the ability to compare Business Process test cases during the re-execution of a test run. When BP tests are re-executed, Smart will automatically display a comparison view in the test case results.

This is especially useful in regards to BP test packs that are run on a regular basis, as the user can easily see if configuration has changed since the previous run of the test.  

To do this navigate to the completed test run in the TEST RUNS screen and select the re-execute button beside the chosen test run, as seen above. The user will then be prompted to choose a tenant to run the test against, when to execute the Test Run (now or later), and the option to set the test case results automatically. This option is checked by default.

Upon re-execution of the test, users can then go into the detailed view of the test case and there will be two extra tabs visible. Standard View, which shows the results of the current test case, and Comparison View, which shows the results of the current test case alongside the results of the test case you chose to re-execute. This allows users to compare the two test runs side by side, providing another way to identify potential issues that are causing tests to fail, and also allowing for visibility of any changes in the overall Business Process.

Additionally, a Comparison Result field will immediately inform users if there are any differences between the initial test case and the re-executed test case. Smart will also highlight any steps that differ between the two test runs.

See the image below for an example of the Comparison View where the two test runs do not match.

Comparison View

Additional Reporting

Smart provides the user with the ability to view a summary of one or more test runs. From the Test Runs screen, click GENERATE SUMMARY REPORT. Users can then select their tests via the available radio buttons to the left of the test runs and click GENERATE REPORT. Doing this will bring the user to a page showing a summary of their selected test runs.

This feature can be used in conjunction with filters placed on the Test Runs page, allowing users to easily filter and then summarize only the test runs they wish to report on.

By clicking on one of the individual Business Process links, users can see a detailed explanation of the test results for the chosen process. The ability to export the test results as a CSV or PDF file is available by clicking the relevant hyper links at the bottom of the page.