Viewing Results

Viewing Test Run Results

You can view the 'Test Run Details' page by clicking on a the name of test run anywhere in the application. The page contains the following sections:

Security test run and case results differ slightly from business process testing. To view Security results, please refer to the Viewing Security Results section of this user guide.

Execution Details

This shows the key information relating to the execution. This includes who executed it, when it was executed and how long it took.

Results Summary

This provides a visualisation of the pass percentage as well as the numbers of passes and fails.

Test Cases

Lists all the test cases in the execution including some key information. Importantly, it also shows the execution status and test case result. Clicking on the test case ID will take you to the 'Detailed Test Case view'.

Filtering

You can filter the test cases using the following columns:

  • Test Case ID
  • Execution Status
  • Results Status

You can user filters to pinpoint a test case, or look for test cases with a specific execution or result status. Click on the arrow beside the column header to open the respective filter.The test case ID filter is a free text search.

For execution status and result filters you click the checkbox for the options you want. The filters will work in conjunction with one another. For example you can search for 'Warning' executions status that have a result of 'Passed'.

Execution Status

A test case can have one of 6 execution statuses.

  • Queued: Test is waiting to execute
  • In Progress: Test is currently running
  • Completed: Test has successfully ran from start to finish
  • Completed with Warnings: Test has successfully ran but encountered some alert messages
  • Failed to Complete: Test encountered an issue that stopped it from completing
  • Cancelled: Execution of the test has been stopped.

 

Result Status

There are 4 results available in Smart:

  • Blocked: Unable to determine test result
  • Failed: Test case failed
  • Passed: Test Case Passed
  • Unspecified: Test result has not been se

Setting Test Results

You can set the result for a test case by hovering over the result badge. This will reveal the selectable results, the underline indicating the existing result.

The related actions button will also allow you to set the result.

If you have configured the 'Default Test Result' in your tenant to be 'Execution status', results will be set as follows:

  • 'Completed' will return a 'Passed' result
  • 'Failed to Complete' will return a 'Failed' result

Chained Test Case Results

A chained test case that fails to complete will impact the execution of subsequent chained test cases. When a 'parent' test case fails the remaining tests in the chain have their execution cancelled. This is because they have no chance of passing without the data they need. The result for these cancelled test cases is 'Blocked'.

Clicking the 'cancelled' execution status will provide details about the cancellation/blocked result. This makes it easier to find out what has gone wrong and where.

Detailed Test Case View

You can view the test case view by clicking a 'Test Case ID' link. This page contains:

  • Test Case Details
  • Steps ComparisonTab (if previously executed)
  • Steps Tab
  • Generated Test Data Tab (if required)
  • Test Input Data Tab

 

Test Case Details

This section provides some identifiable test case information. It also tells you the result of the test case and its comparison status if it was executed before. You can also change the result here by clicking the 'Change Result' link.

 

Steps Comparison Tab

You can see the steps in the business process Smart encountered during execution for both the current previous execution. It also compares the steps and highlights any differences found. See Test Case Comparison for more information.

 

Steps Tab

You can see the steps in the business process Smart encountered during execution. Each step is time stamped at completion and you can see which Workday user (Person) completed it. Underlined execution statuses are clickable, showing warning/error information about the step. Smart also captures screens shots when errors occur, these are also viewable on click. This level of feedback enables you to troubleshoot tests within Smart.

 

Generated Test Data Tab

Shows the test data Smart has generated as part of the execution.

Test Input Data Tab

This shows the data from the uploaded template used to drive the test

Test Case Comparison

Test case comparison is a powerful way to track change in your configuration. On re-execution, Smart will compare the Business Process steps from the previous execution with the current one. The comparison highlights any differences found making it easy to spot potential issues.

When you view a re-executed test case there will be a comparison result in the 'Test Case Details' section. The comparison will result in a:

  • 'Match' when there are no differences found
  • 'Non Match' when there are differences found

You can view the comparison in the tab labelled 'Steps Comparison'. This shows the step flows from the previous and current executions side by side. Smart makes the comparison easy by highlighting the steps that differ. These differences can be due to a change in:

  • Process/step order
  • Execution Status
  • Error Message

Smart also identifies when there was no corresponding step in the flow. Adding a missing step marker labelled 'No step' makes it easy to spot differences in the flow.

Additional Reporting

Smart provides the user with the ability to view a summary of one or more test runs. From the Test Runs screen, click GENERATE SUMMARY REPORT. Users can then select their tests via the available radio buttons to the left of the test runs and click GENERATE REPORT. Doing this will bring the user to a page showing a summary of their selected test runs.

This feature can be used in conjunction with filters placed on the Test Runs page, allowing users to easily filter and then summarize only the test runs they wish to report on.

By clicking on one of the individual Business Process links, users can see a detailed explanation of the test results for the chosen process. The ability to export the test results as a CSV or PDF file is available by clicking the relevant hyper links at the bottom of the page.