This project is read-only.
Test Result Metrics
Status at a Glance
Automation Metrics
Test Case Metrics
Personal Metrics for Current Month

Metrics

Metrics are calculations derived from test case, test result, and/or user data. Managers and testers use these calculations to determine the state of an application, the efficiency of a test team, risk of release, and more. Crystal Test offers several built-in metrics.

Test Result Metrics

Crystal Test displays test result metrics in various places throughout the application. On the Dashboard, the Test Result Metrics section shows the number of Pass/Fail/Other results per browser. These numbers are affected by the filter dropdowns on the left column. This table only shows the metrics for the project-supported browsers. If a browser is added or removed, these metrics will change accordingly.

Test Result Metrics

Status at a Glance

The Test Case List page offers the Status at a Glance section. This section gives a concise view of the status of the current test effort. These metrics refer only to the test cases currently listed in the Test Case List and therefore are directly affected by the filter dropdowns on the left column. In the top section, the first value is the total percentage of all test cases that are passing. The second value is the percentage of test cases that are passing out of the ones that have been tested. The third value is the percentage of test cases that have been tested. The middle section is a bar chart that displays the number of test cases with a status of Pass/Fail/Other. In the bottom section, the first value shows the number of test cases currently displayed in the Test Case List. The second value displays the number of project-supported browsers. The third value displays the total number of tests to be tested. (NumberOfTestCases X NumberOfBrowsers)

Test Result Metrics

Automation Metrics

Automation metrics give us a good understanding of automation coverage and the amount of work to be done as well as the value of automation. Earlier, we discussed the automation fields of the test cases table in the Crystal Test database: autoTestClass, autoMetaDataTable, autoMetaDataRow, automated, and reasonForNotAutomated. These are the fields used to determine the values in the Automation Metrics table on the Dashboard.
Metric Algorithm
# of Test Cases that can be Automated in the Future (Automated_Future) The total number of test cases with the automated field set to ‘Future’
# of Test Cases that cannot be Automated (Automated_No) The total number of test cases with the automated field set to ‘No’
# of Test Cases already Directly Automated (Automated_Yes) The total number of test cases with the automated field set to ‘Yes’
# of Child Automated Test Cases (Automated_Child) The total number of test cases with the automated field set to ‘Child’
# of Test Cases not Evaluated for Automation The total number of test cases where the automated field is blank or null or is not Future, No, Yes, or Child.
# of Total Automated Test Cases (Automated_Total) The total number of test cases with the automated field set to ‘Yes’ or ‘Child’
# of Total Manual Test Cases (Manual_Total) The total number of test cases where the automated field is not Yes, or Child.
# of Total Test Cases (TestCases_Total) The total number of the test cases for the selected project.
% of Test Cases Currently Covered By Automation (Automated_Total * 100) / TestCases_Total
Maximum % of Test Cases That Can be Automated ((Automated_Future + Automated_Yes + Automated_Child) * 100) / TestCases_Total
% of Possible Automated Test Cases Completed (Work completed) ((Automated_Yes + Automated_Child) * 100) / (Automated_Future + Automated_Yes + Automated_Child)
Automated Test Scripts In Queue The total number of test results with a status of In Queue
Automated Test Scripts In Progress The total number of test results with a status of In Progress

Test Case Metrics

Test case metrics are based on test case activities, test case statuses, or things derived from test runs other than results such as number of defects found. The following test case metrics can be found on the dashboard:
  • Manual Cases Written
  • Automated Cases Written
  • Manual Cases Executed
  • Automated Cases Executed
  • Manual Defects Discovered
  • Automated Defects Discovered
  • Automated Tests in Progress
  • Automated Tests in Queue
  • Outdated Automated Scripts
  • Outdated Test Cases

Personal Metrics for Current Month

In addition to the dashboard metrics, a user can see their own personal monthly test case metrics in their profile. These numbers are derived from values created since the first day of the current month to present. This gives a user a good idea of their current productivity. They can see their lifetime productivity on the dashboard page in the Metrics by User section which is also filterable by date range.

Last edited Apr 2, 2014 at 8:54 PM by jacquelinewalton, version 2