Crystal Test is an open source Test Management System with a built-in test automation framework for Selenium and can be extended for use with other frameworks.

Crystal Test Features

  • 100% Free Test Management System and Automation Test Architecture
  • We took the leg-work out of automation and did it for you. All you need to do is set it up and start writing scripts.
  • Web-based system hosted on your servers for security and privacy
  • Run it local on your machine or publish the entire test grid. (Its already setup for you! You are up and running after a few configurations)
  • Open Source so you can add to it or change it to meet your companies needs.
  • Access maintained by Active Directory
  • Track manual and automated test results
  • Upload Excel sheets with test cases and/or results (No need to change the way your company is used to doing things)
  • Ability to manually enter test cases and test results via the UI
  • Export test cases to Excel
  • Assign test cases to yourself or others
  • Assign test cases to Sprints, Releases, Functional Groups
  • Personal Groups visible only to you
  • Personal Task Lists
  • Dashboard with Test Result metrics, Test Case metrics, and Automation metrics
  • Dashboard allows you to see test cases in progress, in queue, needing updated at first glance
  • Allows for unlimited number of projects
  • Each user can choose a default project and Crystal Test always opens to that projects data for that user.
  • Link with popular defect management applications such as Jira and TFS
  • Cross-browser testing
    • Chrome
    • Firefox
    • IE9
    • IE10
    • Safari
    • Mobile (Coming Soon)
  • Control your automation test grid from the Crystal Test GUI
    • Stop start your servers/VMs
    • View the Test Automation Engine, Selenium HUB, and Selenium Node consoles right from the Crystal Test GUI
  • Start/Stop an entire suite of automated tests with a single click
  • Innovative data-driven architecture for easy maintenance of automated scripts
  • Plenty of documentation to walk you through every step of setup and script writing

What is Crystal Test?

Crystal Test is a test management system created to normalize the way the Quality Assurance department writes, stores, and executes test cases and their results. With its included data-driven automation test architecture (D-DATA), this system will serve to exponentially reduce the time and cost of testing and enhance manual efforts via increasing test coverage and replacing manual and labor intensive tasks.

Crystal Test consists of 3 main components:
  • Website: The interactive web-based application that allows users to import, write, execute and view test cases and test results, view test metrics, and more.
  • Automation Infrastructure: The behind-the-scenes data-driven automated test architecture (D-DATA) that executes automated test scripts and stores their results.
  • Database: Contains both the data needed for the Crystal Test website as well as the data for D-DATA automated test scripts
All three are integrated but are built in such a way so they can be separated to be used with other systems.

The Test Management System

The Crystal Test frontend was initially implemented as a website. The website includes pages with forms for inserting, updating, or deleting test cases and test results. It supports both automated and manual test cases, with each being saved to the same tables, just with different flags to identify them. Tests can also be exported in bulk to Excel or likewise imported, to allow for populating test cases or test results in bulk. Other pages display summaries or reports by date, user, project, or whatever criteria are desired. Finally, the test case page is where automation tests are kicked off. As with other pages, numerous filters and sorting options are available. In particular, filters by test case Project, AUtomated/Manual, Functional Group, Release, Sprint, Environment, and Keyword have been implemented so far in the TMS. This page also displays most recent statuses per browser and when each test was last run to help provide users the information they may need when choosing what tests they should execute. Additional administrative options are available, for such things as aborting tests, restarting the selenium grid, viewing log files on selenium grid servers, etc.

D-DATA (Data-Driven Automation Test Architecture)

Within the Crystal Test database, a metadata table is created for each unique type of test; columns of these metadata tables represent configurable choices in the test, such as which option to choose for a radio button or dropdown, what text to enter in a field, how many times to iterate over a repeatable step, etc. A metadata table for a test like “request refund” might even reference rows in other metadata tables like “create user”, “purchase”, etc (each of which might also serve as a standalone test). Testing code has to be developed only once per table, then any number of rows can be created to satisfy various test conditions. These metadata rows are then linked to specific test cases so their results can be properly displayed.

While many systems claim to be data-driven, most of them merely allow for parameterization and require you to run all sets of parameters each time you run a test failing everything if an of them fail. Crystal Test not only uses the database tables for parameterization but for decision making as well, allowing one script to process a multitude of similar tests. Link each database row to a separate test case. This allows you to run some tests for a smoke test or perhaps all during a regression. You are in complete control of what you want to run and when.

The Automation Engine

The Automation Engine is where the code for the tests themselves resides, and this engine runs autonomously in the background, independently processing any requests for test execution that are inserted into the database. It starts by scanning the database for test cases with status of “In Queue”, and if it has available servers in its selenium test grid, it flags those test case(s) as “In Progress” as it begins its work. Based on the criteria defined in the database record, one or more threads might be kicked off to support the browsers being tested in simultaneous tests. Depending on how the test fares, the final result might be pass or fail, but that’s just the tip of the iceberg. The system also takes screenshots, saves them to the server with unique names, and stores the location of the saved file in the results. The engine stores additional relevant data in the results; for example if the test created a new user and performed a purchase, then it might store the user credentials and other identifiers for what they purchased. This way if there’s ever a problem or a concern about the results, you can always look up that user, their purchase, etc., in the target system to verify they were created properly.

Test Results

Crystal Test stores test results with very granular detail; each combination of project, test case, browser, environment, etc. are stored as separate result records, and all historical results are also stored for future analysis. Not only are simple results like pass/fail stored, but also screenshots, detailed error messages, and custom output text (e.g. what unique email address was used for that registration test). To help easily parse through all this historical data, views are created for such things as the latest test result record in each browser and environment. The database also tracks tests in progress and in queue, allowing for such functionality as the displaying of debugging messages for a test in progress.

Child Test Cases (Reduce redundant code)

Child test cases are also supported. Running a test script will often cover more than one test case. Crystal Test allows you to assign child test cases to be marked as passed when its parent passes.


The frontend of Crystal Test could take many forms - For example, a continuous integration system might directly insert “In Queue” records into the database, skipping over the UI altogether. Multiple different frontends could even be implemented that all feed into the same backend through the database. Likewise multiple backend applications tie to the same database, one for handling Selenium tests, another for handling SoapUI tests, REST APIs, etc. The frontend(s) are not programmatically tied to the automation engine backend(s), each is developed separately, allowing for a more flexible and scalable system. If desired, the automation engine could be developed in Java while the frontend is in ASP .NET; they don’t need to interact with each other directly, only through their respective interactions with the shared database. (Perhaps a developer may want to contribute a JAVA version of the Automation Engine)

Cant decide if Crystal Test is right for you?

Read these articles comparing Crystal Test to other popular automation tools:
Crystal Test vs Microsoft Test Management
Crystal Test vs JUnit
Crystal Test vs QTP Coming Soon

Want to be a Crystal Test Developer or Contributor?

Email Jacqueline Walton for more information.


QAI Conference 2015

Monday, April 20: 8:30 AM – 4:30 PM | Format: One-Day Class | Focus: Automation Automation Architecture - A Blueprint to Success, Instructor, Jacqueline Walton

Thursday, April 23: 1:00 PM - 2:30 PM | Format: Workshop | Focus: Automation Quest Conference 2015 - 3 Steps to a Maintainable Automated Test Suite, Instructor, Jacqueline Walton

The Legal Stuff

Crystal Test is copyright 2013-2015 Pixeltrix, LLC. If you want to use, copy, modify or distribute Crystal Test, you may do so under the terms of the GNU General Public License as long as you include this copyright notice with any and all distributions. You may not remove the Powered By Crystal Test notification in the footer of Crystal Test.

Last edited Mar 9, 2015 at 5:03 AM by jacquelinewalton, version 31