WiredTiger uses a combination of several different tools and test programs for testing WiredTiger.

WiredTiger testing is done automatically using the Jenkins and Evergreen continuous integration testing frameworks. This allows us to be confident we don't introduce problems in a non-standard configuration, or performance regressions.

Unit Test Suite

Python

WiredTiger's primary functionality and regression testing is done in a Python unit test suite (found in the source tree under test/suite).

The WiredTiger Python test suite includes approximately 10,000 separate tests which are run on all platforms supported by WiredTiger. Each test is intended to test a single operation in a reproducible way, making it easy to diagnose errors. The test suite runs multiple test cases in parallel, allowing it to be run in a relatively short period of time.

The WiredTiger unit test suite includes tests that cover:

WiredTiger functionality (for example, cursors, transactions and recovery),
Combinations of WiredTiger's configuration settings and APIs,
Bug regression tests.

The WiredTiger Python test suite is built using the WiredTiger Python API and the Python unittest functionality (the test suite requires at least Python version 2.6).

The WiredTiger test suite automatically runs as part of every commit into the WiredTiger GitHub source tree.

CSuite

CPPSuite

Format

WT Perf

Workgen

Compatibility Testing

Correctness Testing

Performance Testing

Performance testing is primarily done using the bench/wtperf utility. A variety of database configurations are run based on the scripts in bench/wtperf/runners.

WiredTiger performance tests are automatically run as part of each commit into the develop branch of the WiredTiger GitHub source tree and compared against previous runs to detect performance regressions.

Performance testing is also done using the bench/workgen utility.

Stress Testing

Stress testing is primarily done using the test/format utility. This test program randomly configures a database and then runs some number of randomly selected operations, using some number of randomly selected threads, on that database. WiredTiger stress testing is run continuously on the WiredTiger GitHub develop branch.

Concurrency testing

Concurrency testing is primarily done using the test/format utility. Additionally, the test/thread and test/fops test utilities test specific heavily threaded operations. WiredTiger concurrency testing is run continuously on the WiredTiger GitHub develop branch.

Runtime correctness checking

ASAN, MSAN, UBSAN, Valgrind

Static analysis

WiredTiger static analysis is done using three tools:

The Coverity, Inc.software analysis tool; current results and historic defect reports are available at Coverity's WiredTiger page.
The Gimpel Software FlexeLint implementation of the UNIX lint tool.
The University of Illinois LLVM project's Clang Static Analyzer.

Coverage

Tests

The test coverage documentation is generated in each branch, the link provided is to the current develop version of the document.

Tagging scheme

Each test file shall contain a tag using the following scheme:

[TEST_TAGS]
<COMPONENT>:<TESTING_TYPE>:<TESTING_AREA>
[END_TAGS]

One test file can have multiple tags:

[TEST_TAGS]
backup:correctness:full_backup
checkpoints:correctness:checkpoint_data
caching_eviction:correctness:written_data
[END_TAGS]
`

If a test file shall be ignored, the following tag can be used:

[TEST_TAGS]
ignored_file
[END_TAGS]

Code

WiredTiger code coverage is measured by gcov.