Regressions
Regressions are groups of Jobs, launched within a single context.
Regressions can contain any type of Job Categorized with a pass/fail result:
- Compilations (Builds)
- Test generation
- Simulations (tests)
- And can contain either passing or failing jobs.
Regressions are generally launched automatically with an external Regression tool, and imported into Simscope for analysis.
Regression Context
Within a single regression, all jobs must share the same Context:
| Regression Field | Example |
|---|---|
| Regression Name | sanity_regr/4 |
| Component | compress |
| Project | rhino |
| Model Branch | b2 |
| Model Version (source control) | f81249a |
Regression Example
Regression compress_random/412 is run with component compress, on the branch
testing, with Git SHA f81249a, and contains:
- 1200 jobs
- 2 pass compiles
- 901 pass simulations
- 297 fail simulations
Categorization
Simscope categorizes results by Signature, which allows filtering and statistic tracking based on job types.
- An example categorization grouping is simulations vs test generations.
Regressions are allowed to have multiple categories within a single regression.
The following example includes compile, gen, and sim Jobs and Scores:
Statistics
Simscope collects aggregate statistics on each regression, which can be used to determine the health of the regression:
- Score (pass/fail rate)
- Total Cycles
- Total Runtime
- Average CPS (cycles per second)
- Duration (wallclock start-to-finish)
- Number of signatures
Live Regressions
Simscope can track both completed regressions, and LIVE (incomplete) regressions.
→ See Live Regressions on the Administration Guide
Compare/Diff Regressions (Comparisons)
→ Simscope can Compare two or more Regressions.
Tanking Regressions
Managers and Administrators can Tank bad Regressions.
- This is equivalent to killing or hiding a regression.