Test Search API

Note: This script requires an API Token to run.

Simscope includes a script simscope-test-search to search tests from the command-line (or from API).

This is orthogonal to searching by Signature.

For example, users can see all tests:

Run in the past 7 days for a component/branch
Or from a regression
Or filtered by config/stimulus (regex)
Or filtered by build (regex)
Or filtered by error message (regex)
Or for specific signatures

Output

The script will dump a CSV file containing each of the tests matching the query, which can be analyzed via script or viewed in Excel.

Script Usage

> bin/simscope-test-search --help
Options:
  -h, --help            show this help message and exit
  -v, --verbose         Verbose logging mode
  --addr=ADDR           Simscope address: hostname:port
  --apitoken=APITOKEN   Simscope API token
  --date-low=DATE_LOW   Low Date (YYYY-MM-DD format)
  --date-high=DATE_HIGH
                        High Date (YYYY-MM-DD format)
  --days=DAYS           Number of days
  --limit=LIMIT         Result limit
  -o OUTPUT, --output=OUTPUT
                        Output CSV to a file (default is stdout)

  Test Search fields:
    --component=COMPONENT
                        Component filter
    --category=CATEGORY
                        Signature category filter
    --regr=REGR         Regression filter
    --sigid=SIGID       Signature ID filter
    --branch=BRANCH     Branch filter
    --build=BUILD       Test build filter (regex)
    --config=CONFIG     Test config filter (regex)
    --group=GROUP       Test group filter (regex)
    --message=MESSAGE   Test error message filter (regex)
    --result=RESULT     Filter by result status (pass or fail)
    --classifications=CLASSIFICATIONS
                        Signature classification filter
    --regex             Enable regex matching mode
    --search=SEARCH     Filter by job fields or metadata (e.g. "cycles>0"
    --show-message      Show the binned signature title, rather than the raw
                        error message
    --show-regr-command
                        Show the regression command
    --metadata=METADATA
                        Additional job metadata to retrieve for the CSV
    --shortest-sigs=SHORTEST_SIGS
                        Grab the N shortest jobs from each signature
    --sort=SORT         Sort the results

Regression Example

This example dumps all tests in a single regression as CSV:

> bin/simscope-test-search --regr=myregr/123
component,regr,ymd,sigid,jobid,branch,model,build,config,runtime-ms,cycles,message
simscope,16290x,2017-09-18,836,bfug8ryuwzhi,default,16290x,,make -j1 test_zen test_uhttp test_rest
 test_simscope test_verops,20442,0,--- FAIL: TestLotsOfRegressions (3.29s)
simscope,16290x,2017-09-18,3,bfug8r69iijp,default,16290x,,make -j1 lint vet,1728,0,

Job Metadata

Note: the message field is hidden by default. To see the message field, add the flag --show-message.

Simscope can dump any additional metadata fields into the CSV. Use the --metadata argument for each field.

Note: adding metadata will slow down the search dramatically, as this performs a table join.

Metadata Example

> bin/simscope-test-search --component=simscope --branch=default --days=1 --metadata=testgroup --metadata=host
component,regr,ymd,sigid,jobid,branch,model,build,config,runtime-ms,cycles,message,testgroup,host
simscope,16308x,2017-09-20,3,bfvzf77zd0il,default,16308x,,make -j1 lint vet,2025,0,,,my-host-machine

Metadata Fields

There are many available metadata fields:

issue
state
assignee
classification
designation
category
host
branch
or any custom job fields

Filtering by Job Fields/Metadata

To filter by job field/metadata, use the --search parameter.

This supports the standard job comparison operators

=
!=
>
<
=~
etc.

Example job search filter:

> bin/simscope-test-search --component=simscope --branch=default --days=1 --search=testgroup=abc
component,regr,ymd,sigid,jobid,branch,model,build,config,runtime-ms,cycles,message,testgroup
simscope,16308x,2017-09-20,3,bfvzf77zd0il,default,16308x,,make -j1 lint vet,2025,0,,abc-group

You can join multiple terms with commas (e.g. cycles>0,build=~vcs).

Simscope Knowledge Base