Statuses and results

Below are the possible statuses and descriptions for suite, job and test executions.

Note that every API method responds with the standard HTTP error codes (200, 300, 400…) but also includes a response body for errors. Within the error body, it is possible to find the list of errors in a human-readable manner together with an internal application error code that can help Quali when debugging an issue. For example, trying to stop an already finished suite will result in a 400 HTTP error code with the following JSON response body:

{
   "errors":[
      {
         "message":"Suite execution already completed",
         "name":"SuiteExecutionCompleted",
         "code":1006
      }
   ]
}

In the above response, 1006 is the internal application error code for the specific use case and the message is the human-readable error description

Suite statuses

API status Status Description
0 Not started Triggered. None of the suite’s jobs have started running, waiting for either slot or sandbox.
1 Running At least one job started running.
2 Succeeded All tests in all jobs have executed with test result either COMPLETED or PASSED.
3 Failed All tests in all jobs have executed with at least 1 test with test result FAILED.
4 Terminated At least one test has test result TERMINATED.
5 Automatically Cancelled Option 1 - sandbox error (active with error, didn't find a sandbox in the retries). Option 2 - no test slot found
6 Manually Cancelled Suite execution was cancelled manually.

Job statuses

API status Status Description
0 Not started Triggered. Job has not started running, waiting for either slot or sandbox.
1 Running Test execution is in progress (test has not failed/terminated).
2 Succeeded All tests have been executed with COMPLETED or PASSED result.
3 Failed All tests have been executed. At least 1 test with result FAILED.
4 Terminated At least 1 test with result TERMINATED
5 Automatically Cancelled

Job was automatically cancelled.

Possible reasons are:

  • No available slots for the job's execution

  • Teardown started while tests are running

  • Job details:

  • Job's blueprint had been deleted or modified
  • Job's test(s) had been deleted or modified

  • Sandbox errors:

  • Sandbox creation took too long (configurable value ~35 minutes)
  • Sandbox cannot be used (has errors or failed to be created)

  • Connection errors:

  • Test Execution Service was unable to load blueprint details due to Quali Server connection failure
  • Job Scheduling Service was unable to send tests to Test Execution Service for execution
  • Job Scheduling Service was unable to publish sandbox to RabbitMQ
6 Manually Cancelled The job was cancelled manually while in progress (Not Started, Running).
10 Waiting For Job Slot Job is waiting for a suitable test execution server slot to become available
11 Waiting For Sandbox Job is waiting for sandbox to become Active

Test results

API result Result Description
0 Completed Test finished execution.
1 Passed Test completed successfully.
2 Failed

An error during the test’s execution caused it to fail.

Note: Test execution failure does not stop the job from running.
3 Terminated* Test execution was stopped by the user or job status is "Automatically Cancelled".
4 FailedToStart* Job triggered. Test failed to execute.

* Test execution failure does not stop the job from running its other tests.