Statuses and results

Below are the possible statuses and descriptions for suite, job and test executions.

Note that every API method responds with the standard HTTP error codes (200, 300, 400…) but also includes a response body for errors. Within the error body, it is possible to find the list of errors in a human-readable manner together with an internal application error code that can help Quali when debugging an issue. For example, trying to stop an already finished suite will result in a 400 HTTP error code with the following JSON response body:

{
   "errors":[
      {
         "message":"Suite execution already completed",
         "name":"SuiteExecutionCompleted",
         "code":1006
      }
   ]
}

In the above response, 1006 is the internal application error code for the specific use case and the message is the human-readable error description

Suite statuses

API status Status Description
0 Not started Triggered. None of the suite’s jobs have started running, waiting for either slot or sandbox.
1 Running At least one job started running.
2 Succeeded All tests in all jobs have executed with test result either COMPLETED or PASSED.
3 Failed All tests in all jobs have executed with at least 1 test with test result FAILED.
4 Terminated At least one test has test result TERMINATED.
5 Automatically Cancelled Option 1 - sandbox error (active with error, didn't find a sandbox in the retries). Option 2 - no test slot found
6 Manually Cancelled Suite execution was cancelled manually.

Job statuses

API status Status Description
0 Not started Triggered. Job has not started running, waiting for either slot or sandbox.
1 Running Test execution is in progress (test has not failed/terminated).
2 Succeeded All tests have been executed with COMPLETED or PASSED result.
3 Failed All tests have been executed. At least 1 test with result FAILED.
4 Terminated At least 1 test with result TERMINATED
5 Automatically Cancelled

Job was automatically cancelled.

Possible reasons are:

  • Validation of job test data failed

  • Max retry count limit exceeded when trying to find an available slot

  • Max retry time limit exceeded when trying to find an available slot

  • Teardown started while tests are running

  • Job details:

  • Job's blueprint was deleted or modified in the online repository
  • CloudShell detected that the job's test(s) was deleted or modified
  • Job duration was reached while some of the job’s tests did not execute

  • Sandbox errors:

  • Sandbox creation took too long (configurable value ~35 minutes)
  • Sandbox cannot be used (has errors or failed to be created)

  • Connection errors:

  • Test Execution Service was unable to load blueprint details due to Quali Server connection failure
  • Job Scheduling Service was unable to send tests to Test Execution Service for execution
  • Job Scheduling Service was unable to publish the sandbox request to RabbitMQ
6 Manually Cancelled The job was cancelled manually while in progress (Not Started, Running).
10 Waiting For Job Slot Job is waiting for a suitable test execution server slot to become available
11 Waiting For Sandbox Job is waiting for sandbox to become Active

Test results

API result Result Description
0 Completed Test finished execution.
1 Passed Test completed successfully.
2 Failed

An error during the test’s execution caused it to fail.

Note: Test execution failure does not stop the job from running.
3 Terminated*

Possible reasons are:

  • Test execution was stopped by the user or job status is “Automatically Cancelled”
  • The Test execution service failed to start the test:
  • API request failed
  • Test execution service failed to clone/pull repository (failed to create workspace)
  • Test execution service failed to create virtual environment
  • Requirements file is missing
  • Test execution failed/aborted on the Test execution service
  • Test execution service, which had a running job, was deleted from the Job Scheduling portal
4 FailedToStart*

Job triggered. Test failed to execute.

Possible reasons:

  • Test execution service failed to clone/pull the repository (= failed to create workspace)
  • Test execution service ES failed to created virtual environment
  • Test's requirements file is missing

* Test execution failure does not stop the job from running its other tests.