Collecting Logs

This topic will guide you on how to troubleshoot CloudShell execution errors. CloudShell saves details of execution activity in different logs, according to automation piece and execution server type (Windows or Linux).

If you are a user or an Admin, you can use these logs to troubleshoot your issue. In some cases, if you have approached Quali Customer Support for assistance, they may ask for these logs to analyze the circumstances that led up to the error.

CloudShell logs include the following:

We also recommend that you use the system messages displayed in CloudShell Portal, for example, the Activity Details Window, which presents detailed information pertaining to a specific error or message in the activity feed, including the event's logging information and (for command execution errors) the execution server on which the command is supposed to run. For more information on all CloudShell system messages, see the CloudShell help topic Viewing System Messages.

CloudShell system logs

CloudShell Monitor is a tool that collects CloudShell logs and data from the machine it is running on. It delivers a snapshot of the situation immediately prior to when the logs were extracted.

We suggest that you watch this short video demonstrating how to collect and save system log files using CloudShell Monitor:

To collect the system logs, perform the following steps:

You will need the logs from the Quali Server machine and the relevant client machine (Studio, Execution Server, CloudShell Portal, etc.).

  1. On the appropriate machine, open CloudShell Monitor.

  2. Click Save Logs and from the drop-down list, select Save Logs and Windows Events.

    A zip file containing the log files is downloaded to your computer.

  3. Save the zip file.

  4. Open a support ticket at https://support.quali.com/ and attach the zip file to your ticket.

    If the zip file is too large to attach to your support ticket (greater than 20 MB), you will receive a Payload is Too Large message in the top right corner of your screen. If this is the case, do the following:

    1. Log on to the QualiSystems FTP site.
      If you do not have the required credentials, contact customer support.
    1. Save the zip file on QualiSystems FTP.
      The file is saved in your account's FTP folder.
    2. In the ticket, state that you uploaded the log files to QualiSystems FTP.
      A Quali support representative will review the logs and contact you.

    The zip file contains the following files, depending on the CloudShell component:

    Log name Type
    EventLogs File folder
    RemoteTeamServer File folder
    SetupLogs File folder
    Monitor File folder
    CloudShell Authoring File folder
    CloudShell Configuration File folder
    CloudShell License Server File folder
    CloudShell Remote Runner File folder
    CloudShell Resource Management Client File folder
    CloudShell Runtime File folder
    CloudShell Sandbox API File folder
    CloudShell Sandbox API Gateway File folder
    CloudShell Server Proxy File folder
    Quali Server File folder
    TestShell Execution Server File folder
    TestShell QSMQ Server File folder
    TestShell Studio File folder
    RunningProcesses.txt Text Document
    SytemInfo.exe.txt Text Document
    SystemInformation.txt Text Document
    QualisystemsRegistry.txt Text Document
    InstallInfo.txt Text Document
    cs_usage_data.zip ZIP package

QualiX logs

Collecting QualiX logs

  1. Open Quali's QualiX Download Page, search the page for "utilities", and click the Utilities link of the appropriate QualiX version.
  2. Extract the files to your local directory.
  3. Edit the .\ GetQualiXInfo.ps1 file and set the Username and Password to the QualiX machine.

  4. Save the file and run the .\ GetQualiXInfo.ps1 file with Windows PowerShell.

    The script connects to QualiX using SSH and extracts the logs from the QualiX machine.

  5. When prompted, enter the IP address of the QualiX machine.

    The command returns the QualiX logs.

  6. Transfer the files from the Linux machine to your local machine.
  7. Open a support ticket and attach the files to your ticket.

Connection request logs

To get the connection request logs for all services (QualiX over vSphere/Docker):

  • SSH to the QualiX Server/Docker hostand run:

    /var/log/nginx/access.log

Guacamole service logs

For details about the Guacamole services, see QualiX High-level Setup Process.

To get the following logs, SSH to QualiX Server and run the appropriate command.

To get the Qualix-tomcat service application logs:

journalctl -u qualix-tomcat

To get the Qualix-guacd service application logs:

journalctl -u qualix-guacd

Docker logs

To set the log level for the Docker logs:

  • SSH to the Docker host and run:

    sudo GUACD_LOG_LEVEL=debug

To get the docker logs for a specific container:

  • SSH to the Docker host and run:
    sudo docker logs -f <container_id>

    To get the container ids, run sudo docker ps -a.

To get the logs for the guacamole/guacd services running in Docker:

  1. SSH to the Docker host.

  2. If containers are running, destroy them:

    sudo docker stop guacamole guacd && docker rm guacamole guacd

  3. Edit /opt/qualix/docker-compose.yml and set:

    sudo GUACD_LOG_LEVEL=debug

  4. Start the containers:

    cd /opt/qualix && ./start.sh

  5. View the logs:

    sudo docker logs guacamole
    sudo docker logs guacd

Linux execution server logs

Note: If you are using a Windows machine to extract logs from the Linux execution server, we recommend using an SCP client to connect via SSH to the Linux execution server to browse though the logs. You can also transfer the files to a Windows machine in order to send them, if needed.

To extract logs if you are using Linux execution server on a Virtual Appliance (VA):

  • General execution server logs: ~/ExecutionServer/Logs
  • Shell execution logs: /var/log/qualiSystems/
  • Orchestration script execution logs: ./usr/share/QualiSystems/logs

To extract logs if you are using Linux ES on CloudFormation (AWS) or CloudShell Template (Azure):

  • ./opt/ExecutionServer/Logs
  • /var/log/qualiSystems/

Windows execution server logs

When a Windows-based execution server runs a Python command, either shell-based or an orchestration script, the execution server's logs will be saved here: C:\ProgramData\QualiSystems\logs.

  • Shell execution logs (resource discovery and inventory commands): C:\ProgramData\QualiSystems\logs\inventory

    Note that older logging packages are sometimes logged to C:\ProgramData\QualiSystems\logs\Autoload

  • Latest shell/cloud provider execution logs: C:\ProgramData\QualiSystems\logs\<reservation-id>\<driver name>
  • Latest orchestration script logs: C:\ProgramData\QualiSystems\logs\<reservationid>

Shell/script execution logs

Shell execution logs

Shell execution logs reside on the Execution Server running the shell at %ProgramData%\Qualisystems\logs\<<reservation_id>>\<<shell name>>\*.logs folder in a dedicated sub-folder with the reservation ID as the name.

For example:

To customize the shell’s logging policy, enable debug logging:

To change the log level:

  1. Navigate to C:\ProgramData\QualiSystems\venv\<shell-name>\Lib\site-packages\cloudshell\core\logger\qs_config.ini.
  2. Change LOG_LEVEL='INFO' as appropriate. For example, LOG_LEVEL='DEBUG'.

Orchestration script execution logs

  • Orchestration script logs reside on the Execution Server running the script at %programdata%\QualiSystems\logs<reservationid> folder in a dedicated sub-folder with the reservation ID as the name.

    For example:

    Setting up and retrieving script logs is detailed in Logging in orchestration scripts.

Virtual environment logs

The virtual environment creation process is as follows:

  • Shell/script package installations logged at InstallRequirements.txt file: ~\QualiSystems\TestShell\ExecutionServer\Logs\QsPythonDriverHost\
  • \QualiSystems\TestShell\ExecutionServer\Logs\QsPythonDriverHost
  • If the virtual environment fails to build, CloudShell will create a log record in the ExecutionServer\Logs file (~\QualiSystems\TestShell\ExecutionServer\Logs\QsRuntimeService.txt). But in some cases you might want to enable DEBUG logs to get more details.

L1 shell execution logs

Look here for the L1 shell execution logs on the Quali Server machine:

~QualiSystems\CloudShell\Server\Logs\<L1 shell name>.

New Job Scheduling Logs

CloudShells logs events related to the New Job Scheduling's Kubernetes services. For details, see New Job Scheduling Service Logs.

QAC logs

QAC does the following:

  • Scheduler of jobs in the original Job Scheduling: Determines whether a job can be executed within a given time range and when a blueprint can be resolved (for jobs that have an attached blueprint)
  • Determines the next available time slot (available time slots are proposed in the Resolve Conflicts window that is displayed when reserving a blueprint with resource conflicts)

Note: QAC logs are disabled by default. Once enabled, future QAC logs will be included in the CloudShell system logs ZIP file.

To enable QAC logs:

  1. Edit the file C:\Program Files (x86)\QualiSystems\CloudShell\Server\log4net.config.

  2. Remove <-- and --> at the beginning and end of the qacLogs section:

    So it look as follows:

  3. Restart the Quali Server service.