Summary

This article will provide a breakdown of basic troubleshooting steps or "must haves" for Aqua Enterprise components to include the Aqua Console, Gateway, Scanner and Enforcer. 


General Information

Log Level

When troubleshooting any component of the Aqua ESE solution, the log level must be set to DEBUG.  This allows for more verbose logging for troubleshooting or if an issue is to be pushed up to our development team. 


To set the log level to DEBUG:


Navigate from Aqua UI Settings > Supportability > "Server Log" tab > Adjust "LEVEL" dropdown to "DEBUG" > Adjust Log Level Expiration Time (minutes) to 10 then click Set > Adjust Log Tail to 100 > Save > Refresh. 


You should see an INFO log entry on the same page that indicates the log level was successfully changed to DEBUG:


INFO Setting log level to DEBUG


Supportability Package

Aqua is constantly collecting information about the Aqua deployed components in the environment. From within the UI, Aqua makes it easy to collect information about multiple components at one time.  When troubleshooting any component of the Aqua ESE solution, collecting the entire supportability package will provide a full picture of the issue at hand.


To collect the entire supportability package: 


Navigate from Aqua UI Settings > Supportability >  "Collector" tab > Choose the "Select All" checkbox > "Collect" button at the bottom of the page. 



Note: If the Collect Button is greyed out - uncheck Vulnerable Difference Info


Aqua Console 

When troubleshooting an issue with the Aqua console:


  1. Set the log level to DEBUG
  2. Reproduce the issue within the Aqua UI, if possible. 
  3. Collect the supportability package from your test environment or from the client. 



Aqua Gateway

When troubleshooting an issue with the Aqua gateway:


  1. Set the log level to DEBUG
  2. Reproduce the issue, if possible
  3. Collect the supportability package from your test environment or from the client.


Aqua Scanner 

When troubleshooting an issue with an Aqua Scanner (scanner-cli daemonset):


Note: If the issue is with a specific image, regardless of scanner type, ask the client if they can provide the image for testing on the Aqua side.


  1. Scale scanner-cli to 0. This may require some additional configuration changes if the client is using an orchestrator that automatically re-deploys containers/pods when one goes down.
  2. Set the log level to DEBUG from within the scanner configuration, you can do this by adding SCALOCK_LOG_LEVEL=DEBUG as an environment variable to the scanner deployment.
  3. Re-deploy your scanner. 
  4. In the scanning options, navigate to System > Settings > Scanning make sure that "only scan images with scanner-cli daemons (if exists)" is checked to force the scanner that was just deployed to be used.
  5. Reproduce the issue, if possible.
  6. Collect the container logs from the scanner container/pod that was used for reproducing the issue.  You can copy and paste the logs to a file or use the following commands to send them to a file, use the appropriate command for the type of deployment:

Docker: docker logs -f (container id) > filename.txt
Kubernetes: kubectl get logs (pod name) > filename.txt
OpenShift: oc get logs (pod name) > filename.txt


When troubleshooting an issue with an Aqua embedded server scanner:

  1. Set the log level to DEBUG.
  2. Reproduce the issue, if possible.
  3. Collect the supportability package from your test environment or from the client. 


Aqua Enforcer 

When troubleshooting an issue with an Aqua Enforcer(agent):


Option 1, via command line:

  1. Identify the node that the enforcer is running on. 
  2. Enter the following command: 
    /opt/aquasec/slk trace modify --module slkd --level 3
  3. Reproduce the issue, if possible
  4. Collect the container logs from the enforcer container/pod.   You can copy and paste the logs to a file or use the following commands to send them to a file, use the appropriate command for the type of deployment:


Docker: docker logs -f (container id) > filename.txt
Kubernetes: kubectl get logs (pod name) > filename.txt
OpenShift: oc get logs (pod name) > filename.txt



Option 2, via the Aqua UI:

  1. Navigate to the Enforcers tab. 
  2. Identify the Enforcer Group you are troubleshooting.
  3. Identify the Enforcer you are troubleshooting. 
  4. Set the Level, Log Level Expiration (minutes) and Select Module to the configurations below:

Note: A log level of 3 is usually high enough for troubleshooting, if it is not, adjust the log level to a higher number, then continue with steps 5 and 6. 

   
    5. Reproduce the issue, if possible. 

    6. Collect the supportability package from your test environment or from the client. 


Note:

The primary method for file upload is by attaching to the ticket, however there is a file size limit. For files that are too large to be attached to the ticket, you're welcome to use our Support Upload portal. This SFTP server was created for customers to securely upload large files.

Our SFTP can be accessed by opening your browser and navigating to https://upload.aquasec.com. Enter your Aqua User Center credentials, then upload the file. It may be helpful to note the filename in the ticket, simply to make it easier for us to find, though we'll be able to track it down either way!