Summary


One of AQUA Security log management integration options is 'Elasticsearch' and using 'Kibana' one can collect and view/filter AQUA Security events to best suit their needs.



Problem


Once integrated, AQUA Security time fields will show as a 10 digit (Long) number and not as dates in Elasticsearch/Kibana.


So if one looks at AQUA Security events in Kibana, one will see something like the below:

Where the time field is showing the number of seconds and not in a real human-friendly view.



Cause


As AQUA Security is sending the timestamp fields in epoch time (number of seconds since 00:00:00 January 1st 1970) yet without the millisecond the auto-generated Elasticsearch index 'aqua-container-security' is lacking the option to index or search by timestamp as it will get the time field in 'long' format and not 'date' format.

In this article we will look at 3 time-related fields from AQUA Security which are sent as epoch seconds:


time, (scan_started) seconds, data_date fields.


* This is an ELK (Elasticsearch) limitation that needs the milliseconds in an 'epoch' field to auto-generate as a 'date' format.



Solution


In order for a more human-friendly view and the option to index/search by the timestamp fields, one can follow the below example:


Disable the integration of Elasticsearch in AQUA to avoid getting a new index auto-generated.


Navigate to your Kibana --> management  --> Elasticsearch  --> index_management  --> indices

Select your AQUA Container Security index and click on the 'Mapping' option, now select the existing mapping and copying it to your favorite editor.


Navigate to your Kibana --> (Stack) Management --> Elasticsearch --> index_management  --> AQUA Container Security and delete the existing AQUA Security auto-generated index.

Also, do the same from your Kibana --> (Stack) Management --> Index Patterns --> AQUA Container Security and delete the existing AQUA Security auto-generated index.


In your favorite editor, update the time field from the below format:


        "time": {

          "type": "long"

        },


To:


        "time": {

          "type": "date",

          "format": "epoch_second"

        },


Now modify the data_date field from:


        "data_date": {

          "type": "long"

        },


To:


        "data_date": {

          "type": "date",

          "format": "epoch_second"

        },


And also update the scan_started 'seconds' field as well from:


            "seconds": {

              "type": "long"

            },

To:


            "seconds": {

              "type": "date",

              "format": "epoch_second"

            },


Now you are ready to re-create the index using the below curl command:


--header 'Content-Type: application/json' \
--data-raw '{
        "mappings": {
...


Where you replace the '...' with your exported mapping from your Editor.


Or using the Kibana Dev Tools and paste the exported mapping there and add on the top the below:


PUT /aqua-container-security/

{

  "mappings": {

...


Where you replace the '...' with your exported mapping from your Editor.


This will create the AQUA Container Security index with the time and data_date as a human-friendly date field.


For example:



Additional Information


This article was written using AQUA Security versions: 4.6.20195 and 5.0.20190.

Tested against ELK suit 7.8.0.


More information on ELK (Elasticsearch suit) time options can be found in elastic.io online pages:

https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-date-format.html 


More information on EPOCH time can be found online:

https://en.wikipedia.org/wiki/Unix_time