Subject: Important: ClickHouse crashes after U42/U43 upgrade
Feature Flag: SAAS-28408

Overview

We’re investigating an issue where ClickHouse may crash after upgrading to U42 due to an interaction between ClickHouse runtime integrity checks and Aqua runtime detection behavior.  

Symptoms

After upgrading to U42, ClickHouse may terminate shortly after startup (or intermittently), with an error similar to:

DB::Exception: Calculated checksum of the executable (...) does not correspond to the reference checksum stored in the executable (...) ... (CORRUPTED_DATA)

This is the same ClickHouse integrity-check failure pattern documented by ClickHouse in their issue tracker.

What’s happening

  • ClickHouse can perform runtime memory integrity checks. In U42, new AI-related runtime detection (including Uprobe-based events and AI behavioral signatures) can change user-mode memory signatures in a way ClickHouse may treat as tampering, causing ClickHouse to terminate.  

Why it can occur even if the UI feature flag is not enabled

Some SecureAI controls may be active at runtime even if the feature flag is not enabled, because the flag controls UI visibility while backend settings are enabled by default. In addition, U42 includes two new behavioral signatures that can contribute to the conflict even when SecureAI is otherwise disabled.  

Mitigation options (immediate)  

Option 1: ClickHouse-side mitigation 

Upgrade ClickHouse to a version that supports disabling the binary checksum integrity check via the server setting:

  • skip_binary_checksum_checks — “Skips ClickHouse binary checksum integrity checks”

Important note: This option was introduced later in ClickHouse (added via PR “Introduce option to skip binary checksums”, merged July 15, 2025) and therefore is not available in older ClickHouse versions.

(Background: ClickHouse maintainers and users discussed this behavior in relation to eBPF/uprobes and checksum failures. )


Option 2: Aqua-side mitigation 

To fully mitigate on the Aqua side, disable both:

  1. SecureAI controls (container and host level), and

  2. AI-related Behavioral Detection, including the LLM request/response signatures

You can apply this in one of two ways:

  • Manual (small deployments): Enable the SAAS-28408 UI feature flag on console and gateway side to expose settings, then turn off SecureAI controls and AI behavioral detection.

  • Programmatic (recommended at scale): Use an API-based script to update settings across all Enforcer Groups (needed for environments with many groups). 

How to Enable FF on console and Gateway Side

We need to add AQUA_FF_OVERRIDES='{"SAAS-28408":true}' in extraEnvironmentVars in helm values.yaml file or as env in deployment manifest where name will be AQUA_FF_OVERRIDES  and value will be '{"SAAS-28408":true}' . After this we need to redeploy Console and Gateway.


Once this is done, UI will start showing Secure AI control in Enforcer group. We need to Disable both Secure AI control on Container and Host runtime along with Behavioral detection. 

Fix timeline

A new service pack release (SP4) is being prepared to remove the AI-specific behavioral signatures contributing to this conflict and will be available on both the U43 and U42 releases. Once upgraded to SP4, you may re-enable behavioral detection.  

Need help?

If you’re seeing ClickHouse crashes or want help applying the mitigation (manual or programmatic), please contact Aqua Support or your Customer Success Manager.  

References