How to avoid out of memory with Spill to Disk

When queries require a huge amount of memory to be processed, Indexima can offload intermediate computation results on disk. Memory is used as its best, without exceeding the cluster maximum available memory. This way, your cluster can avoid running out of memory and is still able to answer queries.

How to use Spill to Disk

Spill to disk can be enabled with the following command in the Developer Console:

SET_ spill.enable = true
CODE

If you want to resize the maximum memory used for results :

SET_ spill.memory.size.mb = 1024
CODE

All configuration parameters for the spill to disk function can be set on-the-fly, or be written in the galactica.conf

How to know if a query has spilled to disk

In the monitor, on the queries page, a query with the icon 

 has spilled to disk.

Configuration

Go to SpillToDisk configuration