At the end of November, we’ll be migrating the Sematext Logs backend from Elasticsearch to OpenSearch

Linux Out of Memory Killer (OOM Killer)

Definition: What Is Out of Memory Killer (OOM Killer)?

The Out of Memory Killer (OOM Killer) is a component of the Linux kernel designed to prevent system-wide memory exhaustion, which could lead to system instability or unresponsiveness. When a Linux system runs out of available physical or swap memory due to excessive memory usage by processes, the OOM Killer intervenes to free up memory and maintain system stability.

Finding Out Why a Process Was Killed

There are several clues you could choose to follow. The MySQL process in the example below was killed by the OOM Killer because of an out-of-memory condition. The capital K in Killed tells us that the process was killed with a -9 signal. This is often a clue that OOM Killer took care of things.

grep -i kill /var/log/messages*
host kernel: Out of Memory: Killed process 2626 (mysqld).

How to Disable OOM Killer

It’s not really possible to fully disable the Linux OOM killer. The best you can do is set a limit after which memory allocations should fail. You can set the vm.overcommit_memory to 2. For more info see overcommit accounting docs.

How to Configure Linux to avoid OOM Killing a Specific Process

You can use the oom_adj range for this. The possible values of oom_adj range from -17 to +15. The higher the score for a process, the more likely the associated process is to be killed by the OOM Killer. If oom_adj is set to -17, the process should not be considered for termination by OOM Killer.

echo -17 > /proc/2626/oom_adj

How do I Monitor OOM Killer Events

Use a tool like Sematext Monitoring, which includes process monitoring capabilities. Sematext captures OOM Killer events, among other things. You will see OOM Killer events on the Events screen in Sematext Cloud. You can also see OOME if you run dmesg on individual nodes.

OOM Killed Alerting in Sematext Cloud