The idea: free memory is a buffer, not a goal
Linux tries to use all of its RAM. Memory that no program needs right now holds the page cache: copies of file data, so the next read doesn't go to the disk. So "free" memory is normally small, and that is fine. The kernel only has to keep a small reserve, and it refills that reserve by taking pages back (reclaim) when it gets low. Three watermarks per memory zone decide when: below low the background thread kswapd starts reclaiming, at high it stops, and at min a process that wants memory has to reclaim it itself. If nothing can be reclaimed at all, the OOM killer (out of memory) kills a process to get memory back.
The page follows one 4 GiB machine through one load at a time. Pick a load in the first select, press Tick to go one tick at a time, or pick a demo. Every page of memory is a cell, so you can see where each one goes.
Reading the canvas
- File and Anon are the two LRU rows. In each row the active list is left of the dark bar │ and the inactive list is right of it. The right end is the tail: reclaim takes pages from there. The numbers next to the row name are the lengths of the two lists.
- The letter in a cell is the owner: W web, D db, B backup, L leak, K kernel. File pages with no letter are old cache that nobody uses any more.
- The Free memory gauge shows the first 16 free pages, with the min, low and high lines. Under it are
kswapd0, a few lines of/proc/meminfoand the counters from/proc/vmstat. - The process table shows each process's resident pages (RSS), its pages in swap,
oom_score_adjandoom_score. A row turns amber while the process is stalled in direct reclaim and red when it is killed. The kernel log (dmesg) is at the bottom.
| Model value | On this page | On a real 4 GiB machine |
|---|---|---|
| page size | 64 MiB per cell (64 cells) | 4 KiB (about a million pages) |
| min / low / high | 4 / 6 / 8 pages | about 8 / 12 / 16 MiB: scaled up here so they are visible |
| kswapd | 3 pages per tick | thousands of pages per second |
| swap | off, or 8 pages (512 MiB) | a swap partition, swap file or zram |
Where the memory goes
Almost every page in use is one of two kinds:
- File pages (page cache): data of files that were read or written. A clean page is the same as the disk, so the kernel can just drop it and read it again later. A dirty page was written by a program and not yet saved, so it must be written back to disk before it can be freed. See File System for how
write()fills the page cache. - Anonymous pages: the heap and stack of processes, memory with no file behind it. The only place the kernel can put one is swap. With no swap, anonymous pages can't be reclaimed at all.
/proc/meminfo shows the totals: MemFree, Cached, Dirty, AnonPages, SwapFree. The line to watch is MemAvailable, the kernel's estimate of free memory plus what it can easily reclaim. A small MemFree with a big MemAvailable is healthy.
Watermarks: min, low and high
Each zone has three watermarks. The kernel computes min from vm.min_free_kbytes (about 8 MiB on a 4 GiB machine; it grows with the square root of RAM). The gap from min to low and from low to high is a quarter of min, or vm.watermark_scale_factor (default 10, meaning 0.1% of RAM), whichever is bigger. /proc/zoneinfo lists them for every zone, in pages.
| Free pages | What happens on an allocation |
|---|---|
| above low | the page is taken from the free lists, nothing else |
| below low | the page is still given, and kswapd is woken to reclaim in the background until free reaches high |
| at min | the process must free pages itself first: direct reclaim |
| nothing reclaimable | out_of_memory(): a process is killed |
The pages below min are kept for the kernel itself and for allocations that can't wait, such as those in interrupt handlers. A bigger watermark_scale_factor wakes kswapd earlier, so programs hit direct reclaim less often.
kswapd and direct reclaim
Both kinds of reclaim do the same work. They take pages from the tail of the inactive lists: a clean file page is dropped, a dirty file page is sent to the disk and freed once the write is done, and an anonymous page is written to swap. A page that is used again while on the inactive list moves to the active list. When an inactive list runs short, pages from the tail of the active list are moved over (deactivated). The result: a big file read once fills the inactive list and is the first to go, while pages that are used often stay. In Demo: read a big file the cache of web and db (the hot pages on the active list) survives while 30 pages of backup's file pass through. Which page to evict is the subject of Page Replacement (FIFO, Clock, LRU); this page uses the Linux lists without going into that.
The difference is who waits. kswapd is a kernel thread (one per memory node, kswapd0) and reclaims while programs keep running. Direct reclaim happens inside the allocation: the process can't continue until it has freed some pages. It doesn't write dirty pages itself, because that is slow and could need memory. If memory is used faster than kswapd frees it, programs stall, and that is the latency spike you see on a machine under memory pressure (Demo: read too fast). /proc/vmstat counts it: allocstall_normal is the number of direct reclaims, pgsteal_kswapd and pgsteal_direct the pages each one freed.
vm.swappiness (0 to 200, default 60) sets how much reclaim prefers swapping anonymous pages over dropping file pages. The page simplifies this: it takes file pages first and swaps only when no file page is left.
The OOM killer
When reclaim frees nothing, even after retries, the kernel calls out_of_memory(). It goes through all processes and computes a badness for each (oom_badness() in mm/oom_kill.c):
points = RSS + swap entries + page tables (in pages)
points += oom_score_adj × totalpages / 1000 (totalpages = RAM + swap)The process with the most points gets SIGKILL. So the OOM killer kills the process that would free the most memory: the biggest, not necessarily the one that is leaking. /proc/<pid>/oom_score shows the same value scaled: (1000 + points × 1000 / totalpages) × 2/3, so a process that uses almost nothing shows about 666. oom_score_adj runs from -1000 to 1000: positive values make a process a preferred victim, negative ones protect it, and -1000 means never kill it (its oom_score reads 0).
The kernel log tells you what happened. The first line names the process whose allocation failed; the last names the victim and how much memory it had:
leak invoked oom-killer: gfp_mask=0x140dca(GFP_HIGHUSER_MOVABLE|__GFP_COMP|__GFP_ZERO), order=0, oom_score_adj=0
Out of memory: Killed process 1400 (leak) total-vm:2492884kB, anon-rss:2490368kB, file-rss:1792kB, shmem-rss:0kB, UID:1000 pgtables:4864kB oom_score_adj:0anon-rss is the anonymous memory that is freed. The oom_reaper thread frees it right away, without waiting for the process to finish dying. In Demo: memory leak, no swap all page cache is dropped first, even the hot pages, then the leak is killed. With swap (Demo: memory leak with swap) kswapd also swaps out idle pages, which buys a few ticks, and then the same thing happens: swap only delays an OOM caused by a leak. In Demo: leak with oom_score_adj -1000 the leak can't be chosen, so db is killed, then web, and the leak keeps going. With nothing left to kill the kernel would panic: System is deadlocked on memory. Set vm.panic_on_oom = 1 to panic (and reboot) instead of killing at all.
cgroups: an OOM inside a container
A cgroup v2 limit memory.max is a smaller wall around a group of processes, for example a container started with docker run --memory 512m. Every page a process in the group uses is charged to it. When a charge would go over the limit, the kernel reclaims inside the group only; if that frees nothing, the memcg OOM killer kills a process in that group, even if the machine has plenty of free memory (Demo: leak in a container). The log says Memory cgroup out of memory, memory.events counts oom_kill, and the container exits with code 137 (128 + 9, SIGKILL); Kubernetes reports it as OOMKilled. memory.high is a softer limit that throttles and reclaims instead of killing. See What a Container Really Is and The Life of a Kubernetes Pod.
Overcommit: why malloc rarely fails
malloc and mmap only reserve addresses; a page is taken when the program first touches it (a page fault; see Linux Virtual Memory). Linux lets programs reserve more than RAM + swap, because most never touch all of it. That is why a program usually doesn't get NULL from malloc: it gets the memory, and the trouble comes later, when touching a page needs a free page that isn't there.
vm.overcommit_memory | Rule |
|---|---|
| 0 (default) | heuristic: refuse only requests that obviously can't fit |
| 1 | always say yes |
| 2 | strict: refuse when Committed_AS would pass CommitLimit = swap + RAM × vm.overcommit_ratio% (default 50); malloc returns NULL and nobody is killed |
How to see it on a real system
free -m # total, used, free, buff/cache, available
vmstat 1 # si/so = swap in/out per second
grep -E 'allocstall|pgsteal|oom_kill' /proc/vmstat
dmesg -T | grep -i -A2 'out of memory'
cat /proc/$(pidof postgres | cut -d' ' -f1)/oom_score
echo -500 | sudo tee /proc/1234/oom_score_adj # protect a process (or OOMScoreAdjust= in a systemd unit)See also Buddy Allocator for how free pages are kept in blocks, and Backpressure for an unbounded queue that ends in an OOM kill.
What the page leaves out
One memory zone and one NUMA node are drawn; real machines have several zones (DMA32, Normal) and nodes, each with its own watermarks and kswapd. The page reclaims file pages before anonymous ones instead of balancing them with vm.swappiness, and skips MGLRU (the multi-generational LRU used by many distributions since Linux 6.1), refault detection, slab shrinkers, shared memory and tmpfs (which can only be swapped), huge pages and compaction for larger allocations, zswap, memory.low/memory.min protection, throttling of writers by vm.dirty_ratio, and user-space OOM daemons (systemd-oomd, earlyoom) that act early on pressure stall information (PSI). Page tables are left out of the oom_score, and the numbers are scaled so they are easy to read.