The National Institute for Computational Sciences

General: What is Optimistic Memory Allocation? How does it affect me?

Linux uses "virtual memory" for each process, which creates the illusion of a contiguous memory block when a process starts, even if physical memory is fragmented, or residing on a hard disk. When a process calls malloc, it is given a pointer to an address in this virtual memory. When the virtual memory is first used, it is then mapped to physical memory.

Optimistic Memory Allocation means that Linux is willing to allocate more virtual memory than there is physical memory, based on the assumption that a program may not need to use all the memory it asks for. When a node has used all its physical memory, and there is another call to malloc, instead of giving a null pointer, the program will receive a seemingly good pointer to virtual memory. When the memory is used, the kernel will try to map the virtual memory to physical memory, and enter an "Out of Memory" condition. To recover, the kernel will kill one (or more) process. On Darter, this will almost certainly be your executable. You should see "OOM killer terminated this process."

For more information, see O'Reilly's writeup or man malloc under "Bugs".