cannot allocate memory error

Had a weird memory error my Linux servers at work today. These were heavy duty machines running  heavy jobs throughout the day. /var/log/messages kept reporting that the kernel had no more memory to process new jobs. Most of the errors read ” cannot allocate memory “

Some digging around later we found the fix. Kernel 2.6 has a new parameter “min_free_kbytes” which allows it to reserve a dedicated amount of memory for itself to process jobs. This kept the kernel from choking up when the servers were faced with sudden spikes in heavy jobs.

I set my server’s  “min_free_kbytes” to “4096” kbytes which was the recommended value. It’s more of a trial and error configuration so I’ll have to monitor the server over a period of time and increase the value if needed till I hit the sweet spot.

How to set it?

To have the new value take effect immediately, edit the “/proc/sys/vm/min_free_kbytes” file. Remember!, reboot and the changes will be forgotten.

echo "4096" > /proc/sys/vm/min_free_kbytes

To have it permanent, add “vm.min_free_kbytes=65536” to the /etc/sysctl.conf file.

echo "vm.min_free_kbytes=4096" >> /etc/sysctl.conf