No space left on device

The DBA I work with was reporting this error on the server she was working on. Every time she tried to add a new cron job the error would come up.

[oracle@###### - /db/home/oracle]
$ crontab -e
crontab: installing new crontab
cron/tmp.5821: No space left on device
crontab: edits left in /tmp/crontab.5821

From the error it’s quite obvious that it’s a space related issue. It’s not fun when you get woken up at 3am but anyway this is how I fixed the problem.

First I had to see what the space situation was on the server. Running “df -h” showed sufficient free space available on the server so apparently storage space was not the issue.

Next I ran “df -i” and the result painted a different picture instead, the server had hit the maximum inodes allowed for the “/var” partition. What are inodes?

Ok, so now I know the issue was in “/var”. There’s most likely a directory within “/var” with a whole bunch of files in it causing the problem.

Running “find /var -size -8k” would return me all files within “/var” smaller then 8k. What I found was a long list of unsent mails in the “/var/spool/mquee” directory. 261517 files to be exact.

How I got “261517″ ?

find /var/spool/mquee -size -8k | wc -l

The next step was to remove all these files. The “rm” command would not work because the amount files I was trying to delete would exceed the maximum argument count allowed for “rm”.

A different approach had to be taken. The command would search for files within “/var/spool/mquee” that were smaller then 8k and removed them 1 by 1.

find /var/spool/mqueue -size -8k -exec rm -f {} \;

The command ran for 30 minutes and fixed the problem and now I have a happy DBA again.

Have you faced familiar situations before? I would love to hear about them and how you fixed them.