The DBA I work with was reporting this error on the server she was working on. Every time she tried to add a new cron job the error would come up.
Next I ran “df -i” and the result painted a different picture instead, the server had hit the maximum inodes allowed for the “/var” partition. What are inodes?
Ok, so now I know the issue was in “/var”. There’s most likely a directory within “/var” with a whole bunch of files in it causing the problem.
Running “find /var -size -8k” would return me all files within “/var” smaller then 8k. What I found was a long list of unsent mails in the “/var/spool/mquee” directory. 261517 files to be exact.
How I got “261517” ?
find /var/spool/mquee -size -8k | wc -l
The next step was to remove all these files. The “rm” command would not work because the amount files I was trying to delete would exceed the maximum argument count allowed for “rm”.
A different approach had to be taken. The command would search for files within “/var/spool/mquee” that were smaller then 8k and removed them 1 by 1.
find /var/spool/mqueue -size -8k -exec rm -f {} ;
The command ran for 30 minutes and fixed the problem and now I have a happy DBA again.
Have you faced familiar situations before? I would love to hear about them and how you fixed them.