No space left on device

The DBA I work with was reporting this error on the server she was working on. Every time she tried to add a new cron job the error would come up.

[oracle@###### - /db/home/oracle]
$ crontab -e
crontab: installing new crontab
cron/tmp.5821: No space left on device
crontab: edits left in /tmp/crontab.5821

From the error it’s quite obvious that it’s a space related issue. It’s not fun when you get woken up at 3am but anyway this is how I fixed the problem.

First I had to see what the space situation was on the server. Running “df -h” showed sufficient free space available on the server so apparently storage space was not the issue.

Next I ran “df -i” and the result painted a different picture instead, the server had hit the maximum inodes allowed for the “/var” partition. What are inodes?

Ok, so now I know the issue was in “/var”. There’s most likely a directory within “/var” with a whole bunch of files in it causing the problem.

Running “find /var -size -8k” would return me all files within “/var” smaller then 8k. What I found was a long list of unsent mails in the “/var/spool/mquee” directory. 261517 files to be exact.

How I got “261517″ ?

find /var/spool/mquee -size -8k | wc -l

The next step was to remove all these files. The “rm” command would not work because the amount files I was trying to delete would exceed the maximum argument count allowed for “rm”.

A different approach had to be taken. The command would search for files within “/var/spool/mquee” that were smaller then 8k and removed them 1 by 1.

find /var/spool/mqueue -size -8k -exec rm -f {} \;

The command ran for 30 minutes and fixed the problem and now I have a happy DBA again.

Have you faced familiar situations before? I would love to hear about them and how you fixed them.

8 Responses to “No space left on device”

Author comments are in a darker gray color for you to easily identify the posts author in the comments

1. aizatto says:

   September 14, 2007 at 10:55 pm

   Should have just ran:
   ls | xargs rm

   Reply
2. Wing Loon says:

   September 15, 2007 at 12:01 am

   Bro, you may try to run the command below: -

   cd /var/spool/mqueue
   for f in *; do rm -f $f; done

   This will delete all the files in the /var/spool/mqueue

   Hope this help,

   Reply
3. Danny says:

   September 15, 2007 at 3:34 am

   Thanks man. That will work.

   Reply
4. Danny says:

   September 15, 2007 at 3:41 am

   thanks aizatto, worked like a charm

   Reply
5. Wing Loon says:

   September 16, 2007 at 11:07 pm

   aizatto, cool…it works great,

   Reply
6. Maski says:

   May 6, 2008 at 11:58 pm

   The exact same thing just happen to me, well i just rm -Rf /var/spool/mqueue and then mkdir the same dir with the owner nad group and samba finally started!!!

   thanks

   Reply
7. dragoncity99 says:

   June 30, 2008 at 10:24 pm

   For long term resolution if it occurs again:
   1. Put in a cronjob the commands to make it go again.

   Another way might be creating a ruleset in /etc/logrotate.conf to make the files get archived and compressed if the logs are still needed for reference.

   Reply
8. Sky says:

   August 4, 2009 at 9:26 am

   I also met this issue and resolved it with these options:

   $ cd /var/spool/cron/
   $ ls
   root tmp.XXXXIsZHyp tmp.XXXXqggkQI
   $ rm -rf tmp.XXXX*

   Question:
   Who created these two tmp files in “/var/spool/cron/” ?

   Thanks

   Reply