Hello,

Today, my server's / partition was full.

root@server [~]# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/xvda 4030144 3866588 0 100% /
/dev/xvdc 11216120 1109484 9992688 10% /home
/dev/xvdd 1007896 19152 978504 2% /tmp

Size in / partition is listed below :-

root@server [/]# du -h --max-depth=1 ./
0 ./sys
8.4M ./etc
4.0K ./selinux
25M ./opt
5.1M ./scripts
956M ./home
4.7M ./bin
24M ./sbin
2.7M ./tmp
26M ./lib
360M ./var
4.3M ./root
0 ./proc
2.4G ./usr
4.0K ./srv
4.0K ./mnt
4.0K ./boot
16K ./lost+found
40K ./dev
4.0K ./media
3.8G ./


I logged into linode manager and increased the / size to 5 GB and it fixed the problem.

root@server [~]# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/xvda 5166712 2947572 2009100 60% /
/dev/xvdc 10077504 1109484 8968020 12% /home
/dev/xvdd 1007896 19096 978560 2% /tmp


But after a few hours, when I checked, the disk space was again full. See below :-

root@server [~]# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/xvda 5166712 4957028 0 100% /
/dev/xvdc 10077504 1109476 8968028 12% /home
/dev/xvdd 1007896 19132 978524 2% /tmp

though nothing has changed in terms of size.

see the disk space usage again,

root@server [/]# du -h --max-depth=1 ./
0 ./sys
8.4M ./etc
4.0K ./selinux
25M ./opt
5.1M ./scripts
956M ./home
4.7M ./bin
24M ./sbin
2.7M ./tmp
26M ./lib
356M ./var
4.3M ./root
0 ./proc
2.4G ./usr
4.0K ./srv
4.0K ./mnt
4.0K ./.spamassassin
4.0K ./boot
16K ./lost+found
40K ./dev
4.0K ./media
3.8G ./

Finally, I had to reboot the system once again to fix this :

root@server [~]# w
15:24:42 up 1 min, 1 user, load average: 0.52, 0.27, 0.11
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
root ttyp0 61.17.17.91 15:24 0.00s 0.02s 0.00s w
root@server [~]# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/xvda 5166712 2971412 1985260 60% /
/dev/xvdc 10077504 1109476 8968028 12% /home
/dev/xvdd 1007896 19132 978524 2% /tmp

It is again happening with no apparent change in disk. If I reboot the system it will be alright. Is anybody else facing the same issue? Can anyone help here?

Are you swapping by any chance?

This is not an uncommon issue. It looks like:

Process 1 opens a file.
Process 2 deletes the file from the filesystem so it can't be seen by ls, du, or the normal tools. df will see the space used by this hidden file. This does not actually free the space used by this file until process 1 closes it, it's still on disk just most tools don't see it.
Process 1 keeps writing to the file unaware that it was deleted by process 2. It's still a totally valid file from the point of view of process 1.

My guess would be that process 1 is syslog and process 2 is logrotate but they could be any other processes that write and remove files. Homebrew logfile handling scripts are notorious for causing these kind of problems.

ls -l /proc/*/fd should help you find out what process 1 is.

Another less likely thing that could be happening is a process is opening and deleting a file so it can use the resulting hidden file as a kind of temp storage. It could then be putting way to much data in there. Some processes use deleted files as a temp working area.

lsof | grep "(deleted)"

... is a fun way to see deleted (but still open) files.

It again happened today :

Before reboot :

root@server [/]# du -h --max-depth=1 ./
0 ./sys
8.4M ./etc
4.0K ./selinux
25M ./opt
5.1M ./scripts
956M ./home
4.7M ./bin
24M ./sbin
2.8M ./tmp
26M ./lib
401M ./var
4.4M ./root
du: `./proc/1': No such file or directory
du: `./proc/2': No such file or directory
du: `./proc/3': No such file or directory
du: `./proc/4': No such file or directory
du: `./proc/149': No such file or directory
du: `./proc/233': No such file or directory
du: `./proc/235': No such file or directory
du: `./proc/271': No such file or directory
du: `./proc/272': No such file or directory
du: `./proc/273': No such file or directory
du: `./proc/274': No such file or directory
du: `./proc/278': No such file or directory
du: `./proc/280': No such file or directory
du: `./proc/283': No such file or directory
du: `./proc/284': No such file or directory
du: `./proc/285': No such file or directory
du: `./proc/286': No such file or directory
du: `./proc/287': No such file or directory
du: `./proc/288': No such file or directory
du: `./proc/289': No such file or directory
du: `./proc/290': No such file or directory
du: `./proc/291': No such file or directory
du: `./proc/292': No such file or directory
du: `./proc/293': No such file or directory
du: `./proc/852': No such file or directory
du: `./proc/855': No such file or directory
du: `./proc/945': No such file or directory
du: `./proc/971': No such file or directory
du: `./proc/1001': No such file or directory
du: `./proc/1023': No such file or directory
du: `./proc/2961': No such file or directory
du: `./proc/2962': No such file or directory
du: `./proc/2968': No such file or directory
du: `./proc/2969': No such file or directory
du: `./proc/2974': No such file or directory
du: `./proc/2975': No such file or directory
du: `./proc/2980': No such file or directory
du: `./proc/2981': No such file or directory
du: `./proc/3291': No such file or directory
du: `./proc/3339': No such file or directory
du: `./proc/3376': No such file or directory
du: `./proc/3378': No such file or directory
du: `./proc/3387': No such file or directory
du: `./proc/3475': No such file or directory
du: `./proc/3482': No such file or directory
du: `./proc/3504': No such file or directory
du: `./proc/3513': No such file or directory
du: `./proc/3514': No such file or directory
du: `./proc/3541': No such file or directory
du: `./proc/3543': No such file or directory
du: `./proc/3544': No such file or directory
du: `./proc/3608': No such file or directory
du: `./proc/3959': No such file or directory
du: `./proc/4057': No such file or directory
du: `./proc/7267': No such file or directory
du: `./proc/12794': No such file or directory
du: `./proc/12890': No such file or directory
du: `./proc/12892': No such file or directory
du: `./proc/13868': No such file or directory
du: `./proc/16815': No such file or directory
du: `./proc/16959': No such file or directory
du: `./proc/16960': No such file or directory
du: `./proc/16961': No such file or directory
du: `./proc/16962': No such file or directory
du: `./proc/16963': No such file or directory
du: `./proc/16964': No such file or directory
du: `./proc/22942': No such file or directory
du: `./proc/24010': No such file or directory
du: `./proc/24016': No such file or directory
du: `./proc/24017': No such file or directory
du: `./proc/24018': No such file or directory
du: `./proc/28621': No such file or directory
du: `./proc/28803': No such file or directory
du: `./proc/28805': No such file or directory
du: `./proc/28842': No such file or directory
0 ./proc
2.4G ./usr
4.0K ./srv
4.0K ./mnt
4.0K ./.spamassassin
4.0K ./boot
16K ./lost+found
40K ./dev
4.0K ./media
3.8G ./
root@server [/]# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/xvda 5037696 4988440 0 100% /
/dev/xvdc 10206520 1110084 8992728 11% /home
/dev/xvdd 1007896 19176 978480 2% /tmp
root@server [/]# w


After reboot :

root@server [~]# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/xvda 5037696 2985716 2000780 60% /
/dev/xvdc 10206520 1110084 8992728 11% /home
/dev/xvdd 1007896 19144 978512 2% /tmp


I tried the ls -l /proc/*/fd

the result is in the link here :-

http://leelahost.com/tmpfiles/tmp.html

Any luck on how to solve this? Within minutes of posting this reply, the disk usage has become like this,

root@server [~]# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/xvda 5037696 3632084 1354412 73% /
/dev/xvdc 10206520 1110132 8992680 11% /home
/dev/xvdd 1007896 19196 978460 2% /tmp
root@server [~]#

Yikes, that's long!

De-HTMLing it (yuck) and grepping for "(deleted)":

l-wx------ 1 root   root   64 May 13 14:44 12 -> /usr/local/apache/logs/ssl_mutex\ (deleted)
lrwx------ 1 root  root  64 May 13 14:44 12 -> /tmp/ib8JF7Hq\ (deleted)
lrwx------ 1 root  root  64 May 13 14:44 5 -> /tmp/ibcGKxZ5\ (deleted)
lrwx------ 1 root  root  64 May 13 14:44 6 -> /tmp/ib3ZP57a\ (deleted)
lrwx------ 1 root  root  64 May 13 14:44 7 -> /tmp/ibwhaEgg\ (deleted)
lrwx------ 1 root  root  64 May 13 14:44 8 -> /tmp/ibbd77pl\ (deleted)
l-wx------ 1 root root 64 May 13 14:44 12 -> /usr/local/apache/logs/ssl_mutex\ (deleted)
l-wx------ 1 root root 64 May 13 14:44 12 -> /usr/local/apache/logs/ssl_mutex\ (deleted)
l-wx------ 1 root   root   64 May 13 14:44 12 -> /usr/local/apache/logs/ssl_mutex\ (deleted)
l-wx------ 1 root   root   64 May 13 14:44 12 -> /usr/local/apache/logs/ssl_mutex\ (deleted)
l-wx------ 1 root   root   64 May 13 14:44 12 -> /usr/local/apache/logs/ssl_mutex\ (deleted)
l-wx------ 1 root   root   64 May 13 14:44 12 -> /usr/local/apache/logs/ssl_mutex\ (deleted)
l-wx------ 1 root   root   64 May 13 14:48 12 -> /usr/local/apache/logs/ssl_mutex\ (deleted)
lrwx------ 1 root  root  64 May 13 14:48 12 -> /tmp/ib8JF7Hq\ (deleted)
lrwx------ 1 root  root  64 May 13 14:48 5 -> /tmp/ibcGKxZ5\ (deleted)
lrwx------ 1 root  root  64 May 13 14:48 7 -> /tmp/ibwhaEgg\ (deleted)
lrwx------ 1 root  root  64 May 13 14:48 8 -> /tmp/ibbd77pl\ (deleted)
l-wx------ 1 root root 64 May 13 14:48 12 -> /usr/local/apache/logs/ssl_mutex\ (deleted)
l-wx------ 1 root   root   64 May 13 14:48 12 -> /usr/local/apache/logs/ssl_mutex\ (deleted)
l-wx------ 1 root   root   64 May 13 14:48 12 -> /usr/local/apache/logs/ssl_mutex\ (deleted)
l-wx------ 1 root   root   64 May 13 14:48 12 -> /usr/local/apache/logs/ssl_mutex\ (deleted)
l-wx------ 1 root   root   64 May 13 14:48 12 -> /usr/local/apache/logs/ssl_mutex\ (deleted)

The /usr/local/apache ones seem to be Apache (duh); the /tmp ones are MySQL.

Regarding actual *answers*, I dunno...

(caker's "lsof | grep '(deleted)'" is better, BTW. It reports the name of the process and the size of the file.)

_________________
Matt Nordhoff (aka Peng on IRC)

Using caker's trick, I'm seeing lots of temporary lock files and such, but also 36 entries for "/var/log/apache2/access.log.1". They all seem to have the same size. I'm not sure how to interpret the results; do all 36 refer to the same file?

# lsof | grep "(deleted)"
apache2     432   www-data   48w      REG      202,0         0      24632 /var/run/apache2/ssl_mutex (deleted)
apache2     432   www-data   52u      REG      202,0         0       1485 /tmp/.xcache.0.0.1125898167.lock (deleted)
apache2     796   www-data   48w      REG      202,0         0      24632 /var/run/apache2/ssl_mutex (deleted)
apache2     796   www-data   52u      REG      202,0         0       1485 /tmp/.xcache.0.0.1125898167.lock (deleted)
apache2     915   www-data   48w      REG      202,0         0      24632 /var/run/apache2/ssl_mutex (deleted)
apache2     915   www-data   52u      REG      202,0         0       1485 /tmp/.xcache.0.0.1125898167.lock (deleted)
apache2     916   www-data   48w      REG      202,0         0      24632 /var/run/apache2/ssl_mutex (deleted)
apache2     916   www-data   52u      REG      202,0         0       1485 /tmp/.xcache.0.0.1125898167.lock (deleted)
apache2    1001   www-data   48w      REG      202,0         0      24632 /var/run/apache2/ssl_mutex (deleted)
apache2    1001   www-data   52u      REG      202,0         0       1485 /tmp/.xcache.0.0.1125898167.lock (deleted)
apache2    1157   www-data   48w      REG      202,0         0      24632 /var/run/apache2/ssl_mutex (deleted)
apache2    1157   www-data   52u      REG      202,0         0       1485 /tmp/.xcache.0.0.1125898167.lock (deleted)
apache2    1287   www-data   48w      REG      202,0         0      24632 /var/run/apache2/ssl_mutex (deleted)
apache2    1287   www-data   52u      REG      202,0         0       1485 /tmp/.xcache.0.0.1125898167.lock (deleted)
apache2    1299   www-data   48w      REG      202,0         0      24632 /var/run/apache2/ssl_mutex (deleted)
apache2    1299   www-data   52u      REG      202,0         0       1485 /tmp/.xcache.0.0.1125898167.lock (deleted)
apache2    9310   www-data    2w      REG      202,0  16049417      43253 /var/log/apache2/error.log.1 (deleted)
apache2    9310   www-data   11w      REG      202,0    217594      43255 /var/log/apache2/other_vhosts_access.log.1 (deleted)
apache2    9310   www-data   12w      REG      202,0  28194695      41355 /var/log/apache2/access.log.1 (deleted)
apache2    9310   www-data   13w      REG      202,0  28194695      41355 /var/log/apache2/access.log.1 (deleted)
apache2    9310   www-data   14w      REG      202,0  28194695      41355 /var/log/apache2/access.log.1 (deleted)
apache2    9310   www-data   15w      REG      202,0  28194695      41355 /var/log/apache2/access.log.1 (deleted)
apache2    9310   www-data   16w      REG      202,0  28194695      41355 /var/log/apache2/access.log.1 (deleted)
apache2    9310   www-data   17w      REG      202,0  28194695      41355 /var/log/apache2/access.log.1 (deleted)
apache2    9310   www-data   18w      REG      202,0  28194695      41355 /var/log/apache2/access.log.1 (deleted)
apache2    9310   www-data   19w      REG      202,0  28194695      41355 /var/log/apache2/access.log.1 (deleted)
apache2    9310   www-data   20w      REG      202,0  28194695      41355 /var/log/apache2/access.log.1 (deleted)
apache2    9310   www-data   21w      REG      202,0  28194695      41355 /var/log/apache2/access.log.1 (deleted)
apache2    9310   www-data   22w      REG      202,0  28194695      41355 /var/log/apache2/access.log.1 (deleted)
apache2    9310   www-data   23w      REG      202,0  28194695      41355 /var/log/apache2/access.log.1 (deleted)
apache2    9310   www-data   24w      REG      202,0  28194695      41355 /var/log/apache2/access.log.1 (deleted)
apache2    9310   www-data   25w      REG      202,0  28194695      41355 /var/log/apache2/access.log.1 (deleted)
apache2    9310   www-data   26w      REG      202,0  28194695      41355 /var/log/apache2/access.log.1 (deleted)
apache2    9310   www-data   27w      REG      202,0  28194695      41355 /var/log/apache2/access.log.1 (deleted)
apache2    9310   www-data   28w      REG      202,0  28194695      41355 /var/log/apache2/access.log.1 (deleted)
apache2    9310   www-data   29w      REG      202,0  28194695      41355 /var/log/apache2/access.log.1 (deleted)
apache2    9310   www-data   30w      REG      202,0  28194695      41355 /var/log/apache2/access.log.1 (deleted)
apache2    9310   www-data   31w      REG      202,0  28194695      41355 /var/log/apache2/access.log.1 (deleted)
apache2    9310   www-data   32w      REG      202,0  28194695      41355 /var/log/apache2/access.log.1 (deleted)
apache2    9310   www-data   33w      REG      202,0  28194695      41355 /var/log/apache2/access.log.1 (deleted)
apache2    9310   www-data   34w      REG      202,0  28194695      41355 /var/log/apache2/access.log.1 (deleted)
apache2    9310   www-data   35w      REG      202,0  28194695      41355 /var/log/apache2/access.log.1 (deleted)
apache2    9310   www-data   36w      REG      202,0  28194695      41355 /var/log/apache2/access.log.1 (deleted)
apache2    9310   www-data   37w      REG      202,0  28194695      41355 /var/log/apache2/access.log.1 (deleted)
apache2    9310   www-data   38w      REG      202,0  28194695      41355 /var/log/apache2/access.log.1 (deleted)
apache2    9310   www-data   39w      REG      202,0  28194695      41355 /var/log/apache2/access.log.1 (deleted)
apache2    9310   www-data   40w      REG      202,0  28194695      41355 /var/log/apache2/access.log.1 (deleted)
apache2    9310   www-data   41w      REG      202,0  28194695      41355 /var/log/apache2/access.log.1 (deleted)
apache2    9310   www-data   42w      REG      202,0  28194695      41355 /var/log/apache2/access.log.1 (deleted)
apache2    9310   www-data   43w      REG      202,0  28194695      41355 /var/log/apache2/access.log.1 (deleted)
apache2    9310   www-data   44w      REG      202,0  28194695      41355 /var/log/apache2/access.log.1 (deleted)
apache2    9310   www-data   45w      REG      202,0  28194695      41355 /var/log/apache2/access.log.1 (deleted)
apache2    9310   www-data   46w      REG      202,0  28194695      41355 /var/log/apache2/access.log.1 (deleted)
apache2    9310   www-data   47w      REG      202,0  28194695      41355 /var/log/apache2/access.log.1 (deleted)
apache2    9310   www-data   48w      REG      202,0         0      24632 /var/run/apache2/ssl_mutex (deleted)
apache2    9310   www-data   49u      REG      202,0         0        169 /tmp/.xcache.0.0.846930886.lock (deleted)
apache2   14186       root   48w      REG      202,0         0      24632 /var/run/apache2/ssl_mutex (deleted)
apache2   14186       root   52u      REG      202,0         0       1485 /tmp/.xcache.0.0.1125898167.lock (deleted)
mysqld    20694      mysql    5u      REG      202,0         0       7611 /tmp/ib0Y84AP (deleted)
mysqld    20694      mysql    6u      REG      202,0        21       7612 /tmp/ibQBz5Vu (deleted)
mysqld    20694      mysql    7u      REG      202,0         0       7613 /tmp/ibU5rgha (deleted)
mysqld    20694      mysql    8u      REG      202,0         0       7614 /tmp/ibwpy51P (deleted)
mysqld    20694      mysql   12u      REG      202,0         0       7615 /tmp/ibglE9iC (deleted)
apache2   29206   www-data   48w      REG      202,0         0      24632 /var/run/apache2/ssl_mutex (deleted)
apache2   29206   www-data   52u      REG      202,0         0       1485 /tmp/.xcache.0.0.1125898167.lock (deleted)
apache2   30618   www-data   48w      REG      202,0         0      24632 /var/run/apache2/ssl_mutex (deleted)
apache2   30618   www-data   52u      REG      202,0         0       1485 /tmp/.xcache.0.0.1125898167.lock (deleted)

Is there something I should do about this?

Ok, on the verge of a 100% disk usage now and I have no idea how to stop this. Will an apache recompilation stop this from happening?

root@server [/]# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/xvda 5037696 4986004 492 100% /
/dev/xvdc 10206520 1110188 8992624 11% /home
/dev/xvdd 1007896 19196 978460 2% /tmp


root@server [/]# lsof | grep '(deleted)'
httpd 1109 root 12w REG 202,0 0 61097 /usr/local/apache/logs/ssl_mutex (deleted)
httpd 1110 nobody 12w REG 202,0 0 61097 /usr/local/apache/logs/ssl_mutex (deleted)
httpd 1112 nobody 12w REG 202,0 0 61097 /usr/local/apache/logs/ssl_mutex (deleted)
httpd 1113 nobody 12w REG 202,0 0 61097 /usr/local/apache/logs/ssl_mutex (deleted)
httpd 1114 nobody 12w REG 202,0 0 61097 /usr/local/apache/logs/ssl_mutex (deleted)
httpd 1745 nobody 12w REG 202,0 0 61097 /usr/local/apache/logs/ssl_mutex (deleted)
httpd 1754 nobody 12w REG 202,0 0 61097 /usr/local/apache/logs/ssl_mutex (deleted)
httpd 1755 nobody 12w REG 202,0 0 61097 /usr/local/apache/logs/ssl_mutex (deleted)
httpd 1759 nobody 12w REG 202,0 0 61097 /usr/local/apache/logs/ssl_mutex (deleted)
httpd 1760 nobody 12w REG 202,0 0 61097 /usr/local/apache/logs/ssl_mutex (deleted)
httpd 1952 nobody 12w REG 202,0 0 61097 /usr/local/apache/logs/ssl_mutex (deleted)
mysqld 2961 mysql 5u REG 202,48 0 12 /tmp/ibTI2DAX (deleted)
mysqld 2961 mysql 6u REG 202,48 0 13 /tmp/ibgEXiC3 (deleted)
mysqld 2961 mysql 7u REG 202,48 0 14 /tmp/ibBvrYD9 (deleted)
mysqld 2961 mysql 8u REG 202,48 0 15 /tmp/ibCaTSHf (deleted)
mysqld 2961 mysql 12u REG 202,48 0 16 /tmp/ibPVCIXl (deleted)
httpd 3504 root 12w REG 202,0 0 61097 /usr/local/apache/logs/ssl_mutex (deleted)
jsvc 3675 root 1w REG 202,0 2051723331 61873 /usr/local/jakarta/apache-tomcat-5.5.30/logs/catalina.out.bkup (deleted)
jsvc 3678 tomcat 1w REG 202,0 2051723331 61873 /usr/local/jakarta/apache-tomcat-5.5.30/logs/catalina.out.bkup (deleted)
httpd 7068 nobody 10w REG 202,0 73 196612 /usr/local/apache/domlogs/palazhiengineering.in-bytes_log.bkup (deleted)
httpd 7068 nobody 12w REG 202,0 0 61097 /usr/local/apache/logs/ssl_mutex (deleted)
httpd 7068 nobody 17u REG 202,0 384 62317 /usr/local/apache/logs/jk-runtime-status.3504 (deleted)
httpd 7068 nobody 18u REG 202,0 1 62322 /usr/local/apache/logs/jk-runtime-status.3504.lock (deleted)
httpd 25982 nobody 10w REG 202,0 35 196620 /usr/local/apache/domlogs/palazhiengineering.in-bytes_log.bkup (deleted)
httpd 25982 nobody 12w REG 202,0 0 61097 /usr/local/apache/logs/ssl_mutex (deleted)
httpd 25982 nobody 17u REG 202,0 384 62465 /usr/local/apache/logs/jk-runtime-status.3504 (deleted)
httpd 25982 nobody 18u REG 202,0 1 62466 /usr/local/apache/logs/jk-runtime-status.3504.lock (deleted)
httpd 30956 nobody 10w REG 202,0 2603 139572 /usr/local/apache/domlogs/palazhiengineering.in-bytes_log.bkup (deleted)
httpd 30956 nobody 12w REG 202,0 0 61097 /usr/local/apache/logs/ssl_mutex (deleted)
httpd 30956 nobody 17u REG 202,0 384 61831 /usr/local/apache/logs/jk-runtime-status.3504 (deleted)
httpd 30956 nobody 18u REG 202,0 1 61869 /usr/local/apache/logs/jk-runtime-status.3504.lock (deleted)

Ok, thanks for all the help from members here.

I think I found out what is causing this :

jsvc 3675 root 1w REG 202,0 2051723331 61873 /usr/local/jakarta/apache-tomcat-5.5.30/logs/catalina.out.bkup (deleted)

It is still locking up around 2 GB of disk space and a restart of tomcat seems to have cleared it.

I have now setup a script to stop and start tomcat everyday midnight, which should resolve this issue permanently.

Maybe you should check out what causes it to make such a huge log in the first place...?

_________________
rsk, providing useless advice on the Internet since 2005.