disk space

Have you encountered a situation like you see your root partition gets full (100%) but you see nothing needs to be cleaned up. In other words, the df output shows 100% or some thing huge disk space at a particular folder but the ‘du’ output shows it was not that much huge?

For example, I encountered one such situation in one of our production hosts, where the root partition got full with 100%..but the ‘du’ command on all the folders under that partition showed me nothing that much used space.

It lead me to run the following lsof command to investigate into…

undisclosed-host:/ # lsof|grep -i delete
….
….
lrthdf5   55870  root    1w      REG                8,3 13876053553    3278175 /var/log/lrthdf5/onl_dev_bin_mc3_dev.log (deleted)
lrthdf5   56098  root    2w      REG                8,3 13881587249    3278181 /var/log/lrthdf5/onl_replay_mc3replay.log (deleted)

From the above ‘lsof’ output, we find the processes with the pids 55870 and 56098 have kept the files /var/log/lrthdf5/onl_dev_bin_mc3_dev.log and /var/log/lrthdf5/onl_replay_mc3replay.log as open with the corresponding file descriptors (fd) mentioned..

It seemed to me like, as the part of a maintenance process, somebody deleted those log files while the files are being written by those processes in action.

As these files have been identified, I did free space occupied by those file by shutting down the processes in question.

Before:

undisclosed-host:/ # df -Th
Filesystem Type Size Used Avail Use% Mounted on
/dev/sda3 ext3 63G 60G 384M 100% /
tmpfs tmpfs 16G 0 16G 0% /sys/fs/cgroup
udev tmpfs 16G 212K 16G 1% /dev
tmpfs tmpfs 16G 0 16G 0% /dev/shm
/dev/sdb1 xfs 5.0T 3.7T 1.3T 74% /mnt/sdb1

After shutting down the process (the used space got down from 100% to 78% ):

undisclosed-host:/ # stpcap onl_replay mc3replay
Fri Feb 9 00:18:09 PST 2018: Stopping capture processes. Please wait, it may take a while…

undisclosed-host:/ # df -Th
Filesystem Type Size Used Avail Use% Mounted on
/dev/sda3 ext3 63G 47G 14G 78% /
tmpfs tmpfs 16G 0 16G 0% /sys/fs/cgroup
udev tmpfs 16G 212K 16G 1% /dev
tmpfs tmpfs 16G 0 16G 0% /dev/shm
/dev/sdb1 xfs 5.0T 3.7T 1.3T 74% /mnt/sdb1

Alternatively, it is possible to force the system to de-allocate the space consumed by an in-use file by forcing the system to truncate the file via the proc file system. But it is more advanced. Not needed in my case though..

Leave a Reply

Your email address will not be published. Required fields are marked *