Walker News

Df And Du Command Show Different Used Disk Space

I used to take to a nap during lunch break to recharge my tired soul. But, luck is not with me today, as the Oracle DBA questioned me where was the 73GB disk space gone in his /oracle disk partition (that built on logical volume of LVM v2)?

I thought it was caused by the reserved block count of EXT3 file system. But, it was not that case and I can’t answer him immediately when I look at that screen-shot – a total of 73GB disk space gone “missing” when comparing the output of du and df command!

Why the Linux df and du command reports such big difference of used disk space or free disk space?
Why the Linux df and du command reports such big difference of used disk space or free disk space?

Luckily, Google is everyone friend – it has the answers for the Oracle DBA and myself.

In fact, this “du VS df” disk space problem is not unique to Linux distributions alone. The difference of used disk space reported by df and du command exists in other Unix variants and Sun Solaris too.

So, what has exactly caused the du and df command showing such a big difference of used disk space or free disk space in my Red Hat Enterprise Linux (i.e. total of 73GB)?

Answer – This happens when the DBA deleted a list of Oracle database files (*.dbf) while he was restoring the database using those dbf files!

If the files are deleted (by rm command) while they are being opened or used by a Linux program / process, the evil of “open file descriptor” problem arises and confuse the Linux file system on reporting the real figure of used disk space or free disk space available.

In order to resolve the fake “disk space full” problem, i.e. to reclaim “used disk space”, you need to kill or terminate the “defunct process” – in this case, the rm command that turns to be defunct process while the files are being used.

Once these defunct processes are terminated, the “open file descriptor” problem will be resolved, and both the du and df commands will agree to report the real file system used disk space or free disk space!

How to find out and terminate or kill the defunct processes that cause open file descriptor problem, in order to resolve the difference of used disk space in du and df command?

For this particular scenario, the lsof command (list open file command) is great to show light:

lsof | grep "deleted" or
lsof | grep "oracle" (rather long and messy)

and look for Linux process ID in second column of the lsof command output. The seventh column is the size of file being “deleted” (but not success and turns out to be defunct process). For example, this is one of the lines in my lsof command output:

oracle 906 oradba 28u REG 53,7 3664 45 /oracle/user2.dbf (deleted)

How to recreate the “open file descriptor” problem that causes the difference of used disk space reported by df and du command?
  1. Create one 500MB file in my /home file system:
    dd if=/dev/zero of=/home/zte bs=1024 count=500000
     
  2. Run md5 checksum against the 500MB file with md5sum command:
    md5sum /home/zte
     
  3. Now, open another session and remove the /home/zte file while md5sum still computing its md5 checksum:
    rm /home/zte
     
  4. Now, both the Linux df and du commands will report different used disk space or free disk space, that caused by “open file descriptor” problem:
    df -h; du -h --max-depth=1 /home

Thanks to Vivek@nixCraft on this topic.

Custom Search

  1. Ibrahim 26-11-07@14:35

    Thanks to Vivek@nixCraft on this topic.

    actually im facing the same problem on UNIX Solaris platform , and lsof command is not defined their , Im sure there is an equivalent commnad,can you please help.

    best regards,
    Ibrahim

  2. dd 30-11-07@02:47

    I also have almost the same problem on my Vista home basic, HP . I saved not more 5GB ever (I bought it recently) ; but it reports 50GB left from the total of 120GB hard risk
    What is happening guy??
    Help please

  3. Vahid Pazirandeh 06-04-09@01:50

    To look for locked files, try also: lsof +L1

  4. Masen 28-07-10@02:25

    I had this same problem with the pflogd process going defunct on an openBSD box and writing to a file that didn’t exist in the filesystem! I’m not quite sure how it happened, but I stumbled upon this page (from google I should add) and had the same problem as commenter #1.

    For anyone else who finds this and is on openBSD, freeBSD, or possibly others, try
    # fstat
    to show all open files. For more info hit up $ man fstat

  5. Purpendicular 02-11-10@03:43

    Very well explained and illustrated!

    Tnx,
    John

2014  •  Privacy Policy