写这篇文章是因为客户有一套数据库，AIX 6.1，数据文件存放在文件系统中，在使用DROP TABLESPACE UNDO INCLUDING CONTENTS AND DATAFILES删除了UNDO表空间之后，文件系统空间没有得到释放。如何解决问题就成了写这篇文章的初衷。

    空间没有释放我们可能是通过df命令看确认的，当我们用du去扫描目录的大小可能会发现df和du两个命令看到的空间使用情况是不同的，可能差别很大，下面这篇MOS文章说明了原因：

In this Document

 

Symptoms

Cause

Solution

 

Applies to:

Linux OS - Version Oracle Linux 4.4 and later

Linux x86-64

Linux x86

Linux Kernel - Version: 4.4 to 5.3

 

 

Symptoms

The 'du' (/usr/bin/du) and 'df' (/bin/df) command output displays conflicting space utilisation values, for example:

# df -k /

Filesystem    1k-blocks     Used  Available Use% Mounted on

/dev/sda6       9288792  8672768     144120 99% /

 

# du -xsh /

2.1G /

 

In the example above, 'df' reports 8.6 Gb to have been used on the root (/) filesystem, whereas 'du' reports only 2.1 Gb to have been used.

 

Cause

The 'df' command reports how many disk blocks are used, whilst 'du' traverses the filesystem and reports the actual number of blocks used (directory by directory), including any relating to files in use by processes.

 

In most cases, space utilisation values returned from 'df' and 'du' will be consistent. However, the potential exists for a running process to delete a large file, say. In this instance, according to 'du', the large file no longer exists, so the blocks associated with the deleted file are not reported. With the process still running, and with an open file descriptor still held against the deleted file, 'df' continues to track and report all disk blocks used, including those associated with the deleted (phantom) file. In this situation, the disk space associated with the deleted file will only be relinquished back to the system when the process completely releases the deleted file's descriptor or the process terminates (either gracefully or killed).

Solution

The solution is to identify and stop (or kill) the process that continues to hold a file descriptor open for the deleted file. To do so, run the lsof command (/usr/sbin/lsof | grep deleted) as root to identify the holding process, for example:

 

# lsof | grep deleted

COMMAND     PID   USER    FD  TYPE  DEVICE      SIZE       NODE NAME

cannaserv  3825  canna    0u   CHR   136,0                    2 /dev/pts/0 (deleted)

vmware     4295   root    6u   REG   253,0      6770   13074503 /tmp/vmware-root/ui-4295.log (deleted)

vmware-re  4316   root    6u   REG   253,0      6770   13074503 /tmp/vmware-root/ui-4295.log (deleted)

vmnet-nat  4448   root    0u   CHR   136,0                    2 /dev/pts/0 (deleted)

vmware-se  4454   root    0u   CHR   136,0                    2 /dev/pts/0 (deleted)

gdm-binar  4506   root    0u   CHR   136,0                    2 /dev/pts/0 (deleted)

gconfd-2   5392   root   12wW  REG   253,0       609   13090818 /tmp/gconfd-root/lock/0t1188207163ut519551u0p5392r346479926k3219784492 (deleted)

vmware-vm  5822   root   57u   REG   253,0   6520832   13074477 /tmp/vmware-root/ram0 (deleted)

vmware-vm 16487   root   57u   REG   253,0  11153408   13074520 /tmp/vmware-root/ram0 (deleted)

kdeinit   17991   root   17u   REG   253,0     26712   13074524 /tmp/kde-root/khtmlcacheM7jXYb.tmp (deleted)

kdeinit   17991   root   18u   REG   253,0      5631   13074501 /tmp/kde-root/khtmlcacheZlJmda.tmp (deleted)

kdeinit   17991   root   21u   REG   253,0     44718   13074514 /tmp/kde-root/khtmlcacheH5m4lc.tmp (deleted)

 

In the example above, the 7th column in the output denotes the size of deleted files (in bytes). The 9th column denotes which file remains held open. The 1st and second columns denotes the process and pid that still holds the open file descriptor.

    之所以df和du命令看到的空间使用会有差别，原因在于du不统计已经删除的文件，df会统计已经删除的文件，但该文件依然被进程持有，只有等进程释放了该文件，df才不进行统计。通过lsof | grep deleted命令可以找出被删除的文件依然被进程持有的情况。

    通过上面这篇文章我们得知，在文件系统中删除某个大文件空间没有释放是因为依然有进程在持有它，如果找到相关进程就需要用到lsof命令和fuser命令，熟悉这两个命令非常的重要。

>lsof(list open files)是一个列出当前系统打开文件的工具。
>fuser用于标识访问文件或socket的进程信息。

   fuser也是我们平时常用的找进程的工具，但它不能列出详细的进程信息，更多的只是一个进程号，无法很好的过滤，在这个场景中lsof工具更加的合适。

另外，大家都知道，在Linux平台文件系统中，即使某个文件被删除，但是它如果任然有进程持有它，这个文件是可以被恢复的，这就是为什么数据库所有数据文件被删除，在实例没有停机的情况下可以完整恢复数据文件的原因，下面这篇MOS文章就讨论了在此情况下如何对数据文件进行恢复：

In this Document

 

Goal

Solution

References

 

Applies to:

Linux OS - Version Oracle Linux 5.0 and later

Linux x86-64

Linux x86

Goal

How to recover deleted file on ext3/ext4 filesystem, Which still has file descriptor opened.

Solution

A file in Linux is a pointer to an inode which contains the file data (permissions, owner and where its actual content resides on the disk).

Deleting the file removes the link, but not the inode itself.

If any other process has this file open then inode is not released for writing until the process releases it.

So if a process still has the file open, the data are there somewhere, even though the directory listing shows no files.

# ll

total 4

-rw-r--r-- 1 root root 27 Sep 16 05:19 test

# rm test

# lsof /opt/test/test

COMMAND     PID   USER   FD   TYPE DEVICE SIZE/OFF    NODE NAME

less 21353 root 4r REG 252,2 27 260360 /opt/test/test (deleted)      <<<<<<<<<<<<<</span>

Understanding output of "lsof" command:

COMMAND: Command using the file.

PID: PID of the file

USER: Owner of the file

FD: File descriptor. Different flags of File descriptor are as below:

#:      The number in front of flag(s) is the file descriptor number of used by the process to associated with the file

u:      File open with Read and Write permission

r:       File open with Read permission

w:      File open with Write permission

W:      File open with Write permission and with Write Lock on entire file

mem: Memory mapped file, usually for share library

TYPE: File type. Different flags of File type are as below:

REG - Regular file

DIR - Directory

DEVICE: major, minor number of the device where file resides.

SIZE/OFF: File size

NODE: inode number

NAME: File name

Now we know that process 21353 still has the file open, and the file descriptor is 4.

Now we can look into /proc and there will be a reference to the inode, from which the deleted file can be copied.

Following steps will help to recover the deleted files:

# ls -l /proc/21353/fd/4

  lr-x------ 1 root root 64 Sep 16 05:28 /proc/21353/fd/4 -> /opt/test/test (deleted)

# cp /proc/21353/fd/4 /opt/test/test.bkp

Now verify the content of the restored file.

Note: Don't use the -a flag with cp, as this will copy the (broken) symbolic link, rather than the actual file contents.

另外，找到是某个进程持有的文件，通过下面的方法可以看到这个进程相关的环境信息：

Checking the environment variables of ASM pmon process: It shows ORACLE_HOME is set to /oracle_grid/product/11.2.0.3/grid/ ( with 'slash' at the end )

   总结：对于此类问题，我们首先要明白为什么df和du在空间计算上有所差别，其次要熟悉lsof和fuser两个命令，找出继续持有文件的进程号，通过该进程号可以在/proc目录下恢复文件，查看进程的环境信息，甚至杀掉进程来释放空间。