Linux Debugger

As the disk space utilization was so high in the server, the Administrator has removed few files from the server but still the disk utilization is showing as high. What would be the reason?

In Linux even if we remove a file from the mounted file system, that will still be in use by some application and for this application, it remains available. Its because file descriptor in /proc/ filesystem is held open..So if there are such open descriptors to files already removed, space occupied by them considered as used. You find this difference by checking them using the "df" and "du" commands. While df is to show the file system usage, du is to report the file space usage. du works from files while df works at filesystem level, reporting what the kernel says it has available.

You can find all unlinked but held open files with:

# lsof | grep '(deleted)'

This will list the filename which is open with the pid in which it is running. We can kill those Pids and which will stop these process and will recover the disk space responsible for this file.

lsof command

lsof is a command line utility which is used to list the information about the files that are opened by various processes. In unix, everything is a file: pipes, sockets, directories, devices, etc. So by using lsof, you can get the information about any opened files.
List processes which opened a specific file

# lsof /var/log/syslog

lists all open files belonging to processes owned by the user

# lsof -u username

Kill all process that belongs to a particular user

# kill -9 `lsof -t -u username

sed command

You might want to select specific lines of a file. sed, short for stream editor, is one way to do this. you want to combine multiple files that all had headers or to do a bulk find and replace a file.
insert a blank line above every line which matches "regex"

$ sed '/regex/{x;p;x;}'

change "scarlet" or "ruby" or "puce" to "red"

$ sed 's/scarlet/red/g;s/ruby/red/g;s/puce/red/g'

 grep command

grep is a command used to search text or searches the given file for lines containing a match to the given strings or words. By default, grep displays the matching lines.

print network connection used by firefox

# netstat -pltnu | grep firefox

inodes in Linux

An inode is a data structure on a filesystem on Linux and other Unix-like operating systems that stores all the information about a file except its name and its actual data. 

Inode structure of a directory consists of a name to inode mapping of files and directories in that directory.In a directory, You can find the inode number corresponding to the files using the command ls -i 

Sticky bit : Sticky bit was used on executables in linux so that they would remain in the memory more time after the initial execution, hoping they would be needed in the near future. But mainly it is on folders, to imply that a file or folder created inside a stickybit enabled folder could only be deleted by the owner. A very good implementation of sticky bit is /tmp , where every user has write permission but only users who own a file can delete them. 

 786727 -rw------- 1 root root 4226530 May 29 13:17 sudo.log
786437 -rw-------. 1 root root 32640 Jun 23 20:11 tallylog
786440 -rw-rw-r--. 1 root utmp 276096 Jul 20 06:45 wtmp
786741 -rw------- 1 root root 9653 Jul 17 09:38 yum.log 

The other way we can get the inode details of a file by using the stat command.

Usage : # stat

Example :

-sh-4.1$ stat note.txt
File: `note.txt'
Size: 4 Blocks: 8 IO Block: 4096 regular file
Device: fd05h/64773d Inode: 8655235 Links: 1
Access: (0644/-rw-r--r--) Uid: (69548/nixuser) Gid: (25000/ UNKNOWN)
Access: 2014-06-29 15:27:56.299214865 +0000
Modify: 2014-06-29 15:28:28.027093254 +0000
Change: 2014-06-29 15:28:28.027093254 +0000

 cache in Linux

 Cache in Linux memory is where the Kernel stores the information it may need later, as memory is incredible faster than disk.

Kernels 2.6.16 and newer provide a mechanism to have the kernel drop the page cache and/or inode and dentry caches on command, which can help free up a lot of memory

To free pagecache:

# echo 1 > /proc/sys/vm/drop_caches

To free dentries and inodes:

# echo 2 > /proc/sys/vm/drop_caches