COMPUTER AND NETWORK
SYSTEM ADMINISTRATION
Summer 1996 - Lesson 18
Disk Space Management
A. Disk Hog detection
1. "quot"
- quota summarizes ownership of files on a
particular file system
- doesn't summarize a user's file for the whole
system
- quot -c
> shows number of files of each size
> example below: there are 773 files that use
3 1K blocks
access# quot -c /real/cs18
/dev/sd1b (/real/cs18):
0 344 0
1 3740 3740
2 1366 6472
3 773 8791
4 528 10903
5 343 12618
6 254 14142
...
472 1 117687
480 1 118167
488 1 118655
499 20 142087
- quot -v
> displays three columns containing the number of blocks
not accessed in the last 30, 60, and 90 days
access# quot -v /real/cs18
/dev/sd1b (/real/cs18):
7810 juliano 0 0 0
6534 nicolaou 0 0 0
6366 langley 0 0 0
6231 anderson 0 0 0
5845 eaddy 0 0 0
5474 gieger 0 0 0
> why this result? based on access time which is touched
by the clean program
- quot -f
> display count of number of files as well as space owned
by each user
access# quot -f /real/cs18
/dev/sd1b (/real/cs18):
blocks files user
-----------------------
7810 710 juliano
6534 608 nicolaou
6366 890 langley
6231 263 anderson
5845 286 eaddy
5474 290 gieger
- quot -h
> estimate the number of blocks in the file
> this doesn't account for files with holes in them
access# quot -h /real/cs18
/dev/sd1b (/real/cs18):
13586 gieger
7778 juliano
6470 nicolaou
6318 langley
6071 anderson
5773 eaddy
> how can you explain the discrepancies?
> answer: files with holes
-rw-r--r-- 1 gieger 8421808 Mar 23 20:14 core
> this core file only uses 72 blocks, not 8000 blocks
> du core
72 core
> why is juliano's count smaller?
> answer: quot uses file size to estimate number of
blocks
2. du
- du counts blocks in a particular directory
- doesn't care who the owner is
- may give flag to recursively call subdirectories
- some "du"s report in 512-byte blocks, some in 1024-byte
blocks; usually there's a "-k" option to map to 1024-byte
blocks (see table 26.1, page 621)
> example: du -s juliano
7810 juliano
> note that this agrees with quot without the -h option
3. how can you know the total file usage of a user?
- this is difficult if a user has files on several
partitions
- also, if system supports non-root "chown" command
- that is, if you can give your files away then
du will not be accurate
- I assume that the files inside a user's home directory
are owned by the user which is also the name of the
directory
- here's a script that summarizes file ownership for a class
of users and mails the top users to me:
#!/bin/csh
/usr/bin/du -s /home/cs18/majors/* > /tmp/du.out.majors;
/usr/bin/du -s /home/cs19/majors/* >> /tmp/du.out.majors;
/usr/bin/du -s /home/cs20/majors/* >> /tmp/du.out.majors;
sort -r -n +0 /tmp/du.out.majors | head -8 |
mail -s majors-home-directories jtbauer
- running this via cron nightly locates users whose quotas
weren't turned on and tells you that quotas are running correctly
B. Cleaning the filesystem
1. user files
- the "find" command provides a number of options that
can be used to clean up 'uneccesary' files
- here's the "skulker script" that CompSci uses
find $1 \( -name '*~' -o -name '.*~' -o -name '*.log' -o -name
'*.aux' -o -name '*.dvi' -o -name 'core' -o -name
'*.BAK' -o -name '*.CKP'\)
-a \( -mtime +3 \) -print -exec rm {} \;
- removes named files starting at $1 which have not been
modified in 3 days
> {} means replace with current path
2. /tmp directories
- cleaning /tmp can be problematic
- used to do:
rm -r /tmp; mkdir /tmp; chmod 1777 /tmp
- but users are logged in at all times of day now
- Solaris keeps some /tmp files
- find /tmp -mtime +1 -exec rm {} ';'
> get files, not dirs
- find /tmp -mtime +1 -exec rm -r {} ';'
> misses dirs starting with .
- cd /tmp; find . ! -name . -type d -exec /bin/rm -rf {} ';'
> gets dirs starting with .
> must cd to /tmp first or /tmp will be deleted
B. Quotas
1. setting up quotas
- compile into kernel of file server that maintains the
file system you want to set up quotas on
> quotas are only checked by the file server
> an NFS request gets checked when the request gets
to the file server
> quotas are checked by the file system when
the allocation routine for a new block is called
> if the user is below quota the allocation request is
permitted and the new block is added to their usage
> SunOS 4.x config is easy:
options QUOTA # disk quotas for local disks
> Linux 2.0.0 has a quota kernel module
- create a file named 'quotas' in the root of each
file system for which you want quotas
> quotas won't work without this file
> touch quotas
> it will contain the quota limits for each user
> we'll come back to this file in a minute
- mount each file system with quotas
> for example, in SUNOS modify /etc/fstab to have
quotas option:
/dev/sd1a /real/cs17 4.2 rw,quotas 1 3
/dev/sd1b /real/cs18 4.2 rw,quotas 1 4
/dev/sd1d /real/cs19 4.2 rw,quotas 1 5
/dev/sd1e /real/cs20 4.2 rw,quotas 1 6
> this option adds a pointer in the mount table
to the quotas file for the specified file system
/real/cs17 -----> /real/cs17/quotas
- turn quotas on with the command:
quotaon -a
> this is usually placed in a bootup script
SunOS 4.x: /etc/rc
> the -a option says read /etc/fstab and turn on quotas
for all file systems that are mounted with the quotas
option
- quotacheck
> this command walks throught he file system and cleans
up any inconsistencies between the quotas file and
the user's actual used space
> usually run at boot time
> the option -a catches all file systems mounted with
quotas option
2. using quotas to manage space
- quotas reduce file system performance
- quotas are set for each file system
> if a user has space allocated on more than
one file system the quotas must be established on
each one
> if you move a user's home directory then quotas
must be manually changed
- the default is 'unlimited'
- the nice feature of using quotas is that the
'system' tells the user they are out of space
rather than you
- use "du" scripts as outlined above to supplement
quotas since so many things can go wrong
- use the edquota command to modify quotas
> edquota juliano
fs /real/cs20 blocks (soft = 0, hard = 0) inodes (soft = 0, hard = 0)
fs /real/cs19 blocks (soft = 0, hard = 0) inodes (soft = 0, hard = 0)
fs /real/cs18 blocks (soft = 8000, hard = 8100)
inodes (soft = 1200, hard = 1210)
fs /real/cs17 blocks (soft = 0, hard = 0) inodes (soft = 0, hard = 0)
- must edit on the machine which physically houses the file systems
- can be tedious unless you use the prototype flag
cd
edquota - say juliano
edquota -p juliano `ls`
3. the quota command
- a user can check his or her own quotas
- recall that quotas are implemented on the file server
- most users don't have login access to the file servers
- many users don't even know on which file system or machine
their home directory resides
- to facilitate the retrieval of quota information the
quota invokes the rquotad or rpc.quotad daemon on each
remote machine for which there is a mounted file system
- the file system in which the user's home directory resides
is usually mounted if they type quota
- but, as root, you may want to check the quota of a user
- this will only work if the user's home directory is mounted
- example:
quota -v juliano
Disk quotas for juliano (uid 29918):
Filesystem usage quota limit timeleft files quota limit
cd /home/cs18
quota -v juliano
Disk quotas for juliano (uid 29918):
Filesystem usage quota limit files quota limit
/home/cs18 7810 8000 8100 710 1200 1210
4. repquota
- this command summarizes the quotas for a file system
access# repquota /real/cs17
Block limits File limits
User used soft hard timeleft used soft hard
timeleft
sweeny 1559 3000 3100 170 300 310
labbe 565 3000 3100 86 300 310
gentry 1777 3000 3100 154 300 310
little 2975 3000 3100 308 300 310 EXPIRED
agbunag 3099 3000 3100 EXPIRED 58 300 310
C. Data Compression
- Can use gzip/gunzip, compress/uncompress/zcat, pack/unpack
- Table 26.2, page 622 shows comparison of algorithms
- Don't forget GNUtar has useful "z" option to do the zip/unzip
automatically
D. Moving files around
- Common method is the "tar pipe":
# cd OLDPLACE
# tar cf - . | (cd NEWPLACE; tar xf - )
Check to see if files are all there and owned correctly, then
# cd OLDPLACE/..
# rm -rf directoryname-you-just-tarred-over
- Can also use "cp -r", if the particular UNIX supports it.
- Can use "mv" or "mvdir", on some UNIXes, even across filesystems.
(does an implicit "cp" and "rm")
E. Disk space crisis management
- File systems seem to always hover around 95% full
- Occasionally they fill up!
- As sysadmin, must be able to sometimes quickly recover
disk space
- First find the offending process and kill/suspend it.
(note that the disk may be filling up from an NFS client,
so finding the process can be problematic)
- Next, move files or remove unneeded files:
/tmp, /usr/tmp, /var/tmp
~{all users}/.netscape/cache/*
Run your "skulker" manually to remove old "core", etc.
Various log files that can be truncated:
cat < /dev/null > /var/adm/messages
daemon logs, such as httpd access_log, etc.
Old crash files (memory images):
/usr/crash, /var/crash
Useful to start looking for the "big files" in spool
directories:
cd /usr/spool (or, cd /var/spool)
du -sk *