Core admin: sysadmin toolkit

I will collectively refer to commands/tools usually provided with the base Unix installation as the standard toolset.

Here is a list of what we will see in this section:

These three utilities deal with space and memory.

df gives you an overview of the manner in which your hard disks are partitioned and how much space is used in each partition.

Typical usage:

$ df -h
Filesystem     Size    Used   Avail Capacity  Mounted on
/dev/wd0a     1006M   96.8M    859M    10%    /
/dev/wd1a      183G   32.0G    142G    18%    /home
/dev/wd0d      5.9G    4.8G    826M    86%    /root
/dev/wd0e      5.9G    2.4G    3.2G    43%    /usr
/dev/wd0f      2.0G    187M    1.7G    10%    /tmp
/dev/wd0h     87.9G   31.7G   51.8G    38%    /var
/dev/wd0g     78.7G   32.2M   74.8G     0%    /var/log
	

The du command displays the block system usage (greater than the actual file size) of a specified file or directory. If you specify a directory you will only get statistics on subdirectories. You must have the appropriate permissions to access the statistics.

An example where we want info on usage of a specified directory as well as the summary of the directory itself (the last line):

$ du -h /tmp/bnc
6.0K    /tmp/bnc/help
2.0K    /tmp/bnc/lang
2.0K    /tmp/bnc/log
2.0K    /tmp/bnc/menuconf/help
4.0K    /tmp/bnc/menuconf
2.0K    /tmp/bnc/motd
2.0K    /tmp/bnc/scripts/example
4.0K    /tmp/bnc/scripts
2.0K    /tmp/bnc/src
2.0K    /tmp/bnc/tools
250K    /tmp/bnc

The reason the last line (the directory) and the above lines (the subdirectories) do not add up is because there are some files under the directory that are not considered.

An example where we want system block usage of a directory only:

$ du -sh ~
44.0K   /home/petermatulis

The file program attempts to determine the nature of a file (its type). Possible types include:

• text
• executable
• data (other than the first two types)

In addition, the file command recognizes a plethora of well-known formats. It uses the /etc/magic file for this.

Syntax:

The whatis command will provide a very brief summary of the purpose of a command. Actually, it provides information available in the manual pages.

Syntax:

To get more details on a command, therefore, use the manual pages themselves via the man utility. The manual pages also contain information on non-executable files.

Syntax:

A useful feature of man is to have it search itself for manual entries matching a keyword:

Example:

$ man -k bash
bash (1) - GNU Bourne-Again SHell
bashbug (1) - report a bug in bash
	

The find command is one command you need to have good control over to be an efficient sysadmin. I will give examples of some useful forms.

To find a file with the given pattern, below the /usr directory, owned by the root user, and give a long directory listing of the matches:

To find a directory with the given pattern below the / directory, owned by group wheel, and apply the du command to the returned matches:

The above could of been done by piping the results to the xargs command:

To find any type of file, below the /home directory, with a size greater than 256 kB, and with permissions 775:

To find a file below /usr/local, modified within the last 48 hours, but contained within 4 directory levels (/usr/local being the first level):

To find a file below the / directory, accessed within the last minute, but contained lower than 1 directory level (/ being the first level):

This utility searches files for lines containing a match to a given pattern. I am using grep (GNU grep) 2.4.1 here

To search for the pattern "LaserJet" in the file /etc/magic while ignoring the case of the pattern:

To search for the pattern "LaserJet" in all files in the current directory and give just the number of matches found:

To search for the whole word "LaserJet" in the file /etc/magic and include the 2 preceeding lines and the 2 succeeding lines:

To search among multiple patterns (one per line) found in the file "grepstrings" in the file /etc/magic. Finding any one pattern will produce a match:

To search for a string containing metacharacters in the file /etc/magic:

To search for the string "LaserJet" in all files in the /etc directory (and its subdirectories) and include line numbers:

An inverse search. Only lines not containing the pattern "Laser Jet" in the file /etc/magic:

GNU grep has powerful regular expression capabilities. Here are a few examples.

Two equivalent ways to search for lines beginning with a digit in the file /etc/magic:

To search for 5 characters in the file /etc/magic. The first character is alphanumeric at the beginning of a word, followed by any 2 characters, followed by the digit 4, followed by any character at the ending of the word:

To search for a word in the file /etc/magic that begins with 7 or more digits in the range of 1 to 9:

We can create a situation where one or more expressions finds a match. Below we search for the strings "$5000" and/or "$10000" in the file /etc/magic. Notice how we force grep to act as egrep by employing the '-E' switch. egrep is better suited for regular expressions:

We can also use the output of another command as the input for grep using a pipe:

Above we're taking the output of the dmesg command and filtering it through to grep. Here, grep searches for any lines containing the strings "isa" and/or "irq". I use this one often when troubleshooting network interfaces and modems.

To specify that a line must contain both isa and irq we pipe into a second grep:

These three utilities are used to quickly locate files. Each one is implemented in slightly different ways.

locate searches a database (OpenBSD default is /var/db/locate.database but it can be specified with the "-d" switch) for all pathnames which match the specified pattern. The database is recomputed periodically (usually weekly or daily), and contains the pathnames of all files which are publicly accessible.

The simplified syntax for locate is:

Use locate for files that do not have the executable attribute or for strings found anywhere in the pathname.

For example,

$ locate ml/8

The output,

/usr/local/share/doc/aspell/man-html/8_How.html

which takes a name (or list of names) and returns the absoulte filenames of commands that would be executed had these names been given as commands. This utility depends completely on the PATH environmental variable; if it is not defined then it will fail unless the name is given as a correct relative or absolute pathname. Only the paths found in this variable will be searched so there may well be executables on the filesystem that are not found. Also, only the first match in the PATH is given as output (use the "-a" switch to display all matches).

The simplified syntax for which is:

Use which for files that have the executable attribute.

For example,

$ which man ls ps

The output,

/usr/bin/man /bin/ls /bin/ps

whereis performs the same task as which. The only difference is in the paths it searches. Whereas which uses the paths found in the PATH variable whereis uses those returned by the sysctl(8) utility for the "user.cs_path" string.

On a random system here are the differences in these paths:

$ sysctl -a user.cs_path
user.cs_path=/usr/bin:/bin:/usr/sbin:/sbin:/usr/X11R6/bin:/usr/local/bin
$ echo $PATH
/home/petermatulis/bin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/usr/games:.

A basic but indispensible command. I include it here because it is one of the first commands a new user encounters but then never bothers to learn any of its many useful options.

To show a long listing; show hidden (dot) files but supress the current and parent entries (. and ..); and apply the command to all subdirectories as well:

To show a long listing; show all hidden files; sort by size (largest first); and show size in "human readable" form:

Note: I found the "h" option available only with Red Hat.

To show a long listing; show all hidden files; show the time a file's status was last changed; and express user and group in numerical format:

Notice how the "n" option implies a long listing.

To show a long listing; show all hidden files; show the time a file was last accessed; and reverse the sorting order (here it will show latest accessed file last):

To show a long listing; show all hidden files; sort by modification time (latest first); and show in detailed time format:

Note: Only on OpenBSD does the "T" switch assume the above meaning. On Red Hat, you specify "--full-time".

Determine how many excutables reside in the /usr directory (and its subdirectories):

Above, the files that have their executable bit set (by either owner, group, or other) are listed due to the "F" switch where such files are identified by a star (asterisk). The output to this command on my OpenBSD 3.3 system is:

Other types of files are identifiable using the "F" switch. The OpenBSD man page:

-F      Display a slash (`/') immediately after each pathname that is a
             directory, an asterisk (`*') after each that is executable, an at
             sign (`@') after each symbolic link, a percent sign (`%') after
             each whiteout, an equal sign (`=') after each socket, and a ver-
             tical bar (`|') after each that is a FIFO.

Sed is known as a stream editor and is shipped with all Unix variants. Its chief purpose is to edit text files by performing deletions and substitutions. This editing is based on a rule (or rules) specified on the command line or in a file. This makes it a non-interactive editor. This also allows shell scripts to utilize sed to perform repetitive tasks. The input text is read from a file or a pipe (output from another command) and the modified results are sent to standard output or to a file.

Before getting bogged down in details let's look at sed in action:

$ cat file
The fox jumped over the stream.
$ sed -e 's/fox/horse/' file
The horse jumped over the stream.
$ sed -e 'd' file
..............................

What we did there was replace the word "fox" with the word "horse" and the results were automatically sent to standard output (the screen). Then we deleted every line (there is only one) in the file so there is null as output (the last line is a blank line). An important point to note is that the original file remains unchanged. This is a trivial example. The power of sed becomes apparent when either mass editing of files is required or when editing of files (on any scale) is needed at predetermined times.

Sed syntax

There are four ways to invoke sed:

1. sed -e 'command1' -e 'command2' -e 'command3' file
2. {shell command} | sed -e 'command1' -e 'command2'
3. sed -f sedscript.sed file
4. {shell command} | sed -f sedscript.sed

{ sed commands on the command line with input from a file }
{ sed commands on the command line with input from output of another command }
{ sed commands from a file with input from a file }
{ sed commands from a file with input from output of another command }

Input is always processed line by line. So if the input is a file consisting of fifty lines then sed is applied fifty times (with each line being processed independently). And as explained above, the "command" may be a deletion or a substitution. When multiple commands are used each successive command acts upon the result of the preceeding command. The text that sed is currently working on is called the pattern space.

Deletion

The format of the delete command is as follows:

In sed, addresses represent lines. If one address is given, then the command is applied to lines containing that address. An address can be either a regular expression enclosed by forward slashes /regex/ , or a line number . The "$" symbol can be used in place of a line number to denote the last line. If two addresses are given, then the command is applied to all lines between the two lines that match the pattern.

Here we are deleting the first six lines of the file "test":

Below we remove all lines ending with the string "taoist.". Notice that I escape the period. I do this to be explicit about the ending period because if I don't do this then "." will act as a regex (any one character):

We can only delete entire lines using the delete command. In order to delete parts of a line (like a word) you must use the substitution command and use a null character as the replacement.

Substitution

The format of the substitute command is as follows:

The pattern (not the pattern space) is some regular expression (or simply a string or word) and the replacement is a string we want to replace the pattern with. sed regular expressions are essentially the same as those used with grep.

The flags can be any of the following:

• n replace nth instance of pattern with replacement
• g replace all instances of pattern with replacement
• p write pattern space to StdOUT if a succesful substitution takes place
• w file write pattern space to file if a succesful substitution takes place

For some reason we cannot use the "w" option with the delete command. If we want to send the results to a file we need to use redirection:

This next example tells sed to operate on any line in the file "test" contaning the word "peter" and to erase the second instance of the word "This" (via the use of a null character):

This tells sed to operate on lines 3 through 15 (inclusive) in the file "test" and to replace all instances of the word "This" with the word "That" and send the results to the file "test_modified":

Let's move on to something more mature. My task is now to perform a mass substitution to all my HTML documents. For instance, I want to change an image that resides on all my pages, say, "email.jpg" to "email.gif". The first thing to notice is that I am now dealing with multiple files whereas before I always dealt with single files. In such a case I need to write a script (call it "replace.sh") that contains a reiteration (a loop):

#! /bin/sh

TMP=/tmp/replace.sh.tmp

find $1 -type f -name "*$2*" | while read i
  do
     cat $i | sed -e "s/$3/$4/g" > $TMP
     mv $TMP $i
  done
    

I am employing four positional variables here. The second form of the sed syntax given earlier is being used. Also notice how I need to use double quotes in order to expand my variables.

Here is how I would invoke the script (assuming it resides in my PATH; that all HTML documents are stored under the /www directory; and that the documents contain the string "html" in their filenames):

Caution: Before using a sed command on existing documents ensure you have done enough testing. This is especially true if you are working with multiple files.

More sed information:

I will collectively refer to commands/tools not necessarily provided with the base Unix installation as the custom toolset.

Here is a list of what we will see in this section:

The man page describes it like this:

Screen is a full-screen window manager that multiplexes a physical terminal between several processes (typically interactive shells).

It is also useful as an educative tool since multiple users can attach their login sessions to existing shells. The possibility of remote connections (typically via SSH) amplifies the power of this last feature which should naturally be dampened with the appropriate security measures.

It is invoked very simply:

At this point all commands used to control screen itself must be prepended with the default C-a (Ctrl-a) keystroke. Otherwise you can proceed as you normally would on the command line.

Basic usage

Here are some rudimentary commands to get you started. The commands provided below are the defaults for my installation. YMMV.

creating
The first action you will want to take is to create some extra screen windows. This is analogous to tabbed internet browsing. The command to create another window is c (create). Hence:

selecting
To select a window use n (next) and p (previous). Each window is also assigned a number that can be used instead. A third method is via a menu: ". Yet a fourth, and probably the easiest, way is <SPACE> (the SPACEBAR) which scrolls through the windows in order.

splitting
Another popular feature is to split the current window into two regions (horizontal splitting; one on top of the other). This is accomplished with the S (split) command:

To switch over to the other region use <TAB> (the TAB key). At first there will be nothing in the new region. Just use the selection commands to bring up a window once you're over there. As of April 2006, there is talk of implementing vertical splitting.

To remove the current region: X.

naming
It is useful to assign a name to each window. Do this with the A command.

locking
To lock your entire screen session do x. To unlock you will need to supply the account password of the user who first invoked screen.

closing
To close the current window do K.

quitting
To stop screen completely and kill all its windows and corresponding processes do \.

help
To get online help use ?.

Intermediate usage

detach
To detach the screen session the command is d. It is very important to realize that the session is still alive and all associated processes continue to run. This is true even after you log out.

attach
To attach to a screen session you obviously need one to attach to. To view current sessions use this command at a normal shell prompt:

To attach to a single-existing session simply do:

You will need to specify which session you want to attach to if there are more than one. This is done by adding the argument [pid].tty.host to the above command.

To attach to an already attached session, that is, to share a session, use x instead of r.

copy, paste, and the scrollback buffer
One thing that may be seen as a hindrance is the fact that you lose your terminal's ability to scroll back to see previous console work. That is, you no longer have access to your terminal's scroll buffer. To turn on screen's buffer use [ and to turn it off use ]. Connected to this is screen's copy & paste ability. When the buffer is on use your vi skills to position the cursor at the beginning of the desired text and hit ENTER (no C-a required). Then move the cursor to the end of the desired text and hit ENTER again. Now when you are ready to paste simply turn off the buffer.

Configuration

Screen has a global configuration file and a user-based configuration file. Their default locations are /usr/local/etc/screenrc and ~/.screenrc respectively. Here is a sample:

# Turn off the splash screen
startup_message off

# Screen commands are now prefixed by the "`" character (not C-a)
escape ``

# Increase the default scrollback buffer from 100 to 3000
defscrollback 3000

# Set up a custom status line
hardstatus alwayslastline
hardstatus string '%{= wb}%-Lw%{= bw}%50> %n%f* %t%{= wb}%+Lw%< %{= bw}%-=%D, %M %d, %Y %c%{-}'

# Remove (by null assignment) some default key bindings
bind '\'
bind 'k'

# Create some custom key bindings
bind 'q' quit   # Quit Screen altogether (running programs are killed)

# Start two windows
screen -t SECOND 2
screen -t FIRST 1

Here we will cover utilities pertaining to network matters. Some security software will be included here.

Here is a list of what we will see in this section:

The arp program displays and modifies the Internet-to-Ethernet address translation tables used by ARP. These tables are flushed regularly.

spider.pmatulis.homeunix.net (192.168.1.40) at 00:a0:4b:03:e0:1d on vr0
daffy.pmatulis.homeunix.net (192.168.1.70) at 00:c0:4f:94:64:10 on vr0
    

The arp utility does not actively seek out IP/MAC mappings. It merely displays its current information. The following tool makes up for that.

The arping utility allows one to easily determine a MAC address from an IP address. It does this by "pinging" a host on the local subnet by manually sending an ARP request message (who-is query ethernet broadcast). I say "manually" because the arp protocol is normally employed in an automatic fashion. This occurs whenever two hosts need to communicate. Either way, the result is always the MAC (or hardware) address of the specified host.

Here we specify raw output, the count, and which interface to use:

The raw switch (r) is to display only the MAC address and nothing else.

You may also use this tool to ping a MAC address itself (in which case a directed broadcast ICMP Echo request is sent) to find the associated IP address:

Ifconfig is used to configure and inspect the status/configuration of network interfaces. It is launched automatically at boot time to set up interfaces as necessary.

This utility is indespensible. You will always use it as a network administrator.

The most basic usage is to view the current configuration of a (or all) network interface(s). Below we run ifconfig without any options (to display information for all interfaces):

eth0      Link encap:Ethernet  HWaddr 00:C0:4F:AC:12:3B
          inet addr:192.168.1.32  Bcast:192.168.1.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2062267 errors:12137 dropped:0 overruns:0 frame:22225
          TX packets:1391100 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:100
          RX bytes:651253275 (621.0 Mb)  TX bytes:1151145914 (1097.8 Mb)
          Interrupt:11 Base address:0xdc80

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:503 errors:0 dropped:0 overruns:0 frame:0
          TX packets:503 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:49818 (48.6 Kb)  TX bytes:49818 (48.6 Kb)
    

Let us bind a second IP address to our eth0 card:

$ ifconfig eth0:0 192.168.1.33
# route add -host 192.168.1.33 eth0:0
$ ifconfig

eth0      Link encap:Ethernet  HWaddr 00:C0:4F:AC:12:3B
          inet addr:192.168.1.32  Bcast:192.168.1.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2067418 errors:12139 dropped:0 overruns:0 frame:22229
          TX packets:1393042 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:100
          RX bytes:652603405 (622.3 Mb)  TX bytes:1152071057 (1098.7 Mb)
          Interrupt:11 Base address:0xdc80

eth0:0    Link encap:Ethernet  HWaddr 00:C0:4F:AC:12:3B
          inet addr:192.168.1.33  Bcast:192.168.1.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:11 Base address:0xdc80

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:504 errors:0 dropped:0 overruns:0 frame:0
          TX packets:504 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:49898 (48.7 Kb)  TX bytes:49898 (48.7 Kb)
    

Here we reconfigure the eth0 interface:

eth0      Link encap:Ethernet  HWaddr 00:C0:4F:AC:12:3B
          inet addr:192.168.1.32  Bcast:192.168.1.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2062267 errors:12137 dropped:0 overruns:0 frame:22225
          TX packets:1391100 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:100
          RX bytes:651253275 (621.0 Mb)  TX bytes:1151145914 (1097.8 Mb)
          Interrupt:11 Base address:0xdc80

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:503 errors:0 dropped:0 overruns:0 frame:0
          TX packets:503 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:49818 (48.6 Kb)  TX bytes:49818 (48.6 Kb)
    

Now we view the new configuration of eth0 (notice how we specify this interface):

eth0      Link encap:Ethernet  HWaddr 00:C0:4F:AC:12:3B
          inet addr:192.168.1.34  Bcast:192.168.1.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2069039 errors:12143 dropped:0 overruns:0 frame:22235
          TX packets:1394113 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:100
          RX bytes:652998849 (622.7 Mb)  TX bytes:1152333719 (1098.9 Mb)
          Interrupt:11 Base address:0xdc80
    

Finally, this is how we activate or shut down an interface:

To view available and used services (connections):

Active Internet connections (including servers)
Proto Recv-Q Send-Q  Local Address          Foreign Address        (state)
tcp        0     40  192.168.2.50.22       192.168.1.140.2052     ESTABLISHED
tcp        0      0  *.22                   *.*                    LISTEN
tcp        0      0  *.3306                 *.*                    LISTEN
tcp        0      0  192.168.2.50.80       *.*                    LISTEN
tcp        0      0  *.139                  *.*                    LISTEN
Active Internet connections (including servers)
Proto Recv-Q Send-Q  Local Address          Foreign Address        (state)
udp        0      0  192.168.2.50.138      *.*
udp        0      0  192.168.2.50.137      *.*
udp        0      0  *.138                  *.*
udp        0      0  *.137                  *.*
udp        0      0  *.67                   *.*
udp        0      0  *.514                  *.*
    

This output tells us the following about the host:

TCP

• First line: there is an established connection with 192.168.2.50 on port 22 (ssh) from 192.168.2.100 on port 2052
• Second line: it is listening as well with all other interfaces on port 22 (ssh)
• Third line: it is listening with all interfaces on port 3306 (mysql)
• Fourth line: it is listening with 192.168.2.50 on port 80 (http)
• Fifth line: it is listening with all interfaces on port 139 (smbd; samba)

UDP (does not use states)

• First line: port 138 (smbd; samba) is available on 192.168.2.50
• Second line: port 137 (nmbd; samba) is available on 192.168.1.`00
• Third line: port 138 (smbd; samba) is available on all other interfaces
• Fourth line: port 137 (nmbd; samba) is available on all other interfaces
• Fifth line: port 67 (dhcpd) is available on all interfaces
• Sixth line: port 514 (syslogd) is available on all interfaces

Let's begin by doing a typical TCP scan:

Starting nmap V. 2.54BETA25 ( www.insecure.org/nmap/ )
Interesting ports on daffy.pmatulis.homeunix.net (192.168.2.50):
(The 1540 ports scanned but not shown below are in state: closed)
Port       State       Service
22/tcp     open        ssh
80/tcp     open        http
139/tcp    open        netbios-ssn
3306/tcp   open        mysql

Nmap run completed -- 1 IP address (1 host up) scanned in 18 seconds
    

We can then scan the UDP ports:

Starting nmap V. 2.54BETA25 ( www.insecure.org/nmap/ )
Interesting ports on daffy.pmatulis.homeunix.net (192.168.2.50):
(The 1449 ports scanned but not shown below are in state: closed)
Port       State       Service
67/udp     open        bootps
137/udp    open        netbios-ns
138/udp    open        netbios-dgm
514/udp    open        syslog

Nmap run completed -- 1 IP address (1 host up) scanned in 19 seconds
    

Compare the output of the last two commands to that which we got for the first netstat command we used.

We can specify what ports we wish to scan. This greatly increases the speed of execution:

The output is almost identical to our first TCP scan but it took only 1 second to be displayed whereas the wideopen scan took 18:

Starting nmap V. 2.54BETA25 ( www.insecure.org/nmap/ )
Interesting ports on daffy.pmatulis.homeunix.net (192.168.2.50):
Port       State       Service
22/tcp     open        ssh
80/tcp     open        http
139/tcp    open        netbios-ssn
3306/tcp   open        mysql

Nmap run completed -- 1 IP address (1 host up) scanned in 1 second
    

Note: The speed difference is due to a wideopen scan involving all 65535 ports on a host.

Below we scan multiple hosts and send the output to a log file:

In verbose mode, let's scan candyman on the ssh port and try to uncover what operating system it is running. We will use the stealth scan to avoid being detected:

Starting nmap V. 2.54BETA25 ( www.insecure.org/nmap/ )
Host candyman.pmatulis.homeunix.net (192.168.1.52) appears to be up ... good.
Initiating SYN Stealth Scan against candyman.pmatulis.homeunix.net (192.168.1.52)
Adding TCP port 22 (state open).
The SYN Stealth Scan took 0 seconds to scan 1 ports.
Warning:  OS detection will be MUCH less reliable because we did not find at least 1 open and 1 closed TCP port
For OSScan assuming that port 22 is open and port 35565 is closed and neither are firewalled
Interesting ports on candyman.pmatulis.homeunix.net (192.168.1.52):
Port       State       Service
22/tcp     open        ssh

Remote operating system guess: Linux Kernel 2.4.0 - 2.4.5 (X86)
Uptime 26.143 days (since Fri Jul  5 12:23:39 2002)
TCP Sequence prediction: Class=random positive increments
                         Difficulty=1817165 (Good luck!)
IPID Sequence Generation: All zeros

Nmap run completed -- 1 IP address (1 host up) scanned in 11 seconds
    

Now we scan a range of ports on daffy and slow down the scan to Sneaky level (15 second interval between sent packets):

Starting nmap V. 2.54BETA25 ( www.insecure.org/nmap/ )
Interesting ports on daffy.pmatulis.homeunix.net (192.168.2.50):
(The 115 ports scanned but not shown below are in state: closed)
Port       State       Service
22/tcp     open        ssh
80/tcp     open        http
139/tcp    open        netbios-ssn

Nmap run completed -- 1 IP address (1 host up) scanned in 1772 seconds

Whew, that took almost half an hour to complete.

Again I will use the utility that comes with Slackware 9.0.

Let's first examine the routing table:

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
candyman.pmatul *               255.255.255.255 UH    0      0        0 eth0
localnet        *               255.255.255.0   U     0      0        0 eth0
loopback        *               255.0.0.0       U     0      0        0 lo
default         dr_snuggles.pma 0.0.0.0         UG    1      0        0 eth0
    

Let us disable name resolution:

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
192.168.1.32    0.0.0.0         255.255.255.255 UH    0      0        0 eth0
192.168.1.33    0.0.0.0         255.255.255.255 UH    0      0        0 eth0
192.168.1.0     0.0.0.0         255.255.255.0   U     0      0        0 eth0
127.0.0.0       0.0.0.0         255.0.0.0       U     0      0        0 lo
0.0.0.0         192.168.1.30    0.0.0.0         UG    1      0        0 eth0
    

In OpenBSD we would use:

One important step when configuring a host is to setup its default gateway. Currently it is 192.168.1.30. Let us change it to 192.168.1.34:

On OpenBSD I would use this slightly simpler command:

To delete the original default gateway (apparently you cannot change a default gateway) simply substitute del (delete for OpenBSD) for add.

To set up a route for a network address space we would do the following:

Omit the "gw" for OpenBSD.

Lastly, we delete a destination:

This says to deny access to all servers and from all clients. The syntax is server:ip address/netmask.

We then selectively allow clients access to a service defined and activated in /etc/inetd.conf. For example, if ftpd is available then hosts.allow may contain this line in order to give access to the 192.168.27.0 network:

We now need to modify our inetd.conf accordingly:

#ftp	stream	tcp	nowait  root	/usr/libexec/ftpd	ftpd -US
ftp	stream	tcp	nowait  root	/usr/libexec/tcpd	ftpd -US

So we're telling tcpd to start ftpd only if authentication clears.

Not well known is the ability to use tcp wrappers with SSH even if sshd is not run from inetd. To grant the single address 192.168.1.63 access to sshd add this line to hosts.allow:

The corresponding line in inetd.conf would be:

ssh	stream	tcp	nowait	root	/usr/libexec/tcpd	sshd

Note: inetd needs to be sent a HANGUP signal whenever either control files or inetd.conf are changed.

tcpdmatch is the "tcp wrapper oracle". It predicts how the tcp wrapper would handle a specific request for a service. Cool huh? Here is the basic syntax:

Notice that we do not need superuser powers to run this. Here is a real example with output:

client:   hostname bureau.danville.ca
client:   address  192.168.1.63
server:   process  sshd
access:   granted

Finally we have the tcpdchk tool. This is a kind of diagnostic tool that can identify problems with your tcpd configuration and suggest remedies. We have used very simple configuration so on a real system such a tool can become very useful. Here goes:

Using network configuration file: /etc/inetd.conf

>>> Rule /etc/hosts.allow line 1:
daemons:  sshd
clients:  192.168.1.63/255.255.255.255
access:   granted

>>> Rule /etc/hosts.deny line 1:
daemons:  ALL
clients:  ALL
access:   denied

This output jives with the results we got back from the tcpdmatch command.

For instance, to see that a host is listening to port 80 (the HTTP port) we can do:

And we can do the same for any other port.

ping sends ICMP ECHO_REQUEST packets to a network host. Here we send twenty requests at a rate of one per second:

Now we send thirty requests. The first five are sent out as fast as possible while the remaining 25 are sent out at the default rate of one per second. All this in quiet mode:

Note: most systems require root privilege for this last command.

traceroute utilizes the IP protocol `time to live' field and attempts to elicit an ICMP TIME_EXCEEDED response from each gateway along the path to some host:

Whatmask can work in two modes. The first mode is to invoke it with only a subnet mask as the argument. In this mode Whatmask will echo back the subnet mask in four formats, plus the number of useable addresses in the range.

Example:

---------------------------------------------
       TCP/IP SUBNET MASK EQUIVALENTS
---------------------------------------------
CIDR = .....................: /27
Netmask = ..................: 255.255.255.224
Netmask (hex) = ............: 0xffffffe0
Wildcard Bits = ............: 0.0.0.31
Usable IP Addresses = ......: 30

So you can enter the netmask in either of the four formats to get its equivalent in the other three.

In the second mode, you include the IP address along with the netmask and it will echo back the following:

       - The netmask in the following formats: CIDR, Netmask, Netmask (Hex), Wildcard Bits

       - The Network Address

       - The Broadcast Address

       - The number of Usable IP Addresses

       - The First Usable IP Address

       - The Last Usable IP Address

Example:

------------------------------------------------
           TCP/IP NETWORK INFORMATION
------------------------------------------------
IP Entered = ..................: 192.168.1.63
CIDR = ........................: /27
Netmask = .....................: 255.255.255.224
Netmask (hex) = ...............: 0xffffffe0
Wildcard Bits = ...............: 0.0.0.31
------------------------------------------------
Network Address = .............: 192.168.1.32
Broadcast Address = ...........: 192.168.1.63
Usable IP Addresses = .........: 30
First Usable IP Address = .....: 192.168.1.33
Last Usable IP Address = ......: 192.168.1.62

One nice feature (in either mode) is that you can specify the netmask in any of the four modes and the utility will figure everything out on its own. So the last command could have also been written in these forms to get identical output:

This third and last segment will focus on software I have found to be useful but which does not fit into the first two catagories.

Here is a list of what we will see in this section:

This is one of the best relatively unknown tools I've ever found. Unfortunatly, it does not run on OpenBSD. Supported platforms include Linux versions 2.0 and up, on Alpha, x86 PCs, PC98, Macintosh PowerPC, and Sun hardware.

This is a hard disk tool that allows one to copy, move, and resize partitions that are currently in use on a live system without disrupting associated data.

We begin by looking at the partition table of the first ide drive:

Disk geometry for /dev/hda: 0.000-19092.937 megabytes
Disk label type: msdos
Minor    Start       End     Type      Filesystem  Flags
1          0.031     23.532  primary   ext2        boot
2         23.533  19092.875  extended
5         23.563   3098.474  logical   ext2
6       3098.505   3616.193  logical   ext2
7       3616.225   4133.913  logical   linux-swap
8       4133.944   4392.773  logical   ext2
9       4392.804  11750.668  logical   ext2
10     11750.700  19092.875  logical   ext2

As a trivial example, let's say we want to decrease the size of partition 9

The general syntax is as follows:

So the command could be:

Note: If you will be modifying the root parition or any swap spaces use a Parted boot floppy.

This utility lists information about the open files on a computer. It is versatile since almost everything on a UNIX system is a file.

Show files opened by MySQL:

Note: Using this form, it is not necessary to type the exact command. Just enough to identify it.

Here is the output:

COMMAND   PID     USER   FD   TYPE     DEVICE SIZE/OFF   NODE NAME
mysqld  16863 mysqladm  cwd   VDIR       0,20      512  32950 /usr/local (/dev/wd1e)
mysqld  16863 mysqladm  txt   VREG       0,20  1862568  28905 /usr/local (/dev/wd1e)
mysqld  16863 mysqladm  txt   VREG        0,3    61440 149810 /usr/libexec/ld.so
mysqld  16863 mysqladm  txt   VREG        0,4     4374  17285 /var/run/ld.so.hints
mysqld  16863 mysqladm  txt   VREG        0,3   687426 182436 /usr (/dev/wd0d)
mysqld  16863 mysqladm  txt   VREG        0,3    55094 182467 /usr (/dev/wd0d)
mysqld  16863 mysqladm  txt   VREG        0,3   390677 182464 /usr (/dev/wd0d)
mysqld  16863 mysqladm  txt   VREG        0,3    85720 182452 /usr (/dev/wd0d)
mysqld  16863 mysqladm    0r  VChr        2,2      0t0  12641 /dev/null
mysqld  16863 mysqladm    1w  VREG       0,20     5132  32970 /usr/local (/dev/wd1e)
mysqld  16863 mysqladm    2w  VREG       0,20     5132  32970 /usr/local (/dev/wd1e)
mysqld  16863 mysqladm    3u  IPv4 0xe065b408      0t0    TCP *:mysql (LISTEN)
mysqld  16863 mysqladm    4u  unix 0xe06438f0      0t0        /tmp/mysql.sock
mysqld  16863 mysqladm    5u  PIPE 0xeaa98214    16384        ->0xeaa981c8
mysqld  16863 mysqladm    6u  PIPE 0xeaa981c8    16384        ->0xeaa98214
mysqld  16863 mysqladm    7w  VREG       0,20     6949  32972 /usr/local (/dev/wd1e)
mysqld  16863 mysqladm    9u  VREG       0,20     2048  34795 /usr/local (/dev/wd1e)
mysqld  16863 mysqladm   10u  VREG       0,20     1756  34796 /usr/local (/dev/wd1e)
mysqld  16863 mysqladm   11u  VREG       0,20     2048  34792 /usr/local (/dev/wd1e)
mysqld  16863 mysqladm   12u  VREG       0,20     5624  34793 /usr/local (/dev/wd1e)

Nice huh?

Now my ps command shows me the following line:

I want to know all the open files associated with this process so I take note of the PID:

COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
getty   19195 root  cwd   VDIR    0,0      512      2 /
getty   19195 root  txt   VREG    0,3    20480 149797 /usr (/dev/wd0d)
getty   19195 root  txt   VREG    0,3    61440 149810 /usr/libexec/ld.so
getty   19195 root  txt   VREG    0,4     4374  17285 /var/run/ld.so.hints
getty   19195 root  txt   VREG    0,3    38246 182465 /usr (/dev/wd0d)
getty   19195 root  txt   VREG    0,3   598961 182435 /usr/lib/libc.so.28.3
getty   19195 root    0u  VChr   12,5     0t63  12507 /dev/ttyC5
getty   19195 root    1u  VChr   12,5     0t63  12507 /dev/ttyC5
getty   19195 root    2u  VChr   12,5     0t63  12507 /dev/ttyC5

Let's see what our old friend user bonehead is up to:

COMMAND   PID     USER   FD   TYPE DEVICE  SIZE/OFF   NODE NAME
sh       2646 bonehead  cwd   VDIR    0,5       512 113280 /home/bonehead
sh       2646 bonehead  txt   VREG    0,0    307200   1938 /bin/sh
sh       2646 bonehead    0u  VChr    5,1 0t5058390  12528 /dev/ttyp1
sh       2646 bonehead    1u  VChr    5,1 0t5058390  12528 /dev/ttyp1
sh       2646 bonehead    2u  VChr    5,1 0t5058390  12528 /dev/ttyp1
sh       2646 bonehead   10u  VChr    1,0       0t0  12640 /dev/tty
vi      24825 bonehead  cwd   VDIR    0,5       512 113280 /home/bonehead
vi      24825 bonehead  txt   VREG    0,3    282624 124863 /usr/bin/vi
vi      24825 bonehead  txt   VREG    0,3     61440 149810 /usr/libexec/ld.so
vi      24825 bonehead  txt   VREG    0,4      4374  17285 /var/run/ld.so.hints
vi      24825 bonehead  txt   VREG    0,3    257069 182439 /usr/lib/libcurses.so.8.0
vi      24825 bonehead  txt   VREG    0,3    598961 182435 /usr/lib/libc.so.28.3
vi      24825 bonehead    0u  VChr    5,1 0t5058390  12528 /dev/ttyp1
vi      24825 bonehead    1u  VChr    5,1 0t5058390  12528 /dev/ttyp1
vi      24825 bonehead    2u  VChr    5,1 0t5058390  12528 /dev/ttyp1
vi      24825 bonehead    3rW VREG    0,0         0  38402 /tmp/vi.W24825
vi      24825 bonehead    4u  VREG    0,4         0  30722 /var/tmp/vi.recover/vi.y24825
vi      24825 bonehead    5u  VREG    0,0         0  38403 / (/dev/wd0a)

Heh heh, the little bugger is editing a file with vi.

Now we will get into networking. Yes, we can do this. Let's see the open internet socket files related to the TCP protocol without host name resolution:

COMMAND   PID     USER   FD   TYPE     DEVICE SIZE/OFF NODE NAME
sshd     1707 pmatulis    5u  IPv4 0xe065b810      0t0  TCP 192.168.2.50:ssh->192.168.2.100:2372 (ESTABLISHED)
smbd     2114     root    9u  IPv4 0xe065b2b0      0t0  TCP *:netbios-ssn (LISTEN)
httpd    2867   webadm   16u  IPv4 0xe065b560      0t0  TCP 192.168.2.50:www (LISTEN)
httpd    6642   webadm   16u  IPv4 0xe065b560      0t0  TCP 192.168.2.50:www (LISTEN)
httpd    7529   webadm   16u  IPv4 0xe065b560      0t0  TCP 192.168.2.50:www (LISTEN)
sshd    11303     root    3u  IPv6 0xe065b000      0t0  TCP *:ssh (LISTEN)
sshd    11303     root    4u  IPv4 0xe065b158      0t0  TCP *:ssh (LISTEN)
smbd    11990 pmatulis   12u  IPv4 0xe065bc18      0t0  TCP 192.168.2.50:netbios-ssn->192.168.2.100:2247 (ESTABLISHED)
httpd   12043     root   16u  IPv4 0xe065b560      0t0  TCP 192.168.2.50:www (LISTEN)
sshd    14823     root    5u  IPv4 0xe065bac0      0t0  TCP 192.168.2.50:ssh->192.168.2.100:2052 (ESTABLISHED)
mysqld  16863 mysqladm    3u  IPv4 0xe065b408      0t0  TCP *:mysql (LISTEN)
sshd    24081 pmatulis    5u  IPv4 0xe065bac0      0t0  TCP 192.168.2.50:ssh->192.168.2.100:2052 (ESTABLISHED)
httpd   27634   webadm   16u  IPv4 0xe065b560      0t0  TCP 192.168.2.50:www (LISTEN)
sshd    31884     root    5u  IPv4 0xe065b810      0t0  TCP 192.168.2.50:ssh->192.168.2.100:2372 (ESTABLISHED)

Below we ask for the open files associated with ports 22 to 139 of the TCP protocol and all ports of the UDP protocol (all still without host name resolution):

COMMAND   PID     USER   FD   TYPE     DEVICE SIZE/OFF NODE NAME
sshd     1707 pmatulis    5u  IPv4 0xe065b810      0t0  TCP 192.168.2.50:ssh->192.168.2.100:2372 (ESTABLISHED)
smbd     2114     root    9u  IPv4 0xe065b2b0      0t0  TCP *:netbios-ssn (LISTEN)
httpd    2867   webadm   16u  IPv4 0xe065b560      0t0  TCP 192.168.2.50:www (LISTEN)
httpd    6642   webadm   16u  IPv4 0xe065b560      0t0  TCP 192.168.2.50:www (LISTEN)
httpd    7529   webadm   16u  IPv4 0xe065b560      0t0  TCP 192.168.2.50:www (LISTEN)
nmbd     7658     root    8u  IPv4 0xe065c700      0t0  UDP *:netbios-ns
nmbd     7658     root    9u  IPv4 0xe065c600      0t0  UDP *:netbios-dgm
nmbd     7658     root   10u  IPv4 0xe065c400      0t0  UDP 192.168.2.50:netbios-ns
nmbd     7658     root   11u  IPv4 0xe065c300      0t0  UDP 192.168.2.50:netbios-dgm
sshd    11303     root    3u  IPv6 0xe065b000      0t0  TCP *:ssh (LISTEN)
sshd    11303     root    4u  IPv4 0xe065b158      0t0  TCP *:ssh (LISTEN)
smbd    11990     root    5u  IPv4 0xe0696800      0t0  UDP 127.0.0.1:43301
smbd    11990     root   12u  IPv4 0xe065bc18      0t0  TCP 192.168.2.50:netbios-ssn->192.168.2.100:2247 (ESTABLISHED)
httpd   12043     root   16u  IPv4 0xe065b560      0t0  TCP 192.168.2.50:www (LISTEN)
sshd    14823     root    5u  IPv4 0xe065bac0      0t0  TCP 192.168.2.50:ssh->192.168.2.100:2052 (ESTABLISHED)
dhcpd   18274     root    7u  IPv4 0xe0649100      0t0  UDP *:bootps
sshd    24081 pmatulis    5u  IPv4 0xe065bac0      0t0  TCP 192.168.2.50:ssh->192.168.2.100:2052 (ESTABLISHED)
httpd   27634   webadm   16u  IPv4 0xe065b560      0t0  TCP 192.168.2.50:www (LISTEN)
syslogd 29094     root    5u  IPv4 0xe0645a00      0t0  UDP *:syslog
sshd    31884     root    5u  IPv4 0xe065b810      0t0  TCP 192.168.2.50:ssh->192.168.2.100:2372 (ESTABLISHED)

Let's explicitly request info on port 22 (ssh) involving host daffy and turn off service name resolution (as well as host name resolution):

COMMAND   PID     USER   FD   TYPE     DEVICE SIZE/OFF NODE NAME
sshd     1707 pmatulis    5u  IPv4 0xe065b810      0t0  TCP 192.168.2.50:22->192.168.2.100:2372 (ESTABLISHED)
sshd    14823     root    5u  IPv4 0xe065bac0      0t0  TCP 192.168.2.50:22->192.168.2.100:2052 (ESTABLISHED)
sshd    24081 pmatulis    5u  IPv4 0xe065bac0      0t0  TCP 192.168.2.50:22->192.168.2.100:2052 (ESTABLISHED)
sshd    31884     root    5u  IPv4 0xe065b810      0t0  TCP 192.168.2.50:22->192.168.2.100:2372 (ESTABLISHED)

What files do we have open in the /home directory?:

COMMAND   PID     USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
lsof     1758     root  cwd   VDIR    0,5      512 115217 /home (/dev/wd0f)
sh       2902 pmatulis  cwd   VDIR    0,5      512 109440 /home (/dev/wd0f)
lsof    10891     root  cwd   VDIR    0,5      512 115217 /home (/dev/wd0f)
ksh     26362     root  cwd   VDIR    0,5      512 115217 /home (/dev/wd0f)
sh      27635 pmatulis  cwd   VDIR    0,5      512 109440 /home (/dev/wd0f)

We can also narrow down our requests by using the "-a" (AND) switch. Examine the following session where we pipe the output to a second command to count the number of lines:

# lsof -c httpd | wc -l
     106
# lsof -c httpd -au root | wc -l
      22
# lsof -c httpd -au root -a /usr | wc -l
       9

So here is the output of that last command (we asked for open files associated with 1) the httpd process, 2) the superuser, and 3) that which reside in the /usr directory):

COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
httpd   12043 root  txt   VREG    0,3    61440 149810 /usr/libexec/ld.so
httpd   12043 root  txt   VREG    0,3   820851 182459 /usr (/dev/wd0d)
httpd   12043 root  txt   VREG    0,3    85720 182452 /usr (/dev/wd0d)
httpd   12043 root  txt   VREG    0,3   598961 182435 /usr/lib/libc.so.28.3
httpd   12043 root  txt   VREG    0,3    38246 182465 /usr/lib/libutil.so.7.1
httpd   12043 root  txt   VREG    0,3    20663  69244 /usr (/dev/wd0d)
httpd   12043 root  txt   VREG    0,3   135225 182725 /usr (/dev/wd0d)
httpd   12043 root  txt   VREG    0,3    55094 182467 /usr (/dev/wd0d)

One thing that I have always found extremely annoying is trying to unmount a partition (ex: cdrom drive) only to to be told that it is being used and will not unmount. Well we can now apply our lsof knowledge and request to be told what those files are that are stopping us:

zap is the OpenBSD interactive process killer. The equivalent utility for Linux is killall.

Below we see partial output to the ps command. There is a user editing a file called "test" with the vi program. After you see that we shut down the editor using the killall command:

.
.
.
25728 ?        S      0:00 /usr/sbin/sshd
25729 pts/0    S      0:00 -bash
25787 ?        S      0:00 /usr/sbin/sshd
25788 pts/1    S      0:00 -bash
25798 pts/1    S      0:00 vi test
25799 pts/0    R      0:00 ps ax

pmatulis@candyman:$ killall -i vi
Kill vi(25798) ? (y/n) y

Normally, using the kill command, we need to specify the actual PID (which is 25798). We used the 'i' switch to enter interactive mode.

The script utility is useful for when you want to capture the standard output of a terminal session. It is very easy to use:

Following the command all StdOUT is sent to the file /root/sessions/diagnostics_jan1_2004. Pressing "exit" (or Ctrl-D) ends the program (and the recording process). The resulting file can be viewed with any pager or text editor.

The tree utitlity is nice when you want to get a quick overview of a directory's structure.

This says to display the directories contained within the first two levels underneath the /etc directory. This is what I got on my system:

/etc
|-- X11
|   |-- app-defaults
|   |-- fs
|   |-- lbxproxy
|   |-- mwm
|   |-- proxymngr
|   |-- rstart
|   |-- twm
|   |-- xdm
|   |-- xinit
|   |-- xkb
|   |-- xserver
|   `-- xsm
|-- acpi
|   `-- events
|-- cron.daily
|-- cron.hourly
|-- cron.monthly
|-- cron.weekly
|-- cups
|-- dhcpc
|-- gtk
|-- gtk-2.0
|-- hotplug
|   |-- pci
|   `-- usb
|-- logrotate.d
|-- mail
|-- mutt
|-- news
|-- pango
|-- ppp
|   `-- plugins
|-- profile.d
|-- rc.d
|-- skel
|-- ssh
|-- ssl
|   |-- certs
|   |-- lib
|   |-- misc
|   `-- private
|-- tin
`-- vga

This says to show all the files (whose filenames begin with the letter "a") within the first three levels underneath /home; print file sizes; and send the output to file treetest:

Note that directories will always be shown and that the pattern feature cannot be used on them.