Quick HOWTO : Ap01 : Miscellaneous Linux Topics

From Linux Home Networking
Jump to: navigation, search



From upgrades to headless operation to emulation - some topics, no matter how important, refuse to fit neatly into any one chapter. This appendix serves as a reference center for some of these need-to-know items.

Linux Security With TCP Wrappers

The TCP Wrappers package is installed by default on Fedora Linux and provides host-based security separate from that provided by a firewall running on the server itself or elsewhere.

The application relies on two main files:

  • /etc/hosts.allow: Defines the hosts and networks allowed to connect to the server. The TCP Wrappers enabled application searches this file for a matching entry, and if it finds one, then the connection is allowed.
  • /etc/hosts.deny: Defines the hosts and networks prohibited from connecting to the server. If a match is found in this file, the connection is denied. No match means the connection proceeds normally.

The /etc/hosts.allow file is always read first and both files are always read from top to bottom, therefore the ordering of the entries is important.

The TCP Wrappers File Format

The format of the file is as follows:

<TCP-daemon-name>  <client-list>  : <option>

This example allows all traffic from the and the networks and SSH from only two hosts, and All HTTP Web traffic is allowed. All other TCP traffic to the host is denied. Notice how the subnet masks can use the slash nomenclature or the dotted decimal format.

# File: hosts.allow
httpd:  ALL
# File: hosts.deny


Determining the TCP Daemon's Name

The easiest way of determining the name of a daemon is to use the ps command and then use grep to filter for the name of the service. Here, the example quickly determines the daemon name (/usr/sbin/sshd) for the SSH server process. Because TCP Wrappers only requires the program name and not the path, sshd therefore becomes the entry to place in the TCP-daemon-name column of the configuration file.

[root@bigboy tmp]# ps -ef | grep -i ssh
root     10053  �  1  0 Nov06 ?        00:00:00 /usr/sbin/sshd
root     14145 10053  0 Nov13 ?        00:00:02 sshd: root@pts/1
root     18100 14148  0 21:56 pts/1    00:00:00 grep ssh
[root@bigboy tmp]#

Additional TCP Wrappers Help

For a full explanation of all the options available, refer to section 5 of the man pages for hosts_access.

[root@bigboy tmp]# man 5 hosts_access

TCP wrappers is simple to implement, but you have to set them on every host. Management is usually easier on a firewall that protects the entire network.

Adjusting Kernel Parameters

Unlike many Linux applications that need to be restarted before their configuration parameters are read, many of the Linux kernel's parameters can be instantaneously activated and deactivated.

The Linux kernel stores many of its dynamic parameters in the /proc filesystem, which resides on a virtual RAM disk in memory for maximum performance.

Parameters are generally categorized by function within subdirectories of the /proc; information on IDE hard drives are located in the /proc/ide directory, for example, and general system parameters are located in the /proc/sys directory.

System parameters are held within files with names that mimic their function. For example, a Linux system can become a rudimentary router if the IP forwarding networking parameter is set to 1, not 0. Networking parameters for IPv4 addressing schemes are located in the /proc/sys/net/ipv4/. The file that covers IP forwarding is named ip_forward and contains a single byte: 1 when it's active or 0 when it's not.

You can update these files by redirecting the output of the echo command to overwrite the file contents using the > redirection character. This line activates IP forwarding by overwriting the contents of the ip_forward file with the value 1.

[root@bigboy tmp] echo 1 > /proc/sys/net/ipv4/ip_forward

There are some disadvantages to doing this. It is not a permanent solution, and the system will revert back to its defaults after the next reboot. You can overcome this by adding the echo commands to the /etc/rc.local script that runs at the end of each reboot. This too has its disadvantages; it is the very last script to be run so if your parameters need to be set earlier, as most kernel parameters should be, it isn't a suitable solution.

Linux has a more elegant solution called the /etc/sysctl.conf file. It is a list of all the /proc filesystem files the systems administrator wants to customize and the values they should contain. The file has two columns, the first is the filename relative to the /proc/sys directory, and the second is the value the file should contain separated by an equals sign. You can also replace the slashes in the filename with periods. Continuing with the example, you can set the /proc/sys/net/ipv4/ip_forward file with a value of 1 using either of these configurations.

# Sample /etc/sysctl.conf file
# Activate IP forwarding
net.ipv4.ip_forward = 1
# Activate IP forwarding too!
net/ipv4/ip_forward = 1

Editing the /etc/sysctl.conf file isn't enough, because the update isn't instantaneous. You have to force Linux to reread the file and reset the kernel parameters using the sysctl -p command. In the example, the /proc/sys/net/ipv4/ip_forward file had a value of 0 until the sysctl command ran, after which time, IP forwarding was activated.

[root@bigboy tmp]# cat /proc/sys/net/ipv4/ip_forward
[root@bigboy tmp]# sysctl -p
net.ipv4.ip_forward = 1
net.ipv4.conf.default.rp_filter = 1
kernel.sysrq = 0
kernel.core_uses_pid = 1
[root@bigboy tmp]# cat /proc/sys/net/ipv4/ip_forward
[root@bigboy tmp]#

The use of the /etc/sysctl.conf is important in day to day administration.

Running Linux Without A Monitor

You can reduce the cost of ownership of your Linux system by not using a VGA monitor. This creates what is also known as a headless system. Operating costs may not be important at home, but will be in a corporate environment with large numbers of Linux servers racked in data centers. In such cases, access to the Linux box can be more cheaply provided via the COM port.

I've included this section, largely because I have occasionally hosted the Web site www.linuxhomenetworking.com at friends' homes and felt badly about borrowing their monitors. Having access via the COM ports has also helped me in both the home and business situations. The most common occurrence is when the system is hung, locking out network access, and I need to get to it by using:

  • A notebook PC with a console cable connected to the COM port
  • A modem connected to the COM port
  • Telnet to log in to a terminal server that has one of its ports connected to the Linux box's COM port

Preparing To Go "Headless"

One of the advantages of this method is that you don't need a keyboard either. Unfortunately, your BIOS may halt the system during the Power On Self Test (POST) if it doesn't detect a keyboard. Make sure you disable this feature in the BIOS setup of your PC before proceeding. You can usually find the POST feature on the very first screen under the Halt On option.

You also need to make sure that you have activated your COM ports in your BIOS settings.

For connectivity that doesn't involve modems (PC to PC) connect a null modem cable to the COM port you want to test, connect the other end to the client PC running Hyperterm or whatever terminal emulation software you are using. One popular Linux equivalent to Hyperterm is minicom.

If you're using a modem for connectivity, then you'll need a full modem cable and you'll have to test via a dial up connection.

Configuration Steps

In RedHat/Fedora Linux, the COM1 and COM2 ports are controlled by a program called agetty, but agetty usually isn't activated when you boot up unless its configuration file /etc/inittab is modified. In other versions of Linux, agetty may be called just plain getty. Table I.1 lists the physical ports to their equivalent Linux device names.

Table I.1 How Physical COM Ports Map To Linux TTYS Devices

Port Linux "agetty" Device Name
COM1 ttyS0
COM2 ttyS1

You can determine whether you have valid serial ports using the setserial command. In this case we are looking for all available ports between ttyS0 to ttyS7. You can see we only have two valid selections, ttyS0 and ttyS1.

[root@bigboy tmp]# setserial -g /dev/ttyS[01234567]
/dev/ttyS0, UART: 16550A, Port: 0x03f8, IRQ: 4
/dev/ttyS1, UART: 16550A, Port: 0x02f8, IRQ: 3
/dev/ttyS2, UART: unknown, Port: 0x03e8, IRQ: 4
/dev/ttyS3, UART: unknown, Port: 0x02e8, IRQ: 3
[root@bigboy tmp]#

To configure both valid COM ports for terminal access, add these lines to /etc/inittab:

# Run COM1 and COM2 gettys in standard runlevels
S0:235:respawn:/sbin/agetty -L 9600 ttyS0 vt102
S1:235:respawn:/sbin/agetty -L 9600 ttyS1 vt102

What do these lines mean? At boot time, when the system enters runlevels 2, 3, or 5, agetty must attach itself to devices ttyS0 and ttyS1 and emulate a VT102 terminal running at 9600 baud. The -L means ignore modem control signals, this option should be omitted if you are connecting the port to a modem. The respawn lines mean that agetty will restart automatically if, for whatever reason, it dies.

The next step is to restart the init process to re-read /etc/inittab:

[root@bigboy tmp]# init q

You'll be able to verify the activation of agetty process by querying the process table using the ps command.

[root@bigboy tmp]# ps -ef | grep agetty
root  958  1  0 Dec13 ttyS0 00:00:00 /sbin/agetty -L 9600 ttyS0 vt100
root 1427  1  0 Dec13 ttyS1 00:00:00 /sbin/agetty -L 9600 ttyS0 vt100
[root@bigboy tmp]#

Now you need to configure the terminal client (Hyperterm) to match the speed settings in /etc/inittab. Connect the console/modem cable between the client and your Linux box. Press Enter a couple times, and celebrate when you see something like this:

Red Hat Linux release 8.0 (Psyche)
Kernel 2.4.18-14 on an i586

bigboy login: 

Note: By default, user root will not be able to log in from a terminal. To do this, you'll have to edit the /etc/securetty file, which contains the device names of TTY lines on which root is allowed to login. Just add ttyS0 and ttyS1 to the list if you need this access.

Switching Your Console to the Serial Port

By configuring the options in your /etc/grub.conf file you can redirect console output to one of your newly configured serial ports. This will allow you to see all of the bootup and shutdown messages on this new serial terminal port. In this example, the /dev/ttyS1 port has been configured to become the new console. The text to add to the file is highlighted in red:

title Fedora (
       root (hd0,0)
       kernel /vmlinuz- ro root=LABEL=/ console=ttyS1,9600 rhgb quiet
       initrd /initrd-
title Fedora (
       root (hd0,0)
       kernel /vmlinuz- ro root=LABEL=/ console=ttyS1,9600 rhgb quiet
       initrd /initrd-

With this modification you will still be able to completely administer the system using this port, inclusive of the grub pre-boot commands, after the next reboot. Note: It is possible that the kudzu hardware detection program may not be compatible you’re your non default console configuration that doesn't use your system's video adapter. If your system hangs on reboot, you may have to reset your system; break into the grub bootloader; temporarily remove the console options from the boot command line; and disable kudzu when the system fully reboots.

[root@bigboy tmp]# chkconfig kudzu off

This is an unlikely event with newer Linux versions, but you should be aware that it can happen.

Make Your Linux Box Emulate A VT100 Dumb Terminal

Dumb terminals are devices that allow you to log in to your system via the COM port. You can make your Linux box emulate a dumb terminal quite easily. Why would you want to? For example, you run Linux on a notebook and you need to use it to access a hung headless Linux server via the COM port, Or, perhaps you need to gain access to a modem connected to the COM port. This section will focus on the notebook scenario, not using Linux to dial a modem:

Configuration Steps

The most commonly used Linux terminal emulation program is minicom. It is simple to use mainly because it uses a text-based GUI. You first need to go through all the relevant steps listed in the "Preparing to go Headless" section to ensure you have the right type of cable and correct BIOS settings.

Be warned, minicom will clash with your agetty configuration explained previously: A headless system cannot be used to access another headless system using the headless COM port. You, therefore, have to disable the agetty configuration for the port on which you wish to run minicom.

1. It is important not to have an agetty entry for the COM port you are going to use. If an entry exists, disable agetty on COM1 by commenting out the ttyS0 agetty statements in the /etc/inittab file. COM1 now handles outbound minicom connections to other systems, and other systems using minicom can use COM2 to access this system. The next phase is to restart the init process to reload the new /etc/inittab settings.

2. Edit /etc/inittab:

# Run COM1 and COM2 gettys in standard runlevels
#S0:235:respawn:/sbin/agetty -L 9600 ttyS0 vt102
S1:235:respawn:/sbin/agetty -L 9600 ttyS1 vt102

3. Restart init

[root@bigboy tmp]# init q

4. Run minicom in setup mode using the minicom -s command, which brings you to the setup menu:

[root@bigboy tmp]# minicom -s

| Filenames and paths      |
| File transfer protocols  |
| Serial port setup        |
| Modem and dialing        |
| Screen and keyboard      |
| Save setup as dfl        |
| Save setup as..          |
| Exit                     |
| Exit from Minicom        |

5. Select the Serial Port Setup menu item. Make the speed match that of the remote headless system and make sure the correct serial COM device is chosen. Device /dev/ttyS0 is COM1 and /dev/ttyS1 is COM2. Also make sure that flow control is off.

| A -    Serial Device      : /dev/ttyS0  |
| B - Lockfile Location     : /var/lock   |
| C -   Callin Program      :             |
| D -  Callout Program      :             |
| E -    Bps/Par/Bits       : 9600 8N1    |
| F - Hardware Flow Control : No          |
| G - Software Flow Control : No          |
|                                         |
|    Change which setting?                |

6. Select the Modem and Dialing option and make sure the Init String and Reset String settings are blank.

7. Select the Save Setup as dfl to make this your saved default setting, and then choose Exit from Minicom.

8. Make sure the other system is correctly configured for headless operation. Connect the cables between the systems.

9. Re-enter minicom, this time without the -s.

[root@bigboy tmp]# minicom

10. Press Enter, and you should get a login prompt

Welcome to minicom 2.00.0
OPTIONS: History Buffer, F-key Macros, Search History Buffer, I18n
Compiled on Jun 23 2002, 16:41:20.
Press CTRL-A Z for help on special keys
bigboy login:

11. To exit minicom, type Ctrl-A, then Z, then X

12. Users other than root will get a "permission denied" message if they use minicom, because the COM ports are not normally accessible to regular users. To get around this, user root can either give everyone read/write access using the chmod command or add selected trusted users to your sudo configuration.

Remember that minicom resets the privileges to the COM port each time you change the configuration with minicom -s so you may find yourself having to run chmod from time to time.

[root@bigboy tmp]# chmod o+rw /dev/ttyS0

Note: This configuration is sufficient for connection to a Cisco router, switch or PIX firewall. Just remember to use the Cisco proprietary RJ45 type console cable to connect your COM port to the network device.

Dual Booting Fedora and Ubuntu Linux on Servers

Before virtualization in which multiple and different operating system could be run simultaneously on the same server there was dual booting. This older method allows a system to boot one of many different operating systems residing on its disks. It is still commonly used on slower hardware, by students and in lab testing environments. This section describes how it is done, but first we need to discuss a disk’s master boot record (MBR).

The MBR stores the disk’s partition table, a unique disk id, and information that points to a predefined partition on the disk that should contain files used in booting. The data included in this boot partition includes a file with the initial boot menu screen and the bootable kernels associated with each menu entry. In Fedora the installation process creates a separate boot partition ( /boot ) from the root partition, in Ubuntu, the boot partition and the root partition are the same ( / ).

This information is important to understand when installing Fedora and Ubuntu together in a dual boot configuration. The example shown here assumes that Fedora will be installed first. Let’s go!

Installing and Configuring Fedora

Follow these steps carefully and you should have a functioning system with few installation headaches.

Install Fedora using the custom disk layout without LVM and leave a single free partition area large enough to install Ubuntu later. Fedora will configure the disk’s master boot record (MBR) to point to the /boot partition and the /boot/grub/menu.lst file with the startup menu.

With a brand new Fedora installation in which there is only one version of the kernel, this menu may not be seen when you reboot Fedora. Edit the /boot/grub/menu.lst file and comment out the hiddenmenu parameter to ensure the menu shows up and configure it to be displayed for some time using the timeout parameter. In this case it is set to 60 seconds.

# File: /boot/grub/menu.lst

Now it’s time to configure Ubuntu to work seamlessly with Fedora.

Installing and Configuring Ubuntu

Reboot and install Ubuntu server. Pay attention to two parts of this process.

1. Near the beginning you will be prompted for disk partition information. Select the option that installs Ubuntu in the unused free partition space only. Write down partition number where the root, or / partition will be installed. In this case we’ll assume it is /dev/sda11. This will be needed later.

2. Close to the end you will be asked whether you want to install grub on the MBR or on the root partition. Select the root partition of /dev/sda11.

Note: DO NOT install on the MBR. Ubuntu will read the contents of Fedora’s menu.lst file, will add it to its own menu.lst file in /dev/sda11 and then will make the MBR point to the Ubuntu menu.lst file. The difficulty is that the Fedora menu.lst file contents are specific to the current versions of the Fedora kernel files. If you upgrade Fedora the Ubuntu menu.lst file will not be updated and Fedora won’t be able to boot properly again from the now outdated Ubuntu menu.lst file.

Configuring Fedora grub to dual boot with Ubuntu

Now it’s time to tie the Fedora installation to that of Ubuntu. Here are the steps.

1. Reboot. At the Fedora startup menu enter single user mode to reduce the risk of disk activity interfering with our work.

2. We will now need to properly configure grub to use the Fedora menu for Fedora boots using the Fedora grub boot loader and the Ubuntu menu for Ubuntu boots with the Ubuntu boot loader.

3. As seen before, Ubuntu was installed on the /dev/sda11 partition. We need to access those files. Mount /dev/sda11 on /mnt and run Ubuntu’s version grub which should now be /mnt/usr/sbin/grub

# mount /dev/sda11 /mnt
# /mnt/usr/sbin/grub

4. Grub numbers partitions and disks starting from 0. Linux numbers partitions starting from 1 and disks are defined alphabetically from a-z. So in this case Linux partition sda11 will be grub partition (hd0,10).

Use the device command to map the two together. Follow this with the root command to set up the Ubuntu grub boot loader to accept a transfer of authority from another preliminary loader, in this case it will be Fedora but we don’t specify this yet. The setup command will configure grub correctly, and is done as a precaution as the Ubuntu installation process should have done this already.

grub> device (hd0) /dev/sda
grub> root (hd0,10)
grub> setup (hd0,10)
Checking if "/boot/grub/stage1" exists... yes
 Checking if "/boot/grub/stage2" exists... yes
 Checking if "/boot/grub/e2fs_stage1_5" exists... yes
 Running "embed /boot/grub/e2fs_stage1_5 (hd0,10)"... failed (this is not fatal)
 Running "embed /boot/grub/e2fs_stage1_5 (hd0,10)"... failed (this is not fatal)
 Running "install /boot/grub/stage1 (hd0,10) /boot/grub/stage2 p /boot/grub/menu.lst "... succeeded
grub> quit

Note: DO NOT run the grub-install command to configure grub as you will be running the fedora version of the command against an incompatible Ubuntu partition. It will appear to work as seen in the following example but the rest of the setup will fail.

[root@smallfry-f]# grub-install /dev/sda11
Installation finished. No error reported.
This is the contents of the device map /boot/grub/device.map.
Check if this is correct or not. If any of the lines is incorrect,
fix it and re-run the script `grub-install'. 
# this device map was generated by anaconda
(hd0)     /dev/sda

Note: DO NOT run the grub-install command located on the Ubuntu partition or in a chroot jail will not work either giving messages like these

[root@smallfry-f ~]# mount /dev/sda11 /mnt/ubuntu/
[root@smallfry-f ~]# cd /mnt/ubuntu/
[root@smallfry-f ubuntu]# usr/sbin/grub-install /dev/sda11
/usr/sbin/grub: Not found.
[root@smallfry-f ubuntu]# chroot /mnt/ubuntu
root@smallfry-f:/# /usr/sbin/grub-install /dev/sda11
/dev/sda11: Not found or not a block device.
root@smallfry-f:/# exit

We’re not finished yet. There a few more steps to go!

Configuring Fedora grub to dual boot with Ubuntu

Now it’s time to configure the Fedora menu.lst file to point to the menu.lst of the Ubuntu installation.

Note: You can do the following steps while still in single user mode or you can reboot Fedora and do them in an ssh session as it will be easier to cut and paste some of the commands to follow. Remember Ubuntu will not boot yet or be displayed in the startup menu.

1. Edit /boot/grub/menu.lst and add the entries for Ubuntu with its own menu line option. The title command gives the Ubuntu entry the label “Ubuntu Boot Menu”. The root command points to the /dev/sda11 partition or (hd0,10) as grub expects it to be called. Finally the chainloader command tells Fedora’s grub to hand over control to whatever grub boot loader is found in that partition. In this case that loader that will be found will belong to Ubuntu. Here is a sample of what it should look like.

# File: /boot/grub/menu.lst

title Fedora (
        root (hd0,0)
        kernel /vmlinuz- ro root=UUID=de68defd-1f82-4ad3-b39e-0776b15d6d92 rhgb quiet
        initrd /initrd-

title   Ubuntu Boot Menu 
        root            (hd0,10)
        chainloader     +1

2. Reboot and you should now get a Fedora startup menu with an Ubuntu option. By using your arrow keys you can scroll down and select the Ubuntu option. This will then cause the Ubuntu startup menu to be displayed. You can boot Ubuntu by scrolling down to the menu item and hitting <enter>. From there you will be able to select the Ubuntu kernel of your choice. The default should be sufficient and will boot automatically after the Ubuntu menu.lst file’s timeout period expires.

Your dual booting configuration is complete. Boot your operating systems and configure them as you desire.

Simple Troubleshooting

If you try to boot Ubuntu from the menu and you get an “error 15: file not found” message then you most likely haven’t properly completed the grub configuration steps or you have a typographical error in your Ubuntu menu.lst file or the Fedora menu.lst file’s entry that points to Ubuntu.

If you select the Ubuntu menu item and you immediately return to the Fedora boot menu screen then you probably ran the grub-install command when you shouldn’t have.

How to get Ubuntu to Boot by Default

Making Ubuntu the preferred operating to boot when the server is unattended requires two file edits.

1. The first is done by booting Fedora and editing the /boot/grub/menu.lst and setting the default option to select Ubuntu. A default of 0 causes the first menu item title to be selected by default, a value of 1 selects the second, a value of two selects the third and so on. In this example setting the value to 1 will cause the Ubuntu menu item, the second in the list, to be automatically chosen.

2. Reboot your system. The Fedora menu will change to become the Ubuntu menu and then the default Ubuntu kernel will boot. You can change the Ubuntu default boot kernel by editing the /boot/grub/menu.lst file’s default parameter after the Ubuntu boot is completed using the guidelines given previously.

Note: If you are skillful you don’t have to reboot. Just mount the appropriate partitions and edit the menu.lst file.


Operating system virtualization has reduced the need to create dual booting systems, but this section should still be valuable to those of you who need to troubleshoot a GRUB configuration.

VPN Terms and Methods

A virtual private network (VPN) provides security for transmission of sensitive information over unprotected networks such as the Internet. VPN relationships are established between trusted sites on the Internet making the public network appear to be virtually the same as a private network to the VPN members. VPNs, however, have their own language. Once you know the terms, you'll be ready to get to work.

  • Authentication: The process of ensuring that the VPN data received is both unchanged and from the expected source.
  • Encryption: The use of special mathematical equations to scramble the bits in a data stream, rendering the data stream unreadable to anyone who does not have access to the corresponding decryption equation. The process is usually made even harder through the use of an encryption key, which modifies the way the equations do the scrambling. Only persons with access to this key and the corresponding programs are able to recover the original data. Data encryption helps to prevent unauthorized persons from having access to the data.
  • IPSec: The name given to a number of data communications protocols designed to authenticate and encrypt VPN data to protect it from unauthorized viewing or modification as it is transmitted across a network.
  • Authentication header (AH): One of two IPSec security protocols. Provides authentication and antireplay services, without encryption, by adding its own security header to the original IP packet

Encapsulating security protocol (ESP): The other IPSec security protocol. Provides authentication, encryption, and antireplay services. It encrypts the data within the packet and then adds its own security header to the original IP packet. Because ESP headers don't authenticate the outer IP header as AH headers do, AH and an ESP are often used in combination with each other. This is called transport adjacency.

Transport mode VPN: A style of VPN in which the original source and destination address of the data sent over the VPN is unchanged. Figures I.1 and I.2 provide examples of transport mode VPN IP packets. (For more information on the IP protocol, please refer to Chapter 2, "Introduction to Networking", which introduces networking concepts.)

Figure I.1 Transport mode AH packet format

Original IP Header
Inserted AH Header
Original TCP Header

Figure I.2 Transport mode AH / ESP packet format

Original IP Header
Inserted AH Header
Inserted ESP Header
Original TCP Header

  • Tunnel mode VPN: A style of VPN in which the original source and destination address of the data sent over the VPN is changed by encapsulating the original IP packet within another IP packet. The original packet is frequently encrypted, header and all, in an effort to provide an additional layer of security by not revealing the true identities of the servers communicating with each other. Figures I.3 and I.4 show examples of tunnel mode VPN IP packets.

Figure I.3 Tunnel mode AH packet format

New IP Header
Inserted AH Header
Original IP Header
Original TCP Header

Figure I.4 Tunnel mode AH / ESP packet format

New IP Header
Inserted AH Header
Inserted ESP Header
Original IP Header
Original TCP Header

Authentication and Encryption methods

IPSec data integrity is usually provided by one of two hashed message authentication code (HMAC) methods:

  • Message Digest 5 (MD5)
  • Secure Hash Algorithm (SHA-1).

IPSec usually uses one of two methods to encrypt data:

  • Data Encryption Standard (DES) using a 56-bit encryption key
  • Triple DES using a 168-bit encryption key.

Internet Key Exchange (IKE)

IKE provides authentication of the IPSec peers, negotiates IPSec security associations, and establishes IPSec keys. There are two main methods of establishing a trusted relationship between two devices that want to create a VPN between themselves.

Public Encrypted Keys

The first method is public key cryptography using RSA encryption pioneered by RSA Data Security Inc. Each VPN device has its own public and private keys. Anything encrypted with one of the keys can only be decrypted with the other. This allows you to create a signature when the message is encrypted with a sender's private key. The receiver verifies the signature by decrypting the message with the sender's public key.

A successful exchange requires the receiver to have a copy of the sender's public key and to know with a high degree of certainty that it really does belong to the sender, and not to someone pretending to be the sender. This certainty is assured using certificates and Certification Authorities.

A digital certificate contains information that identifies a user or device, such as a name, serial number, company, department, or IP address. It also contains a copy of the entity's public key. Certificates are managed and issued by Certification Authorities (CAs). A CA can either be a trusted public third party, such as VeriSign, or an in-house private server that you establish within your organization.

Prior to installing a certificate-based VPN, each VPN device must be preconfigured with the certificate generated for them by the CA. The VPN devices are also be preconfigured with the CA's certificate.

During the key exchange, the VPN peers authenticate by sending each other the certificate issued to them by the CA, but encrypted using their private key.

Each peer then uses the pre-installed CA certificate they have to authenticate with the CA and securely receive the other peer's certificate from the CA using public key cryptography.

Each peer then extracts the public key from the certificate they receive from the CA and uses it to decrypt the certificate they just received from the other peer.

Once the certificates received from the CA and the other peer match, authentication is complete.

Private Shared keys

The second method of establishing a trusted relationship is to have the devices at each end of the VPN use a shared key or password. The disadvantage is that each pair of VPN connections needs set of keys, making it difficult for large scale implementations. Unlike the RSA method, there is no CA to provide an impartial audit trail of VPN connection initiations.

IKE's role in creating Security Associations

Once authentication is complete, the VPN peers use IKE to negotiate the security associations (SAs) to be used at each end point. IKE, in turn, uses special ISAKMP IP packets using IP protocol 50 to establish a security association. SAs are comprised of transforms and shared keys. Transforms describe how the data will be transformed by the VPN and between which pre-defined networks at each end of the tunnel to provide the desired security, for example:

  • Packet encryption methods
  • Packet authentication methods
  • Transport versus tunnel mode
  • AH and or ESP usage
  • SA lifetime before it is renegotiated. (SAs are permanent for when manually established)

Shared keys are the actual keywords used by the encryption and authentication processes to protect the data.

VPN Security And Firewalls

All security devices in the path of a VPN connection have to allow IP protocol 50 between the two VPN devices to ensure that IKE works properly.

VPNs also use a separate channel through which the encrypted data passes as UDP packets through port 500. Unusually, the source and destination port is 500.

For the VPN to function correctly these protocols must also be allowed to pass through unimpeded. In certificate based VPNs, you may have to open up these ports and protocols to the CA as well.

Note: VPN tunnels frequently don't operate correctly if they have to pass through a firewall that uses NAT. They will tend to work if the NAT firewall is the termination point of the VPN.

VPN User Authentication Methods For Temporary Connections

The discussion so far has been slanted towards a permanent connection between purpose built VPN devices. Frequently, however, the device at the other end of the connection is a PC. Table I.2 shows some authentication methods used in such cases.

Table I-2 Types Of Dial Up VPN Authentication

Method Description
IKE-XAUTH secured RADIUS Usernames and passwords entered into the VPN remote login software are relayed by the VPN device at the remote end to a trusted RADIUS server. If the username/password combination is valid for remote login, then the RADIUS server authorizes the VPN device to continue with the IKE interchange.
ACE/SecurID Software uses a username and password in conjunction with a small key-ring token with a digital display, also known as a fob, whose authentication serial number changes every few minutes for a login to occur. To log in, the user not only has to enter the username and password, but also the PIN tied to the fob plus the fob's dynamic serial number which is synchronized with the authentication server at the other end of the VPN.
Windows Domain Remote home user authentication relies on the same username/password combination of the Windows Domain Controller that the user would normally use to log in when they are at work.
Local user database Valid usernames and passwords are configured into the VPN device at the other end of the VPN

TCP/IP Packet Format

The TCP/IP packet contains an IP header followed by a TCP or UDP header followed by the TCP/UDP data as seen in Figure I.5.

Figure I.5

IP Header
TCP / UDP Header

Tables I.3 through I.5 show the general format of the various TCP/IP header fields you may encounter.

Table I.3 Contents Of The IP Header

Field Description
Ip Version The version of IP being used. Version 4 is the current version used by most devices on the Internet. Version 6 is a newer format that allows for a more vast range of addresses.
IHL Internet Header Length. Total length of the IP header.
Total Length Total length of the IP packet.
DF Bit Indicator to tell whether the data in the packet may be fragmented into smaller packets due to limitations of the communications line.
MF Bit Indicator to tell whether this data in this packet is the last one of a stream of fragments.
Fragment Offset If the packet is part of a fragmented datagram, then this specifies where in the complete datagram the data in the packet should be inserted.
TTL Time to live. This value is decremented by each router through which the packet passes. When the value reaches zero, the packet is discarded by the router. The server sending the data usually sets the TTL to a value high enough to reach its destination without being discarded. The TTL decrement feature is used by routers as an additional precaution to prevent the packet from mistakenly being routed around the Internet in an infinite loop caused by a routing error.
Protocol Defines the type of protocol header to expect at the end of this header. For example, 6 = TCP. 17 = UDP
Header checksum Used to ensure that the header contents are error free.
Source Address Indicates the IP address of the server sending the data.
Destination Address Indicates the IP address of the server intended to receive the data.

Table I.4 Contents Of The TCP Header

Field Description
Source and Destination Port Identifies points at which upper-layer source and destination processes receive TCP services.
Sequence Number Usually specifies the number assigned to the first byte of data in the current message. In the connection-establishment phase, this field also can be used to identify an initial sequence number to be used in an upcoming transmission.
Acknowledgment Number Contains the sequence number of the next byte of data the sender of the packet expects to receive.
Data Offset Length of the TCP header.
Flags Carries a variety of control information, including the SYN and ACK bits used for connection establishment, and the FIN bit used for connection termination.
Window Specifies the size of the sender's receive window (that is, the buffer space available for incoming data)
Checksum Used to ensure that the header contents are error free.
Data Contains upper-layer information

Table I.5 Contents Of The UDP Header

Field Description
Source and Destination Port Identifies points at which upper-layer source and destination processes receive TCP services.
Length Specifies the length of the UDP header and data
Checksum Used to ensure that the header contents are error free.

ICMP Codes

You'll also encounter ICMP codes in your troubleshooting exercises, especially when viewing your iptables log files. Table I.6 lists the most commonly used codes.

Table I-6 ICMP Codes





Destination Unreachable Codes 


Net Unreachable

The sending device knows about the network but believes it is not available at this time. Perhaps the network is too far away through the known route.


Host Unreachable

The sending devices knows about host but doesn't get ARP reply, indicating the host is not available at this time


Protocol Unreachable

The protocol defined in IP header cannot be forwarded.


Port Unreachable

The sending device does not support the port number you are trying to reach


Fragmentation Needed and Don't Fragment was Set

The router needs to fragment the packet to forward it across a link that supports a smaller maximum transmission unit (MTU ) size. However, application set the Don't Fragment bit.


Source Route Failed

ICMP sender can't use the strict or loose source routing path specified in the original packet.


Destination Network Unknown

ICMP sender does not have a route entry for the destination network, indicating this network may never have been an available.


Destination Host Unknown

ICMP sender does not have a host entry, indicating the host may never have been available on connected network.


Source Host Isolated

ICMP sender (router) has been configured to not forward packets from source (the old electronic pink slip).


Communication with Destination Network is Administratively Prohibited

ICMP sender (router) has been configured to block access to the desired destination network.


Communication with Destination Host is Administratively Prohibited

ICMP sender (router) has been configured to block access to the desired destination host.


Destination Network Unreachable for Type of Service

The sender is using a Type of Service (TOS) that is not available through this router for that specific network.


Destination Host Unreachable for Type of Service

The sender is using a Type of Service (TOS) that is not available through this router for that specific host.


Communication Administratively Prohibited

ICMP sender is not available for communications at this time.


Host Precedence Violation

Precedence value defined in sender's original IP header is not allowed (for example, using Flash Override precedence).


Redirect Codes 


Redirect Datagram for the Network (or subnet)

ICMP sender (router) is not the best way to get to the desired network. Reply contains IP address of best router to destination. Dynamically adds a network entry in original sender's routing tables.


Redirect Datagram for the Host

ICMP sender (router) is not the best way to get to the desired host. Reply contains IP address of best router to destination. Dynamically adds a host entry in original sender's route tables.


Redirect Datagram for Type of the Service and Network

ICMP sender (router) does not offer a path to the destination network using the TOS requested. Dynamically adds a network entry in original sender's route tables.


Redirect Datagram for the Type of Service and Host

ICMP sender (router) does not offer a path to the destination host using the TOS requested. Dynamically adds a host entry in original sender's route tables.


Alternate Host Address Codes


Alternate Address for Host

Reply that indicates another host address should be used for the desired service. Should redirect application to another host.


Time Exceeded Codes


Time to Live exceeded in Transit

ICMP sender (router) indicates that originator's packet arrived with a Time To Live (TTL) of 1. Routers cannot decrement the TTL value to 0 and forward the packet.


Fragment Reassembly Time Exceeded

ICMP sender (destination host) did not receive all fragment parts before the expiration (in seconds of holding time) of the TTL value of the first fragment received.


Parameter Problem Codes


Pointer indicates the error

Error is defined in greater detail within the ICMP packet.


Missing a Required Option

ICMP sender expected some additional information in the Option field of the original packet.


Bad Length

Original packet structure had an invalid length.