(Transferred from the wiki by Peter)

First off, portsentry is a automated port scan detector and response tool made by Psionic Software and is part of thier Abacus Project. I detects and the stops portscan attempts on your Linux/UNiX system.
Get It

Get portsentry from http://sourceforge.net/projects/sentrytools/
Unpack It

gunzip and untar the file downloaded: tar xvzf gunzip portsentry-1.1.tar.gz. If that command fails (and tar isn't compiled with gzip support) try: portsentry-1.1.tar.gz && tar xvf portsentry-1.1.tar (NOTE: && means and. It's like issuing the commands in order.)
Edit Config

open up portsentry.conf in your favorite text editor. The setup here is easy to follow but I'll go through and do a sample set up.

Port Configuration: Generally the default aware ports are fine, though I like the really anal ports myself :-P. To do this we remove the #'s in front of the TCP_PORTS and UDP_PORTS just under the really anal line. Then move down and comment the be aware TCP_PORTS and UDP_PORTS. It will look like this:
# Un-comment these if you are really anal:
# Use these if you just want to be aware:
# Use these for just bare-bones
Dropping Routes

Skip down to Dropping Routes and uncomment the line that has your OS listed. (Under Linux you should uncomment the iptables for kernel 2.4.x and ipchains with logging for 2.2.x)

Skip down to Port Banner Section because it's fun and we like to taunt script kiddies . Uncomment it and add what ever you'd like them to see. Ex: PORT_BANNER="** Don't mess with me k1dd13!!! ** You have been logged and your ISP is automatically being contacted.

Next you'll want to make linux. (you can type make to get a list of os's it will make on)

make install
Configure System

You'll probably want to start portsentry at boot time so here is the line you'll want to add to your rc.local file ...
/usr/local/psionic/portsentry/portsentry -tcp && /usr/local/psionic/portsentry/portsentry -sudp
This will start portsentry at boot time and respond with a banner if the scan is agains TCP ports, and not respnd to the attacker if it's on UDP ports. Either way they will be blocked. (NOTE: rc.local is in diffrent locations depending on distro ... on Slackware it's in /etc/rc.d/.)
Test Fire

Test Fire (Optional ... From LinuxSecurity): The PortSentry program can be configured in six different modes of operation, but be aware that only one protocol mode type can be started at a time. To be more accurate, you can start one TCP mode and one UDP mode, so two TCP modes and one UDP modes, for example, doesn't work. The available modes are:

portsentry -tcp basic port-bound TCP mode
portsentry -udp basic port-bound UDP mode
portsentry -stcp Stealth TCP scan detection
portsentry -atcp Advanced TCP stealth scan detection
portsentry -sudp Stealth UDP scan detection
portsentry -audp Advanced Stealth UDP scan detection

In my case I prefer to start TCP in Advanced TCP stealth scan detection protocol mode and UDP in Stealth UDP scan detection protocol mode. For information about the other protocol modes, please refer to the README.install and README.stealth file under the PortSentry source directory.

For TCP mode I choose:

-atcp Advanced TCP stealth scan detection mode
With the Advanced TCP stealth scan detection mode -atcp protocol mode type, PortSentry will first check to see what ports you have running on your server, then remove these ports from monitoring and will begin watching the remaining ports. This is very powerful and reacts exceedingly quickly for port scanners. It also uses very little CPU time.

For UDP mode I choose:

-sudp Stealth UDP scan detection mode
With the Stealth UDP scan detection mode -sudp protocol mode type, the UDP ports will be listed and then monitored.
The End

Be happy!