Iptables firewall versus nmap and hping3
Part 2 : Port scanning

How to harden your firewall against the powerful tools nmap and hping3, based on the Thylacine iptables based host firewall. This is part 2 : Port scanning

Article published on 21 September 2010
last modification on 28 May 2016

by Emeric Nasi


Note: In order to understand this document it is strongly recommended that you know about the TCP/IP protocol and have iptables, nmap and hping3 installed. I recommend also the use of wireshark that is a great tool to learn about network protocols.
License : Copyright Emeric Nasi, some rights reserved
This work is licensed under a Creative Commons Attribution 4.0 International License.
Creative Commons License

This is the second part of the article "Iptables firewall versus nmap and hping3". If it is not already the case you should read the first part : Iptables firewall versus nmap and hping3 : Part1 the basics

Note : The iptables commands are from the Thylacine host firewall that you can find in the Thylacine security hardening tool.
Note2 : Even if this article is based on a host firewall, you can easily transpose the commands to a network iptables firewall.

II Iptables versus port scan attempts

In this section I will give some tips to set iptables to block various scan attempts. Despite the fact that hping3 can be used to scan, here I will use nmap because it is worldwide known to be THE port scanner.

2.1 Null scan

It consists to scan TCP ports using TCP packets with no flags. Some firewall do not protect against this scan and it can be used to know which ports are closed.
Null scan with nmap example :
nmap -sN 192.168.0.2
Prevent null scan with iptables :
iptables -A INPUT -p tcp --tcp-flags ALL NONE -j DROP
How I prevent null scan in Thylacine firewall :

2.2 Xmas scan

Contrary to null scan Xmas scan consists into setting all or almost all of the TCP flags. Xmas attacks can be used to know which ports are closed but, like Null scan you cannot differentiate an opened port from a filtered port (you only receive the RST packet when you touch a closed port).
Xmas scan with nmap example :
nmap -sX 192.168.0.2

To prevent Xmas scans you need to configure your firewall to detect particular flags inside TCP packets.
How I prevent 3 kinds of Xmas scans in Thylacine firewall :

2.3 FIN scan

It consists in sending TCP packets with the FIN flag on. Depending on the targeted system you can discover closed ports (other ports are stated as open|filtered), on other system like windows you might not get any answers (it can be use for system fingerptinting too).
FIN scan with nmap example :
nmap -sF 192.168.0.2
Protecting from this scan consists in watching for packets with only fin flag set.

Example in Thylacine firewall :

You use these kind of protections for any other kind of scan using abnormal flag settings. Lets have a look at other scan techniques.

2.4 ACK scan

It consists in sending TCP packets with the ACK flag set. In the TCP protocol, this flag is normally set when the packet is an acknowledgment of receipt of another packet. This scan uses the fact that a machine without a firewall will send a RST packet when receiving an acknowledgment of receipt of an unknown connection. The port may be opened or closed it does not matter here.
The thing is, if the port cannot be reached (because there is a firewall protecting it for example), then the machine won’t get the false ACK and won’t send back the RST packet. That means the goal of the ACK scan is to guess if a port is filtered or if it is not. If you combine that with a FIN scan or a XMAS scan you can guess which ports are opened, filtered and closed.
ACK scan with nmap example :
nmap -sA 192.168.0.2

If nmap receive an RST packet port is considered unfiltered, if there is no response, to port is considered filtered.To prevent this scan approach, there are several ways (like always). In the Thylacine firewall I chose to block any attempts of ACK scans, which means the result of an ACK scan displays that all ports are filtered.
There is a very simple way to do that :
iptables   -A INPUT -p tcp ! --syn -m state --state NEW  -m comment --comment "Drop TCP connection not starting by SYN" -j DROP
With this line any new connection must start with only a SYN flag. If it is not the case the packet is discarded. That way, ACK scan is no more possible.
Now you should say, wait a minute! With this only line we could also prevent all the previously mentioned scans! That is true. In the Thylacine firewall I choose another way to prevent these scans because I want these scans to be precisely logged and I want the attacker address to be temporarily banned.

2.5 SYN scan and TCP connect scan

SYN scan is the default scan mode when using nmap and it is also the most used scan technique. This scan method consists in sending TCP packets with only the SYN flag on. Depending on the answer you can guess if the port is opened, closed or filtered (see part I.1).
SYN scan with nmap example :
nmap -sS 192.168.0.2
Because SYN scan is agressive and can be detected by firewalls another more "TCP friendly" method can be used. It is called full connect scan. Instead of just sending a SYN scan and a RST if the port is opened, the full connect scan complete the TCP connection procedure :
1 - SYN is sent
2 - If SYN-ACK is received, ACK is sent
3 - RST-ACK is sent to close the connection.
Connect scan with nmap example :
nmap -sT 192.168.0.2

Preventing SYN scan is difficult for a firewall because starting a connection with a SYN packet is the normal way. Some IDS can use statistics to detect fast and abnormal SYN packets hitting various ports and raise an alert.
In the Thylacine firewall I use a trick against SYN scan. I called that trick "trapped ports". When you think about it, you never opened all your ports, so imagine that some ports you are sure you do not need are trapped to react against any connection attempted to them...
For example, you know that any port scan attempt will target telnet port (TCP 23). Telnet is obsolete since a few years now and I assume that if you have a minimum security knowledge, you never use telnet. That makes port 23 a perfect candidate for a trapped port. How?

If port 23 receive any open connection attempt (SYN packet), blacklist the distant IP for three minutes!
A practical example now (note in the example I trapped port 23 and 79, you can modify that list and put whatever ports you do not need).

These iptables setting will log and block most SYN stealth scan attempts. However there is a limit, if the attacker is cautious and has some time, he will use very slow SYN scan (a handful of port with one connection attempt every 5 minutes). Our "trapped port" technique won’t prevent that type of scan but the good thing is it will log the connection attempt to the trapped port (with informations like the IP of the attacker), so at least we know something is going wrong.

2.6 UDP scan

Scanning UDP open ports is not easy because UDP is not connected so you do not get any response if the port is open. The normal UDP port reaction when receiving a packet is, if the port is closed, is to return an "ICMP Port Unreachable" packet. If nothing is returned you can suppose the port is opened or filtered by a firewall.

UDP scan with nmap example :
nmap -sU 192.168.0.2
If nmap receives an ICMP Port Unreachable packet port is considered closed, if there is no response, to port is considered Opened/filtered.
In the Thylacine firewall, I use a simple mechanism to prevent UDP scans. When you analyze an UDP scan (nmap scan or default hping3 packet), you can see it does not have any data in it, just the UDP header. This is highly unlikely to happen in the real world where UDP is used by the application layer and carries data (ex DNS, DHCP, streaming, etc.). So a quick way to block UDP scan attempts consist to block all UDP packets with no content.

Note : UDP scan will still be available if data is added in the packet. Example (the port 53 is closed) :
hping3 -c 1 --udp -p 53 192.168.0.2
 -> Returns nothing.
hping3 -c 1 --udp -p 53 -d 12 192.168.0.2
 -> Returns ICMP Port Unreachable.

This is it for the second part , Port scanning.
You can now read the third part about D.O.S attacks