OpenBSD Timestamps

It’s not breaking news that OpenBSD strives to be at the forefront of security. It appears that their TCP timestamp option behavior reflects this as well. The purpose of the TCP timestamp is to measure roundtrip times as well as to act as an extension to the TCP sequence numbers by disambiguating two segments that arrive simultaneously, have identical TCP sequence numbers, but contain different TCP timestamps such as in the case of wrapping TCP sequence numbers. The likelihood of this happening is slim to none, however, if it does, the segment with the later timestamp is favored while the one with the older timestamp is dropped. I’ve found a variety of different OS-specific behaviors regarding old TCP timestamps that offer a wealth of opportunities for IDS/IPS evasion.

Most operating systems employ a TCP timestamp behavior where the sender’s value reflects the number of clock ticks since the last reboot. A clock tick is a vague concept described in RFC 1323 as “The timestamp value to be sent is to be obtained from a (virtual) clock that we call the "timestamp clock". Its values must be at least approximately proportional to real time…”. Now, think about this in terms of security. A high TCP timestamp divulges that the host has not been rebooted in a while. Now, let’s say there is a patch released on MS Tuesday for a new Windows remote exploit. Further, installation of this patch requires a reboot. An attacker searching for victim hosts could gain some valuable reconnaissance just from a TCP timestamp if he/she could communicate with a potential target host via TCP.

OpenBSD must have seen the possible leakage of information from the TCP timestamp so they now randomize the first timestamp on every new TCP/IP session. They increment the timestamp in subsequent segments in the session to meet the standard of approximating the passage of time for accurate round trip time measurement (RTTM). But, they eliminate the possibility for the TCP timestamp to be used to reflect the uptime since last reboot.

Let’s take a look at the difference between the more conventional incremental Windows TCP timestamp and the randomized OpenBSD timestamp. I sent five SYNs in rapid succession to a Windows host first and then to an OpenBSD host to examine the TCP timestamp values in the returned SYN/ACK. I’ve edited the tcpdump output to contain only the pertinent information to save space. As you can see the Windows host 75.101.150.118 generates incremental TCP timestamp values.

Now, look at the OpenBSD random timestamp values from host 142.244.12.42.

While this is impressive in itself, I believe OpenBSD has relegated the use of TCP timestamps exclusively for RTTM and not for the almost useless Protection of Wrapped Sequence Numbers (PAWS) that distinguishes between old and new timestamps. As far as I’m concerned the RTTM is the only reason to support TCP timestamps so I’m pleased if this is the case. I say “if this is the case” since I don’t have an accessible OpenBSD to send my Scapy generated traffic so I have to believe that the Netcraft association of www. openbsd.org and www.bsd.org with OpenBSD/NetBSD OS is correct. My only hesitation is that there may be some kind of device that exists between the sending and receiving boxes that acts as a proxy or somehow permutes the TCP timestamp from the actual host. But, I tried two different hosts that reside on different networks to help bolster my confidence that I’m communicating with actual OpenBSD hosts.

My experiment is to try to get the receiving host to drop all segments after the initial SYN by assigning them a lower timestamp. Using Scapy, I place a timestamp value of 100 on the SYN and 0 on all subsequent ones. First, let’s look at how Windows handles this. It never acknowledges any data I send to it, echoes the expecected timestamp of "ecr 100", and even attempts to resend the SYN/ACK in the final segment to give me another chance to send an appropriate higher timestamp in my initial ACK. This is the expected behavior and conforms to RFC 1323 specification.

Conversely, take a look at the same traffic sent to an OpenBSD host 142.244.12.42. It happily accepts and echoes the old timestamp value of 0 and returns the requested web page, albeit in reverse order, meaning it accepted the segment with the old timestamp carrying the web request.

In an effort to corroborate the behavior above, I wanted to test another OpenBSD host – www.bsd.org - that belongs to a totally different network address block with different devices standing between my network and the web server. As you can see host 192.231.225.11 parrots this same behavior of ignoring the old timestamp.

Initially, I thought there might be something wrong with the traffic I was sending. I know some operating systems require you to echo the timestamp they issue in order to receive subsequent segments. I typically don’t do this because it’s a few more lines of code, but I added the code to echo the server’s timestamps and I still witnessed the behavior of OpenBSD ignoring old timestamps. If this is indeed the case, I think OpenBSD has made a good, and as usual, innovative decision to support only the useful features of the TCP timestamp.

My Mistake - Suricata Reassembles on All TCP Ports

Yesterday I erroneously blogged that Suricata does not perform TCP stream reassembly on non-standard ports. The script I used to test this in live mode had an incorrect TCP reset sequence number on it and an alert was not generated. When I corrected this, I did indeed see an alert.

My apologies for reporting this and my apologies to the Suricata team. To be clear - Suricata performs TCP stream reassembly on all TCP ports. I am about to correct yesterday's blog by deleting the misinformation.

Suricata TCP Evasions

If you’re currently using the Open Source intrusion detection/prevention system Suricata, I urge you to upgrade to the recently released version 1.0.2 as soon as possible. In my recent research, I've found more than half a dozen TCP evasions in code from prior versions.

In my experience with Open Source intrusion detection systems like Snort, accurate and thorough TCP reassembly is extremely difficult. Snort's TCP stream reassembly code has transitioned from being pretty lame and rudimentary (see Stick and Snot attacks) into to a flawed stream4 implementation, finally evolving into a solid and comprehensive stream5 module--over a decade later.

I've been interested in Suricata's stream reassembly code since they offered their first beta release over a year ago. When they released their first non-beta stable 1.0 release this past July, I felt it was finally fair game to try some evasions. Unfortunately, it was fairly easy to find them. Some, like the failure to properly perform TCP checksum validation, were trivial; others like the one discussed below took more poking.

My intent was to find evasions that were more or less universal – not affiliated with a particular operating system (AKA target-based). I crafted the evasion techniques with Scapy and sent them to a destination operating system running Linux first and later Windows Vista. Obviously, TCP evasions are most dangerous when Windows is the destination host since Windows is still the most prevalent OS. In all fairness, I ran any TCP evasion that I discovered against Suricata later against a current version of Snort – 2.8.6.1. Snort alerted on all of the techniques that successfully evaded Suricata.

Let’s examine one of the several Suricata TCP evasions I was able to uncover in versions prior to 1.0.2. Suricata uses Snort rules. Assume that the following Snort rule exists and is configured for use:

alert tcp any any -> any 80 (msg:"EVILSTUFF alert"; flow:to_server, established; content: "EVILSTUFF"; http_uri; sid:1234; )

This rule looks for the content of "EVILSTUFF" in an HTTP URL.

First, here is the Scapy code that I used. I'm not going to go over the specifics of the code since I covered how to create the client side of the code in the blog post on an unusual response from a bad acknowledgement on the last segment of the three-way handshake. The code for this evasion shares many of the concepts and some of the code.



Now, let's take a look at the client side traffic that the the Scapy code generated.



Segment 1 represents a SYN with a crafted Initial Sequence Number (ISN) of 10. The next crafted segment 2 shows the acknowledgement of the server's SYN/ACK. Next, I reset the connection in segment 3. This should be the end of tracking this session.

Segment 4 is a SYN sent about 5 seconds after the reset and is identical to the first SYN, except it uses an ISN of 11. Segment 5 completes the acknowledgement of the three-way handshake from the new session and segments 6 and 7 contain the malicious split content of "EVILSTUFF". Suricata fails to alert here; we assume this is because it gets confused after improperly closing/cleaning up after the first session and mistakenly considers the new session as a continuation of old one. It doesn't track the new session SYN with the ISN of 11 and when the malicious content is sent later—with a sequence number one more than Suricata expects—it creates a gap in sequence numbers for Suricata. Suricata cannot properly reassemble the malicious content since there is a missing sequence number before it, causing the evasion.

The failure to properly close one session and distinguish it from a new one is neither a trivial nor a target-based issue. So please make sure that you upgrade right away if you are currently using Suricata.

Additionally, if you have or are considering dumping Snort in favor of Suricata, my advice is: Don't do it yet! Suricata is a work in progress. I don't mean to diminish the hard work of some very talented people, but the product needs to mature before I would consider using it as my exclusive means of IDS/IPS. And please note that I've scrutinized only a specific aspect of the product and have found what I consider to be substantial issues. The Suricata team has fixed the issues I reported, so the stream reassembly has improved.

In the interest of full disclosure, I worked at Sourcefire on the Vulnerability Research Team for about 5 years. Specifically, I did research on Snort's current stream reassembly preprocessor known as stream5. Steve Sturges translated the research and improved stream4 to produce stream5. It was at Sourcefire that I learned Scapy and began to use it to try to evade other companies' TCP stream reassembly as well as help build a regression suite to ensure that code changes to stream5 or other modules didn't accidentally introduce unintended consequences. So, yes – I am somewhat biased in my preferences for Snort; I very much appreciate the opportunities presented to me at Sourcefire and I have an abundance of loyalty and allegiance to this day. But, I've got no current financial or other investment in Sourcefire, so I've nothing to gain monetarily from exposing Suricata's TCP reassembly issues.

I'd like to thank Todd Wease, who used to work for Sourcefire on the Snort team, for validating the evasions I found and for examining the code to help me understand the logic issues. And, I'd like to thank Jen Harvey, Co-Founder at Voxilate, for reviewing this post and cleaning it up so it is coherent.

As a final note: You, too, can attempt to find evasions such as these in your IDS/IPSes packet-crafting with Scapy. There’s still time and room to sign up for my Power Packet Crafting Using Scapy course in Las Vegas at the end of September. Please come join me in Las Vegas in September!

Using Scapy to Select a Range of pcap Records

I was reading about Wireshark vulnerabilities that could cause a DoS or buffer overflow. I downloaded the pcap associated with one of the vulnerabilities and read it using a vulnerable Wireshark version 1.2.6. Sure enough, it caused Wireshark to display a gray screen and upon trying to close it, it displayed the following:



I was interested in finding the record(s) that caused the crash by methodically selecting given ranges of the records and narrowing down the problem records(s). At first I thought of using tcpdump to select some records and write only those records to a new output file and reading the new output file into Wireshark. Doing this using tcpdump is not possible because it has a command line switch (-c) that allows you to specify the number of records you want to read, but always begins reading at the first record. So, I wrote a small Python/Scapy program that would allow me to select the starting and ending records from the pcap, write the selected records to a Python list, feed the records into Wireshark and observe whether or not it crashed. It didn’t take me long to find the problem record.

Here is the program I used:

from scapy.all import *
import sys

recs=rdpcap("/tmp/fuzz-2010-06-02-11020.pcap")
lrecs=len(recs)
outrecs=[ ]
start=int(sys.argv[1])-1
end=int(sys.argv[2])-1

if (end >=lrecs):
....print "End value greater than number of pcap records"
....sys.exit()

x=start
while (x <= end):
....outrecs.append(recs[x])
....x=x+1

wireshark(outrecs)


There really isn’t much to it and there is only one statement that uses Scapy. The program starts by importing the Scapy and Python modules required for the program. I stored the downloaded pcap from http://www.wireshark.org/download/automated/captures/fuzz-2010-06-02-11020.pcap in file /tmp/fuzz-06-02-11020.pcap and used the Scapy “rdpcap” to read the pcap into a Python list named “recs”. Next, I created an empty Python list called “outrecs” to stored the user selected records.

The next block of code simply takes two command line arguments and stores the first in an integer named “start” and the second into an integer named “end”. This is the range of records to select and feed into Wireshark. Then there is a check to make sure that the value of the last record selected is less than or equal to the number of records in the pcap. Note, I've used periods to represent spaces since Python requires indentation after the "if" statement and the "while" statement. I did this because the spaces I tried to use were removed by the HTML formatting of the blog software - sorry.

The next block of code creates a loop to read through the list of pcap records and select and append the ones that fall in the user-supplied range into the output list “outrecs”. Scapy has a built-in functionality to invoke Wireshark. The selected records are fed into Wireshark. With a little trial and error, you can discover which record causes the DoS.

My challenge to you is to tell me the record number that causes Wireshark to crash. Make sure you have Scapy installed . Also, make sure that you have a vulnerable version of Wireshark (versions 0.8.20 through 1.2.8) and make sure to supply to “rdpcap” the file location where you stored the downloaded pcap http://www.wireshark.org/download/automated/captures/fuzz-2010-06-02-11020.pcap. There are 704 records in the pcap so that's the upper bounds for the input range.

I stored the Python program in a file named “find-badrec.py”. Now, suppose I want to test to see if the bad record falls in the range of records from 1-10, I’d run it from the command line using:

python find-badrec.py 1 10

The program displays the following Wireshark output and does not crash. I would close Wireshark and press CTRL-C to stop the Scapy program to try again with a different range of values.



Time is running out to sign up for my Scapy course at SANS Network Security in Las Vegas. I cover reading, writing, and altering pcaps using Scapy along with a bunch of other very useful features of Scapy. If you’re already taking a multi-day track at the conference, I’m teaching the one-day course after these end.

This course is a nice complement to SEC503 “Intrusion Detection In-Depth”, SEC502 “Perimeter Detection In-Depth”, SEC504 “Hacker Techniques, Exploits & Incident Handling”, and SEC560 “Network Penetration Testing and Ethical Hacking”. These are just some tracks where you can use Scapy as an additional tool to implement what you’ve learned. I hope to see you there!

Windows LAN Addressing Validation (and a Scapy lesson)

Following the AirTight “hole196” announcement affecting WPA/WPA2-Enterprise networks (see my WillHackForSushi article), I was speaking with a friend who told me how attractive yet naive I am regarding how clients respond to LAN traffic sent to the broadcast address with unicast payloads.  Essentially, I was unaware as to how a client would handle a unicast upper-layer payload (such as a TCP SYN request) when delivered to a broadcast MAC address.

So, mull it over I did, with the help of Scapy.  I wanted to experience first-hand how different clients behaved when they receive unicast IP payloads, sent to a layer 2 broadcast address.  This is a very straightforward test with Scapy, sent in these examples over a wired LAN card.

First, I start Scapy and create an Ethernet header in the variable “p”, using the broadcast destination address, and the source address of my Linux box running Scapy.  The type code 0x0800 indicates that the next payload is an IP packet:

$ sudo scapy
>>> p = Ether(dst="ff:ff:ff:ff:ff:ff", src="00:18:8b:ad:2a:c7", type=0x0800)

Next, I add an IP header, specifying the source of my Linux box and the destination of my Windows 7 box.  I add an ICMP payload as well (using the default type/code of ICMP Echo Request) and display the packet contents:

>>> p /= IP(src="172.16.0.109", dst="172.16.0.113")
>>> p /= ICMP()
>>> p
<Ether  dst=ff:ff:ff:ff:ff:ff src=00:18:8b:ad:2a:c7 type=0x800 |<IP
frag=0 proto=icmp src=172.16.0.109 dst=172.16.0.113 |<ICMP  |>>>

With my finished packet, I call the srp1() Scapy function to send and receive the packet (using srp1() as opposed to sr1() since I am sending a layer 2 packet) and display the associated packet response:

>>> srp1(p)
Begin emission:
Finished to send 1 packets.
.*
Received 2 packets, got 1 answers, remaining 0 packets
<Ether  dst=00:18:8b:ad:2a:c7 src=00:21:5c:7e:70:c3 type=0x800 |<IP
version=4L ihl=5L tos=0x0 len=28 id=31066 flags= frag=0L ttl=128
proto=icmp chksum=0x6888 src=172.16.0.113 dst=172.16.0.109 options=[]
|<ICMP  type=echo-reply code=0 chksum=0xffff id=0x0 seq=0x0 |<Padding
load='\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
|>>>>

Much to my surprise, the Windows 7 host responds to my unicast Echo Request sent to the broadcast LAN address with an ICMP Echo Reply.  Next I wondered if the Windows 7 target validated the Ethernet source address.  I created a second packet, similar to the first but this time using the broadcast LAN address as the source:

>>> p = Ether(dst="ff:ff:ff:ff:ff:ff", src="ff:ff:ff:ff:ff:ff", type=0x0800)
>>> p /= IP(src="172.16.0.109", dst="172.16.0.113")
>>> p /= ICMP()
>>> srp1(p)
Begin emission:
Finished to send 1 packets.
...*
Received 4 packets, got 1 answers, remaining 0 packets
<Ether  dst=00:18:8b:ad:2a:c7 src=00:21:5c:7e:70:c3 type=0x800 |<IP
version=4L ihl=5L tos=0x0 len=28 id=31603 flags= frag=0L ttl=128
proto=icmp chksum=0x666f src=172.16.0.113 dst=172.16.0.109 options=[]
|<ICMP  type=echo-reply code=0 chksum=0xffff id=0x0 seq=0x0 |<Padding
load='\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
|>>>>

Again the target responds to the Echo Request.  Note that the response goes to the correct unicast MAC address, as opposed to the source included in the Echo Request packet, indicating that Windows uses the ARP cache to populate the Ethernet header as opposed to just swapping the address fields in the originating packet.

I was surprised to learn this LAN addressing technique also works with TCP:

>>> p = Ether(dst="ff:ff:ff:ff:ff:ff", src="00:18:8b:ad:2a:c7", type=0x0800)
>>> p /= IP(src="172.16.0.109", dst="172.16.0.113")
>>> p /= TCP(sport=65000, dport=135)
>>> srp1(p)
Begin emission:
Finished to send 1 packets.
...........*
Received 12 packets, got 1 answers, remaining 0 packets

<Ether  dst=00:18:8b:ad:2a:c7 src=00:21:5c:7e:70:c3 type=0x800 |<IP
version=4L ihl=5L tos=0x0 len=44 id=545 flags=DF frag=0L ttl=128
proto=tcp chksum=0x9fac src=172.16.0.113 dst=172.16.0.109 options=[]
|<TCP  sport=loc_srv dport=65000 seq=4159458989L ack=1 dataofs=6L
reserved=0L flags=SA window=8192 chksum=0xda0c urgptr=0 options=[('MSS',
1460)] |<Padding  load='\x00\x00' |>>>>

I continued this testing against other clients as well.  Windows Vista systems behave the same as my Windows 7 target; I did not have a Windows XP box handy for testing.  Interestingly, Linux 2.4, Linux 2.6 and OS X 10.6 systems do NOT behave the same as Windows.  These systems reject LAN broadcast traffic sent to unicast IP addresses.

What does this mean?  First, Scapy rocks, and you should check our Judy’s Scapy class with SANS.  Second, leveraging the “hole196” vulnerability, a malicious insider could send traffic to other WLAN clients without going through the LAN, evading wired IDS systems (but otherwise not creating an additional network exposure).  I have a nagging feeling this behavior is not good (tm) otherwise, but I haven’t figured out why yet.  If you have an idea, please share it in the comments below, or drop me a note.  Thanks!

-Josh