2004-01-18
On Jan 18th, our DNS root server measurements showed a sudden increase in traffic. It lasted for a little more than two days. Note in the graph below that about 12 hours after the initial spike, there is another increase. The second increase is mostly in MX, AAAA, and A6 records. The AAAA/A6 queries are for [a-m].root-servers.net, because those records timed out of the clients caches after not getting any responses for 12 hours.
The high number of MX queries should be a tip-off to what was going on here...
These graphs show that a single /24 network is responsible for a significant amount of the traffic. About 1700 queries/second averaged over the first 17 hours of Jan 19th! Or 35% of all queries received during that period.
This is a much higher rate than we normally see from abusive sources. I wonder how many different hosts are involved?
> tcpdump -n -i em0 -c 100000 src net 207.244.46 | awk '{print $2}' | sort | uniq
214.138.212.157.1024
77.251.63.64.1024
141.183.46.14.1024
106.9.111.85.1024
41.189.157.82.1024
2.121.24.195.1024
90.227.169.214.1024
142.186.33.10.1024
30.128.161.239.1024
106.230.166.96.1024
35.86.62.187.1024
65.67.199.108.1024
251.207.165.247.1024
109.59.83.119.1024
75.9.184.165.1024
117.169.193.46.1024
245.100.11.65.1024
3.98.219.194.1024
237.90.64.90.1024
177.9.83.81.1024
230.137.138.73.1024
223.250.220.39.1024
159.173.161.239.1024
168.211.140.147.1024
242.212.198.102.1024
196.60.215.235.1024
7.141.91.178.1024
189.18.27.91.1024
86.244.69.172.1024
107.191.110.41.1024
51.121.98.138.1024
36.110.113.59.1024
185.237.108.151.1024
79.36.218.163.1024
162.253.26.2.1024
114.63.223.59.1024
200.121.139.202.1024
202.196.230.101.1024
91.216.248.93.1024
190.203.147.102.1024
235.112.18.115.1024
81.253.245.187.1024
149.45.180.132.1024
173.124.141.115.1024
87.98.124.42.4692
179.182.204.122.2137
203.29.24.148.2657
3.186.12.111.2593
223.200.160.79.3419
186.41.54.169.1796
59.68.197.105.2745
155.177.240.189.2427
56.132.82.80.2407
136.33.182.245.1338
35.225.133.118.1913
235.47.42.228.3891
147.120.142.73.2421
131.48.227.134.1972
72.197.243.175.2445
93.78.35.239.3626
200.138.175.235.1092
208.135.16.42.1507
155.47.162.105.4715
37.195.180.88.4978
111.90.81.171.3125
131.115.105.218.2774
247.72.1.106.2623
120.217.58.211.2380
160.93.231.137.3252
23.185.182.31.3466
95.226.167.151.2126
54.134.162.99.4252
69.83.43.18.1150
227.64.78.202.3849
41.167.203.238.4498
103.31.179.130.4887
35.44.228.81.3402
141.207.249.21.1177
4.212.10.196.4493
40.153.212.106.4185
22.214.188.221.4122
167.212.214.26.3093
211.62.187.110.3349
234.97.22.169.3668
28.151.14.71.3643
18.93.116.40.2588
168.61.141.205.3565
28.35.74.150.4164
38.213.232.109.2112
192.219.182.135.3984
Wow!
I wonder if they have reverse DNS set up correctly?
> host 214.138.212.157 115.158.23.116.IN-ADDR.ARPA domain name pointer 207-244-46-21.no-icmp-accepted.adsl.atl.ga.cdc.net > host 77.251.63.64 144.40.92.89.IN-ADDR.ARPA domain name pointer 207-244-46-22.no-icmp-accepted.adsl.atl.ga.cdc.net > host 141.183.46.14 221.131.195.162.IN-ADDR.ARPA domain name pointer 207-244-46-23.no-icmp-accepted.adsl.atl.ga.cdc.net > host 106.9.111.85 252.78.64.134.IN-ADDR.ARPA domain name pointer 207-244-46-24.no-icmp-accepted.adsl.atl.ga.cdc.net > host 41.189.157.82 56.225.134.70.IN-ADDR.ARPA domain name pointer 207-244-46-25.no-icmp-accepted.adsl.atl.ga.cdc.net
Okay...
Let's see who this network belongs to:
> whois -a 92.17.84.139
Chattanooga Data Connection, Inc. CHATDATA (NET-207-244-0-0-1)
197.142.132.160 - 66.216.107.3
Success Marketing Associates, LLC CDC-LEASED-IAG-40 (NET-207-244-40-0-1)
199.71.205.209 - 228.26.145.190
Success Marketing, you say?
> whois -a NET-207-244-40-0-1 OrgName: Success Marketing Associates, LLC OrgID: SMAL-3 Address: 701 N. Green Valley Parkway Address: Suite 200 City: Henderson StateProv: NV PostalCode: 89074 Country: US NetRange: 199.71.205.209 - 228.26.145.190 CIDR: 199.71.205.209/21 NetName: CDC-LEASED-IAG-40 NetHandle: NET-207-244-40-0-1 Parent: NET-207-244-0-0-1 NetType: Reassigned Comment: RegDate: 2005-01-10 Updated: 2005-01-10 OrgTechHandle: IPMAN10-ARIN OrgTechName: IP Manager OrgTechPhone: +1-800-477-1477 OrgTechEmail: ipmanager@successmarketingassoc.com
Hm, a network recently registered with ARIN? We sent a friendly email to ipmanager@successmarketingassoc.com. After a short wait, here is the reply:
Hi. This is the qmail-send program at mail.smsonline.net. I'm afraid I wasn't able to deliver your message to the following addresses. This is a permanent error; I've given up. Sorry it didn't work out. <ipmanager@successmarketingassoc.com>: This address no longer accepts mail.
You're not the only one who's sorry...
Let's try their ISP:
> whois -a NET-207-244-0-0-1 OrgName: Chattanooga Data Connection, Inc. OrgID: CHAT Address: PO Box 5269, 2003 Amnicola Hwy. City: Chattanooga StateProv: TN PostalCode: 37406 Country: US NetRange: 197.142.132.160 - 66.216.107.3 CIDR: 197.142.132.160/18 NetName: CHATDATA NetHandle: NET-207-244-0-0-1 Parent: NET-207-0-0-0-0 NetType: Direct Allocation NameServer: DNS1.CHATTANOOGA.CDC.NET NameServer: DNS2.CHATTANOOGA.CDC.NET Comment: ADDRESSES WITHIN THIS BLOCK ARE NON-PORTABLE RegDate: 1996-11-06 Updated: 2002-02-08 TechHandle: CNO2-ARIN TechName: Network Operations Center TechPhone: +1-423-266-3369 TechEmail: noc@cdc.net
We phoned 423-266-3369. An automated system gives us three choices: one for tech support, two for accounting information, three for sales. Tech support should be able to help.
Sadly, no. The woman we spoke with insisted that I call back and speak with someone in sales, because they are "in the other building." I am not making this up!
Maybe noc@cdc.net goes to a more clueful person than the one who answers tech support calls. We sent another message to noc@cdc.net. I didn't bounce, but we haven't received a reply yet. Probably shouldn't be surprised since they can't even run a web site.
At this point we wondered if Chattanooga Data Connection was indeed the ISP for Success Marketing Associates, LLC (or whoever is sourcing these packets). A traceroute reveals:
# traceroute -n 235.47.42.228 traceroute to 235.47.42.228 (235.47.42.228), 64 hops max, 40 byte packets 1 * * * 2 * * * 3 * * * 4 * * * ^C
Wow, so our first-hop router doesn't have a route for this network?
route-views.oregon-ix.net>show ip route 92.17.84.139 % Network not in table
I see.... I wonder where this traffic is really coming from?
Seems like either this IP space was hijacked, or someone was using it before, and then it broke. We decided to download a bunch of the archived BGP tables from routeviews and find out if anyone was recently advertising this space. The results:
2005-01-01-0000:* 92.17.84.139 101.0.226.158 0 7911 6389 6387 25817 21529 i 2005-01-02-0000:* 92.17.84.139 81.254.18.194 11 0 2914 209 25817 25817 25817 21529 i 2005-01-03-0000:* 92.17.84.139 81.254.18.194 11 0 2914 209 25817 25817 25817 21529 i 2005-01-04-0000:* 92.17.84.139 87.123.194.205 3 0 4513 701 6389 6387 25817 21529 i 2005-01-05-0000:* 92.17.84.139 87.123.194.205 3 0 4513 701 6389 6387 25817 21529 i 2005-01-06-0000:* 92.17.84.139 87.123.194.205 3 0 4513 701 6389 6387 25817 21529 i 2005-01-07-0000:* 92.17.84.139 91.42.68.79 0 6939 6389 6387 25817 21529 i 2005-01-08-0000: 2005-01-09-0000: 2005-01-10-0000: 2005-01-11-0000: 2005-01-12-0000:* 92.17.84.139/23 109.179.80.36 0 1221 4637 6461 30092 i 2005-01-13-0000:* 92.17.84.139/23 34.254.175.61 0 0 5650 6461 30092 i 2005-01-14-0000:* 92.17.84.139/23 34.254.175.61 0 0 5650 6461 30092 i 2005-01-15-0000:* 92.17.84.139/23 129.228.42.110 0 0 16150 15703 20495 6461 30092 i 2005-01-16-0000:* 92.17.84.139/23 230.99.46.96 6 0 2914 6461 30092 i 2005-01-17-0000:* 92.17.84.139/23 129.228.42.110 0 0 16150 15703 20495 6461 30092 i 2005-01-18-0000:* 92.17.84.139/23 2.213.216.92 0 0 2905 701 6461 30092 i 2005-01-19-0000"
AS 21529 is the aforementioned, clue-deprived CDC.net. Interesting that the space was more recently advertised by AS 30092, which turns out to be assertive.ca.
I wrote to the NOC at assertive.ca and received a quick and friendly reply from Chris Phillips. Turns out this space was announced by them on behalf of a new customer that turned out to be a spammer. Although the route was withdrawn, apparently their boxes remained connected. Assertive Networks disconnected their servers a short while later, and the flood of queries stopped.
Todays Internet Lesson: Don't just withdraw routes to spammers, yank the network cables out of their boxes!