Bruce Schneier

 
 

Schneier on Security

A blog covering security and security technology.

« Reverse-Engineering Exploits from Patches | Main | Designing Processors to Support Hacking »

April 24, 2008

Hacking ISP Error Pages

This is a big deal:

At issue is a growing trend in which ISPs subvert the Domain Name System, or DNS, which translates website names into numeric addresses.

When users visit a website like Wired.com, the DNS system maps the domain name into an IP address such as 72.246.49.48. But if a particular site does not exist, the DNS server tells the browser that there's no such listing and a simple error message should be displayed.

But starting in August 2006, Earthlink instead intercepts that Non-Existent Domain (NXDOMAIN) response and sends the IP address of ad-partner Barefruit's server as the answer. When the browser visits that page, the user sees a list of suggestions for what site the user might have actually wanted, along with a search box and Yahoo ads.

The rub comes when a user is asking for a nonexistent subdomain of a real website, such as http://webmale.google.com, where the subdomain webmale doesn't exist (unlike, say, mail in mail.google.com). In this case, the Earthlink/Barefruit ads appear in the browser, while the title bar suggests that it's the official Google site.

As a result, all those subdomains are only as secure as Barefruit's servers, which turned out to be not very secure at all. Barefruit neglected basic web programming techniques, making its servers vulnerable to a malicious JavaScript attack. That meant hackers could have crafted special links to unused subdomains of legitimate websites that, when visited, would serve any content the attacker wanted.

The hacker could, for example, send spam e-mails to Earthlink subscribers with a link to a webpage on money.paypal.com. Visiting that link would take the victim to the hacker's site, and it would look as though they were on a real PayPal page.

Kaminsky demonstrated the vulnerability by finding a way to insert a YouTube video from 80s pop star Rick Astley into Facebook and PayPal domains. But a black hat hacker could instead embed a password-stealing Trojan. The attack might also allow hackers to pretend to be a logged-in user, or to send e-mails and add friends to a Facebook account.

Earthlink isn't alone in substituting ad pages for error messages, according to Kaminsky, who has seen similar behavior from other major ISPs including Verizon, Time Warner, Comcast and Qwest.

Another article.

Posted on April 24, 2008 at 6:43 AM38 CommentsView Blog Reactions

To receive these entries once a month by e-mail, sign up for the Crypto-Gram Newsletter.

Comments

I've seen this is the UK on AOL and Claranet.

Posted by: Sean at April 24, 2008 7:27 AM


Time warner started doing the same thing a while back. They do give you an option to opt out, which is tracked by IP I believe, but they do not advertise it very well.

Posted by: D at April 24, 2008 7:38 AM


Just another reason to be running your own cache. While these days the ISPs are delibrately subverting things, in the past their caches were vulnerable to poisoning.
Until they start putting in transparent DNS proxies, running your own caching name server will save you some grief.

Posted by: Anonymous at April 24, 2008 7:53 AM


@Anonymous

But who, other than us nerds, is able to actually keep something like a transparent proxy working. I wouldn't encourage my Mum to have one on her net connection. It's much easier to just point her router's DNS at a better one.

Posted by: Matthew Schinckel at April 24, 2008 7:58 AM


"Kaminsky demonstrated the vulnerability by finding a way to insert a YouTube video from 80s pop star Rick Astley into Facebook and PayPal domains."

I like the idea of Rickrolling as a vulnerability demo. Financially harmless yet psyche-scarring, it's a good way to drive home the "you've been had".

Posted by: Carlo Graziani at April 24, 2008 8:04 AM


Network Solutions (or was it Verisign?) tried to pull this trick on all of us about 5 years back.

Posted by: Dima at April 24, 2008 8:06 AM


Verizon does this, and it pisses me off. I'm used to typing something in the FireFox address bar and getting an I'm-feeling-lucky search.

This is completely aside from the issues that lack of an NXDOMAIN cause for spam filtering. Verizon has a page that supposedly lets you disable it, but when I checked all the links on it were 404 (they're fixed now, but basically they told you to use a different DNS, so I did).

Posted by: Gordon at April 24, 2008 8:14 AM


A home user wouldn't want to run a transparent proxy. They can run a caching resolver that starts at the root and works downward. Then they don't need to use anyone else's resolver.
However ISPs may want to use transparent proxies to allow them to keep pulling the same shennagins they do now to make extra money.

Posted by: Anonymous at April 24, 2008 8:23 AM


Maybe this is good after all.

At the moment, Google keeps most of its cookies so that they are sent to all subdomains of Google. They do this so they can track your identity as you move through the various Google services. This is also the reason that gmail is read through "mail.google.com", and not "gmail.com".

But if this practice becomes a problem, so does google's practice. Maybe this problem will negate the other one.

Shachar

Posted by: Shachar Shemesh at April 24, 2008 8:40 AM


Domain names are valuable intellectual property to their owners. I believe that a case could be made that an ISP returning a different page for someone else's domain is infringment and theft.

Posted by: Rajiv at April 24, 2008 8:56 AM


Im glad this has finally caught public attention it has been bugging me for quite a while.

Posted by: bob at April 24, 2008 9:05 AM


Why doesn't the target domain return a redirect URL for undefined names within their domain? Ask for "webmale.google.com" and get "www.google.com" back. They can be arbitrarily simple or smart - "music.retailer.com" could take you to the home page or to the music offerings. This puts the control back into the hands of the domain owner. It would also allow "foo.com" to work for as an alias for "www.foo.com" for those lazy typists among us.

Posted by: Andy at April 24, 2008 9:07 AM


I know that TalkTalk in the UK also do this and what's worse is that you are redirected to the ad-laden page (which means you can't just fix the one-letter typo anymore, you have to re-type the entire domain) and the ad-laden page is horrendously slow.

Of course, I changed my DNS servers and I'm no longer with TalkTalk but this is a scary problem.

Shouldn't there be some legislation about returning inaccurate results to DNS requests ?
There isn't much of a conceptual difference between hijacking mail.google.com and hijacking webmail.google.com. The only difference is that Google have decided that one of them should return NXDOMAIN and the other should return an IP address.

Posted by: Dave at April 24, 2008 9:08 AM


This is the reason why I do not let my ISP resolve the DNS lookup. Instead I use an outside firm in another place and time continuum to do the task.

Posted by: Anonymous at April 24, 2008 9:14 AM


What do these ad sites do in response to non-http traffic? If the ad site was malicious (or compromised) and someone misspells the name of a domain the first time they SSH to it (or if they ignore the warning that they haven't connected before), they could divulge their username and password.

Posted by: Trevor Stone at April 24, 2008 9:20 AM


As long as the ISPs dont mind accepting financial liability for the damage they cause; plus lost time and underwriting lost identities incurred when they usurp the 404 mission, its fine with me. I know I never signed a waiver for them to redirect me to their choice of site when I mistyupe something. If they send me to a malware site, they can take some of the revenues they gain through their negligent greed and share it with me to reimburse me for my suffering.

I bet a couple of $M+ lawsuit settlements would quickly put this back the way God and DARPA intended.

Posted by: bob at April 24, 2008 9:21 AM


Verizon lets you "opt out". Figure out what DNS servers you're automagically assigned, then change both so that the last 12 is a 14.

Posted by: Capt. Jean-Luc Pikachu at April 24, 2008 9:29 AM


This basically comes down to a problem of choice. If we actually had competition, the company promising not to do this (and keeping that promise) would get all of the business, even if it cost a few cents more per month. Yet because the marketplace can only hold so many ISPs (due to monopoly and expiration of the access agreements that let smaller ISPs have access to the cable plant at reasonable prices) our only choices (in the US) for "high speed" are the phone company DSL (likely AT&T) or the local wired cable television network. In many locations, only one of the two is available.

Competition is a good thing, but monopolies don't see it that way.

Posted by: derf at April 24, 2008 9:34 AM


> What do these ad sites do in response to non-http traffic?

The issue is on the DNS level, it's not in any way specific to HTTP traffic.

If you have an ISP that is hijacking like this, you can test it using the "nslookup" or "host" utilities (Windows and modern Linux distros, respectively). You can type in any bogus domain name you want, and instead of getting a NXDOMAIN, you'll get an A record that corresponds to the advertising server.

So basically, if you're trying to connect via SSH (or some other non-HTTP protocol) and mistype the address, rather than just getting a DNS error, you'll try to connect to the ISP advertising server. What happens at that point depends on how the server is configured; I expect you'll get a 'connection refused' message.

This idea was crap when Verisign tried to pull it a few years ago, it's crap when OpenDNS does it, and it's crap when ISPs do it. Unfortunately, until it's made illegal, I think we'll see people continuing to try it. It's just too lucrative a revenue source; it doesn't matter if it breaks all sorts of software, if it makes them a few quick bucks they're going to do it if they're allowed.

Personally I use the OpenNIC alternate-root nameservers for DNS resolution, and they've been working well for me. It's most of the good things that you can say about OpenDNS, but without wildcarding/typosquatting. They also have a few TLDs that don't exist in the IANA root, which might be good or bad depending on your point of view. I just ignore them, mostly.

Posted by: Kadin2048 at April 24, 2008 9:58 AM


Three posters have mentioned getting better DNS, but nobody says where. C'mon, guys; share the good news.

Posted by: Peter Pearson at April 24, 2008 9:59 AM


Just another reason to be running your own cache. While these days the ISPs are delibrately subverting things, in the past their caches were vulnerable to poisoning.
Until they start putting in transparent DNS proxies, running your own caching name server will save you some grief.

Posted by: Leon Avalos at April 24, 2008 10:10 AM


Well, I use this for my *own* domains all the time. No matter what subdomain you ask, you will get a valid IP.

If you use HTTP and try a non-existing subdomain, a default virtual host will inform you about this problem. Btw, you wouldn't believe how many people think that host names must start with "www."...
For non-HTTP protocols, you get whatever listens at that port, as hostnames do not matter there.

I originally set this up so that I could conveniently add subdomains (which mostly redirect to subdirectories of the primary website) by only changing Apache configuration.
Now it turns out that my setup even has security benefits ;-D

Posted by: Paeniteo at April 24, 2008 10:10 AM


I use Verizon and it's annoying because it's not only for an unknown subdomain but also for when they're having network issues and cannot bring up reliable sites like google.com... Badness... Make it stop!

Posted by: FDHY at April 24, 2008 10:20 AM


Just thinking aloud. But an interesting if nerdy piece of research would be to query two of a list of different DNS's and compare the results. There'd be some devil in the details but vastly different IPs could be rejected.

You could monitor when your ISP was playing games with you.

Put this in a client and provide feedback to a central site to monitor whos playing games and bring it more out in the open.

Posted by: anonymous at April 24, 2008 10:21 AM


@ Peter Pearson (and anyone else who wants a better DNS server)

Try 4.2.2.1-4.2.2.6. They're run by Level 3 and are quite fast. I've had response times as low as 11 ms.

I also recommend a caching DNS server for high-traffic periods. pdnsd is easy to set up, if you happen to have a Unix box available.

Posted by: skymt at April 24, 2008 10:52 AM


I use TimeWarner for me it only returns ads on bad domains & TLD (ie www.gggooooglleee.com. / www.google.edu) but gives me a 404 on bad subdomains (pay.google.com.)
I had never seen the opt out untill I went looking for it after reading this article. I have not yet determined how it tracks prefrences

Posted by: Boogle at April 24, 2008 11:00 AM


Well I'm very happy with OpenDNS and it's features, as well as DNS-Omatic for Keeping my no-ip addresses updated.

Posted by: Pawned! at April 24, 2008 11:03 AM


ISPs are taking such a huge interest in manipulating data in transit, regardless of the ethics or security risks, that this kind of problem isn't going to go away without legislation.

The market can't solve this problem because limited choices for last mile connectivity (phone or cable) mean most ISPs are essentially monopolies.

We need a dumb pipe act to obligate all network providers to pass IP payloads verbatim. Anything less than this and it's only a matter of time until every protocol and connection is subject to inappropriate monetization.

Posted by: Infosponge at April 24, 2008 12:16 PM


OpenDNS is a really solid alternative, except that I was having some seriously bizarre issues (lookups failing in odd ways, lookup speed getting very slow) with Comcast a couple of months after I set OpenDNS as my only DNS server on my wireless router.

Posted by: tk. at April 24, 2008 12:43 PM


Googler matt cutts probably wouldn't think that this is cool....
http://www.mattcutts.com/blog/confirmed-isp-modifies-google-home-page/

But... they do this themselves via their toolbar:
http://seoker.com/2008/02/11/google-hijacking-404-error-pages/

Whether it is a "hacker" or a search engine, control of your computer is based on understanding your computer and the intent of both the "do no evil" (right) and evil doers.

Posted by: lol at April 24, 2008 1:04 PM


There's also a post at Perimeter Grid about how Kaminsky's script works, and how it could be used to carry out an actual attack.

This is a great example of how the lack of "net neutrality" does more than just give larger, paying websites a "fast lane" -- it also puts all of us at risk by making our own ISPs hostile.

Posted by: Grant at April 24, 2008 1:14 PM


For what it's worth, OpenDNS also hijacks error pages by default. They also invisibly return one of their own servers for lookups of google.com:

http://forums.opendns.com/comments.php?DiscussionID=226
http://blog.opendns.com/2007/05/22/google-turns-the-page/

Both are features you can opt out of. On the other hand, almost all ISPs let you opt out of their DNS hijacking, so there doesn't seem to be any real advantage to switching to OpenDNS, at least in this context.

Posted by: skymt at April 24, 2008 1:56 PM


Charter Communications does it too. It prompted me to set up my router (which runs dd-wrt) to use OpenDNS instead.

As has been mentioned though, OpenDNS does the same damn thing; instead of returning domain not found or something, I get redirected to guide.opendns.com and what look like google results. Better than my ISP's behavior but still not as good as getting the proper error back.

Posted by: John Ridley at April 24, 2008 2:05 PM


@infosponge

"We need a dumb pipe act to obligate all network providers to pass IP payloads verbatim. Anything less than this and it's only a matter of time until every protocol and connection is subject to *inappropriate monetization*."
(emphasis added by me)

What exactly is inappropriate monetization? A legal term? A moral stance? A technical ideal?

I'm all for net neutrality and pipes being pipes, but how does one go about identifying appropriate vs inappropriate monetization of these things? Let alone how does one enforce it?

From the discussion here (I have not done much research on it yet) it appears that this behavior is common and frequently able to opted out of if you choose to. So independant of the bad security at the Ad Hosting site and some mucking with how they present the resulting page as a child domain vs a separate domain (I think I got that right), what is the great offense to net neutrality that they did?

I'm not trying to defend them as much as understand why this is a net neutrality issue. Would it be ok if they presented the hosting page's domain correctly and included a link to the "real" result page?

Posted by: xd0s at April 24, 2008 4:33 PM


I'm surprised that only Rajiv brought up the infringement issue. I'm sure many domain names are trademarked and hijacking nonexistent subdomains sure looks like trademark infringement to me. Of course, I am not a lawyer. I don't even play one on TV.

Posted by: Anonymous at April 24, 2008 6:25 PM


It's not only misspellings that Earthlink is harvesting. We commonly omit the initially "www." from a site name, and that almost always works. When I tried "oup.com/us" I got the infernal page of useless suggestions. I retried with "www.oup.com/us", and that got me what I wanted. Thanks for the helpful service, Earthlink.

Posted by: Otto Defey at April 24, 2008 8:03 PM


"I'm sure many domain names are trademarked and hijacking nonexistent subdomains sure looks like trademark infringement to me."

Since the sub-domains don't actually exist, how can the trademark be infringed?
Here's the relevant UK law:
The Trade Marks Act 1994 states that "a person infringes a registered trade mark if he uses in the course of trade a sign which is identical with the trade mark in relation to goods or services which are identical with those for which it is registered" (section 10(1) of the Act). A person may also infringe a registered trade mark where the sign is similar and the goods or services are similar to those for which the mark is registered and there is a likelihood of confusion on the part of the public as a result (section 10(2)).

A person also infringes a registered trade mark where a sign is identical but the goods are dissimilar if the trade mark has a reputation in the UK and its use takes unfair advantage of, or is detrimental to, the mark’s distinctive character or reputation (section 10(3)).

Looking at it, you might be able to make a case based on Section 10(3), but it seems a bit of a stretch.

IANAL etc.

@Rajiv
"I believe that a case could be made that an ISP returning a different page for someone else's domain is infringment and theft."

Can you explain what has been 'stolen' here?

Posted by: Colossal Squid at April 25, 2008 5:06 AM


"What exactly is inappropriate monetization?"

It's a poorly chosen phrase to describe a very important idea. On the one hand, calling wire fraud "inappropriate" is an incredible understatement, and "monetization" is a red herring when discussing fraud which would be wrong whether done for financial reasons, political or personal reasons, or just for laughs.

On the other hand, the internet works at all only so far as we can trust our ISPs to give us the unedited packets that other users send us, to give us accurate replies to our requests for the services like DNS that they provide, and to never fraudulently impersonate other users. Man-in-the-middle attacks on our network connections are immoral and are supposed to be illegal, even if the attacker is being paid (or perhaps especially if the attacker is being paid) to provide those connections.

Posted by: roystgnr at April 25, 2008 3:49 PM


Post a comment



Real names aren't required, but please give us something to call you. Conversations among several people called "Anonymous" get too confusing.



E-mail is optional and will not be displayed on the site.


Remember Me?


Powered by Movable Type 3.36. Photo at top by Steve Woit.

Schneier.com is a personal website. Opinions expressed are not necessarily those of BT.

 
Bruce Schneier