Comments

Bill Stewart • December 15, 2015 1:07 AM

Would it make sense for the root DNS servers to only accept TCP queries instead of UDP? Almost all legitimate DNS queries should be going to recursive servers at ISPs or other services, many newer query responses don’t fit into 1500-byte packets anyway (especially if there’s an amplification attack), and that would reduce the viability of spoofed queries.

Dan Cooper • December 15, 2015 1:14 AM

Funny you mention this issue! My team just installed new DNS servers in Switzerland on Dec 2. We are having all kinds of issues all over the EU. Our new servers work in the US market, but are very intermittent in the EU. Deutsche telekom actually told us to tell all our customers to use google for DNS(it does work), but that a bullshit fix! F5 support has validated our GTM configs as good!
We are still trying to work with EU telco’s to figure out what the hell it is.

G • December 15, 2015 2:02 AM

This, like the protonmail ddos that lasted a week, is likely deep state attacks. Can’t say who for certain but rest assured its one of the usual suspects. The protonmail attack was a demonstration of the capabilities of the (likely) nsa. This last flood was likely another faction ramping up with a test to evaluate what was affected and then use that data to predict what is needed to bring a country down in a proxy war (since there is usually only circumstantial evidence available).

We are gearing up for war. The false flags/gladio terror events are a red flag that is reminiscent of Reichstag and the leadup to world war 2.

Did anyone notice that the Craft – the mercenaries formally known as blackwater, were present at san bernardino in their same military gear as they were during the BMB?

Being that after every false flag shooting there is a call for gun control, which -like clockwork – creates a gun purchasing bonanza.

The government knows this, its statistically predictable. So why do you think they want us to acquire more and more guns? Clearly a confrontation is going to be provoked, followed by their predetermined reaction, then a national disaster, such as emp or internet shutdown or dollar devaluation, followed by millions of dumbfounded dipshits. A global force will be unleashed on us to quell the chaos, a global currency will be our new deal, and the IMF will loan us some austerity, lest we become a third world country.

Let’s not fool ourselves, the technology in the upcoming revolution will decimate all dissent.

In the end, we will be begging for their Marxist totalitarian regime to move in and take back control and fulfill their long planned goals.

Hegel would be proud.

Michiel Appelman • December 15, 2015 2:38 AM

Small nitpick, Bruce: BCP38 does not need to be implemented in the DNS system but in the backbone Internet routing platform, between ISP’s and as close to the edges as possible.

ibrahim korucuoğlu • December 15, 2015 2:54 AM

When I had finished reading the report published by Root Server Operators, thought that this may be a practice operation for a powerful cyber-attack system which uses the type of DDoS techniques. “The source addresses were widely distributed” clause is very remarkable. We can understand that the real source is remain unknown.
In the technology era, we should expect that these kind of incidents will happen more often. Taking precautions and increasing the awareness for individuals and establishments is so important. Anyone or any system could be a slave for this type of attack in the future without knowing.

Dave • December 15, 2015 2:54 AM

Agreed, I’ve said several times that it feels like a test, both of capacity and also to see how the roots respond and adapt.

As for why, it’s mostly because there doesn’t seem to be a definite target. The roots themselves aren’t really targets, for several reasons, but instead are often used in amplification attacks.

Why aren’t the roots targets? Mostly because attacking the roots doesn’t really do anything. Many large ISPs and resolvers run their own local root servers (anyone can) so users of those resolvers are immune. Even for resolvers that rely on the root servers, remember that many are anycast (so outages are localized), but also that DNS retries and caches very effectively, and uses single UDP packets, so if even a single UDP response for .dog is received, .dog works on that resolver for the 24 hour cache.

Since the roots are anycast, vastly over-provisioned, and since you need to take down all of the local anycast nodes PLUS all of the global roots at once to achieve anything more than a slowdown, going after the roots just isn’t practical. If even a few roots are still responding some percentage of the the time, users will at most notice some mild slowdowns during DNS resolution, or have to retry a few times, until their resolver learns which roots to query and gets the appropriate records cached.

Still, it’s worrisome, but on the plus side, this motivates resolver operators to start considering caching roots locally, which is an effective step.

Stéphane Bortzmeyer • December 15, 2015 4:09 AM

Yes, this attack is public from the beginning: https://twitter.com/bortzmeyer/status/671287839292850176 https://lists.dns-oarc.net/pipermail/dns-operations/2015-November/013892.html

But most medias did not do a proper job: they published nothing before there was an official statement (the text by the root name servers operators) two weeks after.

You can see the effects of the attack on DNSmon : https://atlas.ripe.net/dnsmon/?dnsmon.session.color_range_pls=0-30-30-80-100&dnsmon.session.exclude-errors=true&dnsmon.type=zone-servers&dnsmon.zone=root&dnsmon.startTime=1448853000&dnsmon.endTime=1448963400&dnsmon.ipVersion=both

By its effects, it is the largest attack against the DNS root since 2002. Also, it is the first time that even anycasted name servers stopped to answer. So, it is pretty serious.

Stéphane Bortzmeyer • December 15, 2015 4:13 AM

As mentioned by Michiel Appelman, “source address validation” must NOT be implemented “in the DNS system” Once a forged packet hits the DNS server, it is too late. BCP 38 (source address validation) MUST be done at the beginning of the packet’s journey, inside the first operator network.

Also, nothing says the IP source addresses in this attack were forged (the official statement just says “The source addresses of these particular queries appear to be randomized and distributed throughout the IPv4 address space”). It could be a regular botnet, or it could use open DNS resolvers.

Stéphane Bortzmeyer • December 15, 2015 5:05 AM

Bill Stewart: IMHO, TCP would be a good idea. But the root name servers cannot limit themselves to TCP today: they HAVE TO serve everyone and many resolvers cannot send TCP requests, for instance because a broken firewall prevent them to.

Stéphane Bortzmeyer • December 15, 2015 5:09 AM

matt: the TechWeek report is content-free, many unsubstiantated claims and zero facts. There is no indication that the Janet attack and the root attack (one full week earlier) are linked.

Bob • December 15, 2015 9:08 AM

Even though they are saying the packets were not forged, couldn’t they be? I mean we all know the NSA / GCHQ / etc monitor the backbones with passive taps or whatnot, but can’t it also be assumed they have the ability to inject or spoof any type of packet they want to?

If you have access to the backbone to inject or modify packets, what method is there to be absolutely sure they weren’t forged?

Dirk Praet • December 15, 2015 10:05 AM

@ Stéphane Bortzmeyer, @ Michiel Appelman, @ Bruce

Don’t know if any of you are aware of the long-running Spoofer Project at MIT. They map out which address spaces, prefixes and autonomous systems are vulnerable to spoofing, and make a small tool available to test the network/ISP you’re on.

I’d encourage all esteemed readers of this blog to download a copy, run it and inform your ISP if the tests are positive. It really is trivial for them to apply RFC2827 egress filters, and in not doing so they provide malicious actors with an entry point to send spoofed traffic to the entire internet, thus rendering uRFP (ingress filtering; RFC3704) useless. For those wondering, the tool detects NAT.

@ Bob

If you have access to the backbone to inject or modify packets, what method is there to be absolutely sure they weren’t forged?

Good question, but injecting packets at the backbone level is less simple than it sounds. If configured correctly, backbone routers should discard labeled packets from untrusted or unreliable sources (BGP/MPLS IP virtual private Networks; RFC2547/RFC4364). If you look it up, there’s plenty of excellent Cisco and Juniper manuals/guidelines out there on how to implement L2/L3 MPLS VPNs the right way.

Arthur • December 15, 2015 11:00 AM

This, like the protonmail ddos that lasted a week, is likely deep state attacks.

They explain the details and the cost : https://protonmail.com/blog/ddos-protection-guide/

It was 15 Gb/s. (so in upload)

The cost :

Networking equipment: $30’000
BGP/GRE DDoS Mitigation (per year): $50’000 – $100’000
Dedicated IP Transit (per year): $20’000
Maintenance Overhead: $10’000+

Bob • December 15, 2015 11:10 AM

@Dirk

What if instead of injecting new packets, it just modifies (or modifies a clone of) existing ones? I mean if they have the hardware to passively tap, sort, filter, and analyze the volume of traffic on the backbone (which we know they do via Room 641A), I would think they are in the position to inject or modify without being spotted. What % of backbone traffic could you modify / or clone+modify without being noticed?

I am not a network engineer, just an avid watcher of Person of Interest – and recall an episode where they showed the machine covering its tracks or “helping” the police stop a terrorist threat. The machine modified police report submissions or caused a license plate reader “glitch” that “accidentally” tagged a van carrying a bomb with an expired registration so a generic patrol car pulled them over.

reading the RFC’s now though!

Dirk Praet • December 15, 2015 1:13 PM

@ Bob

I would think they are in the position to inject or modify without being spotted.

The short answer is that they don’t have to. All it takes is finding some providers – preferably sitting on big pipes – that don’t do BCP 38-compliant egress filtering and you’re in business. Kinda like spammers using an open SMTP relay server. So yes, I am with @Earl Killian and others that such parties should be publicly named and shamed.

Nick P • December 15, 2015 1:30 PM

@ Arthur

Thanks for posting. The technical details and cost breakdown are useful information. I expected it to be high given it’s basically dedicated lines, high bandwidth, line-rate processing, and a 3rd party’s profit margin. 😉

WishfulWhisperer • December 15, 2015 3:23 PM

What about implementing dnscurve or dnscrypt as 2nd instance for all root DNS servers?

Grauhut • December 15, 2015 4:43 PM

BCP 38 only works with end user lines, on a transit router its a little bit difficult to decide per ip address wich traffic is legit and wich isnt.

Grauhut • December 15, 2015 5:02 PM

@Bruce: “I can’t precisely explain why, but this feels like someone testing an attack capability.”

Smells like a big spam botnet herder under pressure, black.biz. The big.gov players wouldn’t do a show of force like that.

Surprised • December 15, 2015 5:38 PM

This caught my attention. Kinda scary to think of state-sponsored attacks against DNS root servers these days.

Torrc • December 15, 2015 11:10 PM

You can use “Tor routing” to resolve DNS.

  1. Add this line to torrc.
    MapAddress yourdomain.com xxxxxxxxxxxxx.onion

  2. Look yourdomain.com. Tor will resolve yourdomain as x.onion. No DNS server involved.

  3. You can access to yourdomain.com.

Stéphane Bortzmeyer • December 16, 2015 4:09 AM

Bob: there is zero doubt that the NSA or similar organisations can inject packets (see for example the QUANTUM attacks on the Snowden documents). But you don’t need to be the NSA to do so! Anyone can send packets and, on most networks (those without BCP 38), you can spoof the source IP address.

Stéphane Bortzmeyer • December 16, 2015 4:12 AM

Earl Killian: encryption in the DNS is being worked on at the IETF, in the DPRIVE working group (DPRIVE as “DNS Privacy Enhancement” but also to deprive the attacker of data) See http://tools.ietf.org/wg/dprive/ There is already running code of DNS over TLS.

But it won’t help a lot against dDoS attacks: the root name servers have to serve everyone, including old and non upgraded resolvers.

Stéphane Bortzmeyer • December 16, 2015 4:21 AM

Earl Killian: “Reputable companies should refuse to peer with ones that don’t do BCP 38” Then the traffic will move to the transit links, which will make it even more difficult to analyse (when you see a dDoS coming from just one peer, the source is obvious).

André Lucas • December 17, 2015 4:46 AM

Allowing only TCP is not an option. The roots have to be the most interoperable servers of all.

The kind of encryption I think people are talking about – dnscurve was mentioned – is used to detect spoofed replies, but that’s not the problem the roots need to solve. Even discounting the interoperability concerns, all client-server encryption does for server is increase packet sizes and CPU load. Since any client can generate a perfectly legitimate-seeming request, encrypting that request is of little value in protecting the server.

BCP 38 will help, but it won’t solve it. One can still generate floods from zombie machines, and rate-limiting on the target servers is difficult to apply without breaking legitimate queries from busy servers.

DNSSEC (which doesn’t suffer from those problems) is good for many things, but it increases somewhat spectacularly the potential for amplification attacks, by making some responses gigantic in comparison to the query that generated them.

It’s a shame. DNS has proven astonishingly versatile for a system designed for a different kind of Internet, but securing it in its current form is going to be extremely difficult.

It really doesn’t help that making substantial DNS changes takes a very long time. The EDNS0 extension, for example, is 16 years old and is almost universally accepted as a good thing (as well as being mandatory for DNSSEC); yet it still can’t be completely relied upon. Servers have to have mechanisms to disable it when other servers misbehave, or are behind misbehaving firewalls.

Unfortunately, there seems to be a prevailing attitude that DNS is either a solved problem, a problem not worth solving, or (most frequently) someone else’s problem. That’s a pity, because the state of the art for distributed databases has moved on somewhat since 1987. We could do a lot better if we really wanted to.

Peter • December 17, 2015 9:32 AM

The DNS attack started in November and it is a binary attack that uses the DNS Port 53 and IANA reserved ports. The first part is command/executable code installed on DNS servers. The second part is what is currently causing consternation and that is a memory-resident attack most current technologies cannot detect and it morphs about three to four times a day. In the memory attack there are instructions included that cause the DNS servers to send a quick response to one of multiple Chinese web sites that have only recently been registered. Once that happens, the web site sends the memory resident attack data to the compromised DNS server and that DNS server starts sending out malware to other DNS servers.

From what we have determined, all 13 primary DNS servers have been compromised as well as most of the subordinate DNS systems.

So if you can … imagine what will happen when whoever is behind these attacks suddenly decides to tell the DNS systems to turn off and they all do so simultaneously.

WishfulWhisperer • December 17, 2015 9:23 PM

@Stéphane Bortzmeyer

Quite a lot is going on with DNS over TLS indeed.

I hope in the end the best reliable solution prevails and we do not have to choose between a multitude of DNS protection mechanisms and various [client,server]-side proxies for DNS.
Dnscrypt seems to be a niche solution, wonder why.

Vast amount of information in the drafts. Thanks for providing the link.

DDOS.sucks • December 18, 2015 8:09 AM

Say Peter, could you please provide us with the name of the dll they are using to put in the alternate data stream.

And Stephane, … are you again disguised as Santa … tssssss.

Patrick Star • December 19, 2015 4:07 PM

Because of widespread asymmetric routing, which is perfectly normal and unavoidable, it simply isn’t possible to implement reverse path filtering (RPF) on anything but simple access connections; think “broadband” and “home users”.

Even when the customer doesn’t use dynamic routing there are issues with enforcing it for things like server hosting/co-lo and above-consumer-grade access connections; one scenario being load-balancers/front-ends in different locations than the back-ends pushing the data (ironically enough useful for DDoS defense amongst other things).

Not to mention anycast – another DDoS defense technique that’s atleast made more problematic by strict RPF.

weldon • December 20, 2015 4:21 AM

@G
You’re clearly off your meds.

(It is however immensely entertaining, in an eye-rolly, face-palming sort of way. Is there more where that came from?)

Phantom • January 6, 2016 8:27 AM

DNS, srsly? How to fix the internet, step 1: phase out obsolete central points of failure …

Time for a new rant from the back row.

DNS root servers? A 21st century internet shouldn’t need them.

DNS should at long last be allowed to go into well-deserved retirement. Along with trusted CA’s, BGP and all legacy internet protocols that rely on central authority, it should be allowed to die. It’s served its purpose. Now we know better. We don’t need IP based filtering. We need to do away with the legacy “Internet 1.0” network stack entirely. There are better ideas now. (https://github.com/cjdelisle/cjdns/blob/master/doc/Whitepaper.md – although in my understanding the design is still susceptible to DDoS) Anyway, implementations of more robust-to-central-attacks protocols are ready for testing, the big issue being how to get businesses on board.

As far as I am concerned DNS is a mechanism that promotes central hegemony, is not really needed anymore except for sentimental reasons (and because people invested in it …), and ought to be phased out for the good of everyone ASAP.

Source address validation is not the solution. DNSSEC is not the solution. DNSCurve is not the solution. When public-key based addressing schemes like onion addresses aren’t enough because one does need name records, completely decentral schemes for squaring the Zooko triangle such as GNS are a solution. Maybe the ICANN will have to subordinate their root servers under a future version of GNS one day. I’ve no objections to people keeping their hard-won domain names, on the contrary! They’ll still be the biggest game in town. Only instead of worrying about reserving special use pseudo-TLDs (like .onion recently) inside their obsolete parochial system, the other way round it actually makes sense.

Try explaining that to thick executives or public decision-makers, let alone end users, though. Right now, IANA/ICANN-less alternative concepts are niche applications for nerds. Obviously conceptually superior in all (!) respects to anything based on officialdom, but for the very same reason lacking the humdrum commonplace quality of thoughtless reliance on central authorities, hence hard to sell. Anything that doesn’t come with red tape attached and whose security depends on crypto and PKI instead of trust in central authorities? Suggest it an they paint you a crackpot, a cyber-criminal. People won’t go near it yet. Isn’t societal inertia the only real problem in fixing the internet? Well, isn’t it the only real obstacle to fixing things in general …

To conclude, IPv4, DNS and all that obsolete, outgrown cruft will probably live as long as the civilization that made them. For no reason but tradition, inertia and the unwillingness to try something new.

Cheers

Someone who is maybe professionally interested in the details but will probably experience no special inconvenience from any such attacks …

Leave a comment

Login

Allowed HTML <a href="URL"> • <em> <cite> <i> • <strong> <b> • <sub> <sup> • <ul> <ol> <li> • <blockquote> <pre> Markdown Extra syntax via https://michelf.ca/projects/php-markdown/extra/

Sidebar photo of Bruce Schneier by Joe MacInnis.