DNSSEC Keysigning Ceremony Postponed Because of Locked Safe

Interesting collision of real-world and Internet security:

The ceremony sees several trusted internet engineers (a minimum of three and up to seven) from across the world descend on one of two secure locations -- one in El Segundo, California, just south of Los Angeles, and the other in Culpeper, Virginia -- both in America, every three months.

Once in place, they run through a lengthy series of steps and checks to cryptographically sign the digital key pairs used to secure the internet's root zone. (Here's Cloudflare's in-depth explanation, and IANA's PDF step-by-step guide.)


Only specific named people are allowed to take part in the ceremony, and they have to pass through several layers of security -- including doors that can only be opened through fingerprint and retinal scans -- before getting in the room where the ceremony takes place.

Staff open up two safes, each roughly one-metre across. One contains a hardware security module that contains the private portion of the KSK. The module is activated, allowing the KSK private key to sign keys, using smart cards assigned to the ceremony participants. These credentials are stored in deposit boxes and tamper-proof bags in the second safe. Each step is checked by everyone else, and the event is livestreamed. Once the ceremony is complete -- which takes a few hours -- all the pieces are separated, sealed, and put back in the safes inside the secure facility, and everyone leaves.

But during what was apparently a check on the system on Tuesday night -- the day before the ceremony planned for 1300 PST (2100 UTC) Wednesday -- IANA staff discovered that they couldn't open one of the two safes. One of the locking mechanisms wouldn't retract and so the safe stayed stubbornly shut.

As soon as they discovered the problem, everyone involved, including those who had flown in for the occasion, were told that the ceremony was being postponed. Thanks to the complexity of the problem -- a jammed safe with critical and sensitive equipment inside -- they were told it wasn't going to be possible to hold the ceremony on the back-up date of Thursday, either.

Posted on February 14, 2020 at 6:07 AM • 20 Comments


JakeFebruary 14, 2020 6:28 AM

All the security to ensure only the trusted people get into the room - but staff have access to the safes for a check? Isn't that a weakness?

BoruchFebruary 14, 2020 7:36 AM

The November 2019 video stream is available on youtube. There, you can
see that safe #1 was giving them a lot of trouble, and only after many
attempts was successfully opened. It seems that everyone neglected to
flag this for required maintenance. Maybe this is a consequence of
feeling bound to the 'ceremony' script they use, which cognitively
limits them to a form of security theatre. Without the script, a bunch
of trained engineers could be expected to at least have had a
discussion in situ of whether the safe should be locked before a
maintenance person checked the device.

Sed Contra February 14, 2020 7:42 AM

I thought there was an internet cafe in Athens in the side streets off Syntagma square where you could login to the hardware module from. You use the third computer on the right.

Jim B.February 14, 2020 12:05 PM

Interesting read through. Aside from from my own paranoia of having devices plugged into a running system in a highly secure environment, I found the procedures quite credible. I assume all the dates/times were changed,yes?

Must have been tired typing that whole thing up though. Last sentence on page 38 needs some love.

Kim DaviesFebruary 14, 2020 12:06 PM

To clarify some of the questions above:

  • @Jake: There were trusted community representatives present, not just staff, for the maintenance work being performed the day before. It wasn't merely a 'check' as the article suggests.
  • @Boruch: This particular lock was flagged for maintenance, and in fact, this was intended to be that very maintenance. The safe was being opened to replace the lock. That said, the video from November is of a different lock. It should be noted these locks, in general, are very particular in how you open them in the normal case and it can take a few attempts to open them successfully.

David LeppikFebruary 14, 2020 2:21 PM

Better safe than sorry, I suppose. The alternative would to be more likely to fail in the other direction.

David LeppikFebruary 14, 2020 2:58 PM

So there are two copies of the keys, and they are on opposite coasts of the US. Requires 3-4 ICANN staff members (out of 14 possible, representing 7 regions) and 3 well-known volunteers. I wonder if it's wise to rely on a single country for an international venture like this. What if an administration decided to institute a broad, ill-advised travel ban that made it hard for members of the Internet security community to enter the US?

Similarly, if they wanted to diversify into Canada, what sort of security would be required to transport the keys? I wonder if the risk of moving it is why both are in the US.

I'm reminded of the nuclear waste stored here in Minnesota, in big casks along the Mississippi River, upstream from nearly all the major Mississippi cities. One of the biggest sticking points in moving it to permanent storage is the issue of moving it. It would have to go by rail (or truck) and pass through many cities and towns that wouldn't want nuclear waste even for a second. Also, it would be a very attractive target for terrorists.

As for transporting security keys, they could be transported safely in someone's pocket. Secrecy is the best security for small valuables. However, that runs counter to how the current ceremony is live streamed.

Clive RobinsonFebruary 14, 2020 4:20 PM

@ David Leppik,

As for transporting security keys, they could be transported safely in someone's pocket.

They could use a multidimensional circle... AKA "secret sharing scheme", to make things a little more secure.

But the real insecurity as you note is,

I wonder if it's wise to rely on a single country for an international venture like this.

For various reasons I suspect if ICANN tried doing the sensible thing and tried moving one center to another continent or creating a third etc they would be stopped.

Those "little pieces of paper" have enormous political symbolism if not real power (for now). I can not see any US politician behaving sensibly in this respect.

That said there have already been comments to the effect such things should actually be under the control of a UN agency, in much the way Radio Spectrum and other "shared" resources are.

The fact the US Government and Google have been digging their feet in hard is one of the reasons that various countries have enacted laws to take control away from them.

Whilst all we hear about these days is nonsense over Huawei drumed up by saber rattling idiots in US politics[1], other governments know where the real risk is and it's "The all roads lead to Rome" structure of the Internet which vests way too much power in a single set of what much of the world regards as very unfriendly hands.

Thus some developments now are actively designed to "chop the spider" out of the center of the web. Most of it is going on in the background some companies that are paying for it because they have outsourced so much don't even know it's happening to them... But occasionaly it is sufficiently major that it gets to be heard more publically (Russia for instance).

Thus ICANN's relevance is becoming less and less as we move forward in time...

[1] Heck even the US Dept of Defence faced with the realities of life has come out and said they use Huawei equipment and exprct to continue to do so. Their point like that of the UK Gov is all equipment is suspect, just make the mitigations and move on...

SpaceLifeFormFebruary 15, 2020 4:18 PM

Suspect someone tried to break in.

Or intentionally messed it up.

Otherwise, pure bs story.

Required II (The Sequel)February 15, 2020 4:25 PM

As I read the article, I am sorry to admit I had visuals of
the Opening Title Sequence of _Get_Smart_ (1960's-era TV show)
running through my head ... as the various people went through
the various levels of security to get to the Safe.

Maybe there should have been an Opening Title Sequence to the
show where " Maxwell Smart, Secret Agent 86 " gets a Telephone
Operator who says " Wrong Number " and Max is not allowed to
drop through the Pay Telephone Booth Floor to enter CONTROL
HQ ...

DavidFebruary 15, 2020 11:52 PM

I was reading this imagining this as heist-caper film like Oceans 14.
seriously, there is an interesting bait and switch with live CCTV used in one of those films, where they swap the feed for a mock up room with a safe vault. fairly basic optical illusion - who checks the veracity of the live stream?

The repeated references to 'the ceremony' were a bit pretentious but it's also quite an amusing concept. I appreciate Clives pointed reference to the increasing irrelevancy of the US in all of this 'Great Game'

la abejaFebruary 16, 2020 1:39 PM


The repeated references to 'the ceremony' were a bit pretentious but it's also quite an amusing concept.

Quite the ceremony, yes. Military full dress.

I assure you it is neither pretentious nor intended to be amusing.

Different portions of the private key, protected by a "secret sharing" scheme, are brought together in a carefully choreographed sequence for the signing operation, and then separated and cleared for return to secure storage.

Backups and emergency plans are in place to scram the whole operation at the slightest misstep without endangering the secret keys, despite the enormous cost incurred by such an eventuality.

Leave a comment

Allowed HTML: <a href="URL"> • <em> <cite> <i> • <strong> <b> • <sub> <sup> • <ul> <ol> <li> • <blockquote> <pre>

Sidebar photo of Bruce Schneier by Joe MacInnis.