DNSSEC Keysigning Ceremony Postponed Because of Locked Safe

Interesting collision of real-world and Internet security:

The ceremony sees several trusted internet engineers (a minimum of three and up to seven) from across the world descend on one of two secure locations—one in El Segundo, California, just south of Los Angeles, and the other in Culpeper, Virginia—both in America, every three months.

Once in place, they run through a lengthy series of steps and checks to cryptographically sign the digital key pairs used to secure the internet’s root zone. (Here’s Cloudflare‘s in-depth explanation, and IANA’s PDF step-by-step guide.)

[…]

Only specific named people are allowed to take part in the ceremony, and they have to pass through several layers of security—including doors that can only be opened through fingerprint and retinal scans—before getting in the room where the ceremony takes place.

Staff open up two safes, each roughly one-metre across. One contains a hardware security module that contains the private portion of the KSK. The module is activated, allowing the KSK private key to sign keys, using smart cards assigned to the ceremony participants. These credentials are stored in deposit boxes and tamper-proof bags in the second safe. Each step is checked by everyone else, and the event is livestreamed. Once the ceremony is complete—which takes a few hours—all the pieces are separated, sealed, and put back in the safes inside the secure facility, and everyone leaves.

But during what was apparently a check on the system on Tuesday night—the day before the ceremony planned for 1300 PST (2100 UTC) Wednesday—IANA staff discovered that they couldn’t open one of the two safes. One of the locking mechanisms wouldn’t retract and so the safe stayed stubbornly shut.

As soon as they discovered the problem, everyone involved, including those who had flown in for the occasion, were told that the ceremony was being postponed. Thanks to the complexity of the problem—a jammed safe with critical and sensitive equipment inside—they were told it wasn’t going to be possible to hold the ceremony on the back-up date of Thursday, either.

Tags: , , ,

Posted on February 14, 2020 at 6:07 AM • 25 Comments

Comments

Jake • February 14, 2020 6:28 AM

All the security to ensure only the trusted people get into the room – but staff have access to the safes for a check? Isn’t that a weakness?

Boruch • February 14, 2020 7:36 AM

The November 2019 video stream is available on youtube. There, you can
see that safe #1 was giving them a lot of trouble, and only after many
attempts was successfully opened. It seems that everyone neglected to
flag this for required maintenance. Maybe this is a consequence of
feeling bound to the ‘ceremony’ script they use, which cognitively
limits them to a form of security theatre. Without the script, a bunch
of trained engineers could be expected to at least have had a
discussion in situ of whether the safe should be locked before a
maintenance person checked the device.

Sed Contra • February 14, 2020 7:42 AM

I thought there was an internet cafe in Athens in the side streets off Syntagma square where you could login to the hardware module from. You use the third computer on the right.

Jim B. • February 14, 2020 12:05 PM

Interesting read through. Aside from from my own paranoia of having devices plugged into a running system in a highly secure environment, I found the procedures quite credible. I assume all the dates/times were changed,yes?

Must have been tired typing that whole thing up though. Last sentence on page 38 needs some love.

Kim Davies • February 14, 2020 12:06 PM

To clarify some of the questions above:

  • @Jake: There were trusted community representatives present, not just staff, for the maintenance work being performed the day before. It wasn’t merely a ‘check’ as the article suggests.
  • @Boruch: This particular lock was flagged for maintenance, and in fact, this was intended to be that very maintenance. The safe was being opened to replace the lock. That said, the video from November is of a different lock. It should be noted these locks, in general, are very particular in how you open them in the normal case and it can take a few attempts to open them successfully.

David Leppik • February 14, 2020 2:21 PM

Better safe than sorry, I suppose. The alternative would to be more likely to fail in the other direction.

David Leppik • February 14, 2020 2:58 PM

So there are two copies of the keys, and they are on opposite coasts of the US. Requires 3-4 ICANN staff members (out of 14 possible, representing 7 regions) and 3 well-known volunteers. I wonder if it’s wise to rely on a single country for an international venture like this. What if an administration decided to institute a broad, ill-advised travel ban that made it hard for members of the Internet security community to enter the US?

Similarly, if they wanted to diversify into Canada, what sort of security would be required to transport the keys? I wonder if the risk of moving it is why both are in the US.

I’m reminded of the nuclear waste stored here in Minnesota, in big casks along the Mississippi River, upstream from nearly all the major Mississippi cities. One of the biggest sticking points in moving it to permanent storage is the issue of moving it. It would have to go by rail (or truck) and pass through many cities and towns that wouldn’t want nuclear waste even for a second. Also, it would be a very attractive target for terrorists.

As for transporting security keys, they could be transported safely in someone’s pocket. Secrecy is the best security for small valuables. However, that runs counter to how the current ceremony is live streamed.

Clive Robinson • February 14, 2020 4:20 PM

@ David Leppik,

<

blockquote>As for transporting security keys, they could be transported safely in someone’s pocket.

They could use a multidimensional circle… AKA “secret sharing scheme”, to make things a little more secure.

But the real insecurity as you note is,

I wonder if it’s wise to rely on a single country for an international venture like this.

For various reasons I suspect if ICANN tried doing the sensible thing and tried moving one center to another continent or creating a third etc they would be stopped.

Those “little pieces of paper” have enormous political symbolism if not real power (for now). I can not see any US politician behaving sensibly in this respect.

That said there have already been comments to the effect such things should actually be under the control of a UN agency, in much the way Radio Spectrum and other “shared” resources are.

The fact the US Government and Google have been digging their feet in hard is one of the reasons that various countries have enacted laws to take control away from them.

Whilst all we hear about these days is nonsense over Huawei drumed up by saber rattling idiots in US politics[1], other governments know where the real risk is and it’s “The all roads lead to Rome” structure of the Internet which vests way too much power in a single set of what much of the world regards as very unfriendly hands.

Thus some developments now are actively designed to “chop the spider” out of the center of the web. Most of it is going on in the background some companies that are paying for it because they have outsourced so much don’t even know it’s happening to them… But occasionaly it is sufficiently major that it gets to be heard more publically (Russia for instance).

Thus ICANN’s relevance is becoming less and less as we move forward in time…

[1] Heck even the US Dept of Defence faced with the realities of life has come out and said they use Huawei equipment and exprct to continue to do so. Their point like that of the UK Gov is all equipment is suspect, just make the mitigations and move on…

SpaceLifeForm • February 15, 2020 4:18 PM

Suspect someone tried to break in.

Or intentionally messed it up.

Otherwise, pure bs story.

Required II (The Sequel) • February 15, 2020 4:25 PM

As I read the article, I am sorry to admit I had visuals of
the Opening Title Sequence of Get_Smart (1960’s-era TV show)
running through my head … as the various people went through
the various levels of security to get to the Safe.

Maybe there should have been an Opening Title Sequence to the
show where ” Maxwell Smart, Secret Agent 86 ” gets a Telephone
Operator who says ” Wrong Number ” and Max is not allowed to
drop through the Pay Telephone Booth Floor to enter CONTROL
HQ …

David • February 15, 2020 11:52 PM

I was reading this imagining this as heist-caper film like Oceans 14.
seriously, there is an interesting bait and switch with live CCTV used in one of those films, where they swap the feed for a mock up room with a safe vault. fairly basic optical illusion – who checks the veracity of the live stream?

The repeated references to ‘the ceremony’ were a bit pretentious but it’s also quite an amusing concept. I appreciate Clives pointed reference to the increasing irrelevancy of the US in all of this ‘Great Game’

la abeja • February 16, 2020 1:39 PM

@David

The repeated references to ‘the ceremony’ were a bit pretentious but it’s also quite an amusing concept.

Quite the ceremony, yes. Military full dress.

I assure you it is neither pretentious nor intended to be amusing.

Different portions of the private key, protected by a “secret sharing” scheme, are brought together in a carefully choreographed sequence for the signing operation, and then separated and cleared for return to secure storage.

Backups and emergency plans are in place to scram the whole operation at the slightest misstep without endangering the secret keys, despite the enormous cost incurred by such an eventuality.

SpaceLifeForm • February 17, 2020 6:48 PM

@ Clive

There is someone here spewing pure BS.

I guarantee you the safe has a 4 number combination. With a key also.

And when you only have to open once a year, the safe will not magically fail.

In fact, it will not magically fail after opening for over 2000 consecutive days.

This is pure fake news.

I say this based upon experience.

SpaceLifeForm • February 17, 2020 7:07 PM

‘Staff open up two safes, each roughly one-metre across. One contains a hardware security module that contains the private portion of the KSK. The module is activated, allowing the KSK private key to sign keys, using smart cards assigned to the ceremony participants.’

How smart are those smart cards?

Did Agent 99 issue them?

Clive Robinson • February 17, 2020 8:26 PM

@ SpaceLifeForm,

In fact, it will not magically fail after opening for over 2000 consecutive days.

I rather think that depends on,

    Who is doing the magic…

One common cause for safes to go wrong is people have “played with the mechanism” in some way.

It would be interesting to know if the safe had a “Number of openings” counter in the door…

Clive Robinson • February 18, 2020 4:34 AM

@ SpaceLifeForm, ALL

Mind you it’s not just recalcitrant safes that critically effect Internet Security…

Over in another corner critical Internet security is held up because Internet organisation lawyers are fighting it out over a contract that as the old saying has it “Only in America”,

https://www.theregister.co.uk/2019/10/28/arin_rpki_open_source/

Some people realy are banging them down on the table with all the force that testosterone fueled egos will alow…

SpaceLifeForm • February 18, 2020 3:29 PM

It would be interesting to know if the safe had a “Number of openings” counter in the door…

That would be useful. Not just opens.

Attempts.

Maybe a cam to watch. Nah, nevermind.

Maybe someone could check EVERY DAY ?

And note that the attempt counter jumped?

SpinLocks.

Leave a comment

Login

Allowed HTML <a href="URL"> • <em> <cite> <i> • <strong> <b> • <sub> <sup> • <ul> <ol> <li> • <blockquote> <pre> Markdown Extra syntax via https://michelf.ca/projects/php-markdown/extra/

Sidebar photo of Bruce Schneier by Joe MacInnis.