Interesting collision of real-world and Internet security:
The ceremony sees several trusted internet engineers (a minimum of three and up to seven) from across the world descend on one of two secure locations—one in El Segundo, California, just south of Los Angeles, and the other in Culpeper, Virginia—both in America, every three months.
Once in place, they run through a lengthy series of steps and checks to cryptographically sign the digital key pairs used to secure the internet’s root zone. (Here’s Cloudflare‘s in-depth explanation, and IANA’s PDF step-by-step guide.)
Only specific named people are allowed to take part in the ceremony, and they have to pass through several layers of security—including doors that can only be opened through fingerprint and retinal scans—before getting in the room where the ceremony takes place.
Staff open up two safes, each roughly one-metre across. One contains a hardware security module that contains the private portion of the KSK. The module is activated, allowing the KSK private key to sign keys, using smart cards assigned to the ceremony participants. These credentials are stored in deposit boxes and tamper-proof bags in the second safe. Each step is checked by everyone else, and the event is livestreamed. Once the ceremony is complete—which takes a few hours—all the pieces are separated, sealed, and put back in the safes inside the secure facility, and everyone leaves.
But during what was apparently a check on the system on Tuesday night—the day before the ceremony planned for 1300 PST (2100 UTC) Wednesday—IANA staff discovered that they couldn’t open one of the two safes. One of the locking mechanisms wouldn’t retract and so the safe stayed stubbornly shut.
As soon as they discovered the problem, everyone involved, including those who had flown in for the occasion, were told that the ceremony was being postponed. Thanks to the complexity of the problem—a jammed safe with critical and sensitive equipment inside—they were told it wasn’t going to be possible to hold the ceremony on the back-up date of Thursday, either.