Newly declassified: “A History of U.S. Communications Security (Volumes I and II),” the David G. Boak Lectures, National Security Agency (NSA), 1973. (The document was initially declassified in 2008. We just got a whole bunch of additional material declassified. Both versions are in the document, so you can compare and see what was kept secret seven years ago.)
B • November 25, 2015 10:29 AM
Love this gem:
“We had, in any event, become increasingly skeptical of jettisoning as a viable way to prevent the recovery of equipment as various submersibles attained greater and greater depths. We also found to our astonishment that some of the electronic crypto-equipments built in the fifties (and sixties) float.”
Also interesting discussion about their skepticism of public key cryptography; seems that they were thinking about the need for forward secrecy early on.
Anon • November 25, 2015 1:36 PM
An interesting redaction –
KAG-1/TSEC used to be the bible of US cryptographers, was held in every crypto-center, and covered everything from message preparation to compromise reporting in considerable detail. While we viewed it as a model of clarity, this perception was not always shared in the real world. A frustrated Navy Chief stormed out of his crypto-center on board a carrier at sea, handed KAG-1 to a sailor and jokingly said “Throw this dam’ thing overboard.” He did. Several ships thereafter steamed back and forth for several days, but never found it. Winds, tides, and currents were studied to predict where it might come ashore with results so ambiguous as to offer little hope and, in fact, it was never recovered – at least by us.
Anon • November 25, 2015 1:44 PM
Further (on redaction above) –
This incident triggered an R1 study on what happens to our documents in salt water. A tank was made, and a copy of KAG-1 immersed. It stayed there for a year or so with no sign of deterioration. Agitators were added to stimulate wave action for another few months, with still no appreciable effect. We never did find out how long such a document would last. Subsequent work, however, has shown that good paper is nearly impervious to salt water, apparently indefinitely. A visit to S2’s exhibit of materials recovered from the sea bottom will bear that out. There you can see perfectly legible codes that had been under water since World War II, together with extraordinarily well-preserved items of hardware and magnetic tape that had been on the bottom for many years. These facts add to the previously expressed skepticism about jettison as a way to get rid of our stuff unless at very great depths and in completely secret location. (Shortly after WWII, small Army training crypto-devices called the SIGFOY were disposed of beyond the 100 fathom curve off Norfolk. Some years later, they became prize souvenirs for beach combers as they began washing ashore.)
Anon • November 25, 2015 2:08 PM
Not sure why this was previously redacted –
It seems to me that NSA does not yet have much expertise in computer security. Rather, we are expert in computer insecurity. We do much better in finding security vulnerabilities in any computer complex than in proposing security architectures fro them. Somehow, the attack seems more challenging (fun) than the defense, and this seems true in the general business of cryptosystem design as well. A spin-off of this syndrome manifests itself when a security modification is needed for an existing crypto-equipment. In my experience, most design engineers would /much/ rather attack a brand new problem – meet a new and difficult requirement – starting from scratch, pushing the electronic state of the art, exercising opportunities for innovation, and so on than go through the drudgery of a mere “fix” accepting all the constraints of configuration and technology in some pre-existing piece of hardware.
Corodon • November 25, 2015 2:50 PM
In the first release, lectures 2-6 are completely redacted. The new version is much better, but there are still things like a big chunk redacted in the middle of the discussion of one-time pads… not really a topic I would expect there to be particularly hush-hush.
tyr • November 25, 2015 5:53 PM
@Anon
I didn’t think that stuff was paper. It seemed to
be designed to be impervious to water. Not every
place that should have had the manual had one nor
did every modification get made on time. Having
to clean the privy of incompetence gives you a
new perspective on the whole enterprise.
I’m sure its no longer that way since the younger
generation is a cut above the olde fartes. (sarcasm).
cf • November 26, 2015 3:39 AM
@Corodon
a big chunk redacted in the middle of the discussion of one-time pads… not really a topic I would expect there to be particularly hush-hush.
Section 6 of the National Security Act of 1959 basically says they don’t have to reveal anything about the NSA. I assume that was stamped in the 2008 release.
Exec Order 13526 was signed by Obama in 2009, and is mostly about declassifying info. It says everything should be declassified after 25, except a few very important things. It could be the info was still deemed relevant (or this was a case of over-classification) but one possibility that seems likely is that it excludes information we receive from other governments. So if in their discussion of our one time pad designs, they had a page on how the Brits are doing theirs, then that can remain classified for up to 75 years, regardless of relevance.
Clive Robinson • November 26, 2015 6:25 AM
@ Corodon,
… but there are still things like a big chunk redacted in the middle of the discussion of one-time pads… not really a topic I would expect there to be particularly hush-hush.
It depends on your viewpoint.
As @cf has pointed out there are fairly good reasons. I’ll try to give you a little background of why it might be.
You need to go and lookup the project VERNONA which involved the UK&US breaking One Time Pad “spy traffic”. The official reason is that the Russian’s due to “financial reasons” re-used pads from one area in another area some period later.
This “break” happened back in the time when computing was still very expensive and thus reserved for traffic that was more rewarding in terms of breaks for dollars invested.
However the break happened, thus the questions of Why and How.
The traffic was highly classified and compartmentalized thus to get the traffic from two different areas would be unusual for a jobbing analyst. Further to have to go through years of traffic by hand a Herculean task even for a large team.
Which means that there was some form of reason to do the analysis with a reasonable probability of getting somewhere. This suggests that there were other problems with the Russian pads…
We know that at the time the Russians were texhbology poor manpower rich. It has been said by various sources that the Russian pads were made by typists being told to type randomly… Something that humans are actually very bad at doing, especially when those checking their work are not aware of this failing.
From other sources we know that the NSA were doing quite leading edge mathmatics, especially into random bit generators using electronic and mechanical machines.
One asspect of this is analyzing the likes of stream ciphers of which One Time Pads are a subset.
We are also aware they were into analysing and cataloging intercepted traffic with machines.
One tool from long pre NSA days is the Index of Coincidence and automated machines to measure it.
Thus it is possible that the idustrial largesse of the NSA and the mathmatical skill of the UK had come up with much more refined ways to spot anomalies in stream ciphers and could thus catalog them not just by presumed type but also by anomaly charecteristics. Which could have shown that the Russian pads had commonality just by numbers. Which the very expensive computers would find “bread and butter” work.
Knowledge of the methods and details of such techniques would still be very valuable today. Thus not something you would want the rest of the world to know about. There is president for this view from what we know of the German Enigma machines and the Bletchly Park work to break it which most now call the “Ultra Secret” from the early 70’s book by Winterbottom, and further information in the 80’s from Gordon Welchman and Peter Wright.
For reasons that have been mentioned in this blog from time to time, OTP is becoming attractive again to some. This is based on the assumption it’s proof not just to the likes of FiveEye conventional computers but Quantum Computers that might –or might not– exist now or in the near future. Further it is beguiling simple both in theory and practical use.
However there is a big issue… It’s known that the pads should be truly random, BUT they need to be the right type of random, and few know what the constraints are on this. There are rules of thumb like limiting runlengths to a maximum of five charecters in the alphabets in use. But what is not known is what the FiveEyes know about other aspects of generating pads and extracting information from the resulting ciphertext traffic because of weaknesses they have identified.
Arclight • November 26, 2015 11:11 AM
Regarding the decryption of spy traffic encrypted with one time pads:
Much of this traffic would have originated from “numbers stations” in the Eastern Bloc. These were high-powered shortwave transmitters that could be heard for thousands of miles.
I am told that transcribing these was a sort of “second tier” mission after following Soviet military exercises and such.
The linguists in the ASA during thr Cold War were chosen based on a “synthetic language” test and all had exceptional aptitude. Most quickly learned the letters and numbers in many different languages, regardless of their specialty.
All of the traffic in code went to Washington and the U.K. It wouldn’t be difficult to imagine that all presumed OTP messages went into a database and got periodically analyzed against previous messsages. The volume of spy traffic being spoken out in 5 letter groups would have been low enough for all of it to have been digitized and kept on tape. Bank accounts and electric bills were already being processed in similar data volumes at that time.
Whether those also got compared with messages that came from much more secret sources, I have no idea. But there’s no reason that a mainframe couldn’t have been doing the analysis on a regular basis and generating recommendations for follow-up work by analysts.
Arclight
Godel • November 26, 2015 5:28 PM
@ Clive
The story that I read about Venona is that the foul-up happened in the middle of WW2 when the Russian facility that produced the one time pads was in danger of being overrun by the Nazis.
The contents of the special room in the middle of the facility with the “secret sauce” for producing the random numbers were evacuated east, but the rest of the printing facility was left behind. The OTPs weren’t used in a complete form but the older pages were mixed up and interspersed with other material.
Afterwards even if someone in authority among the Russians did discover the problem they would likely have been too afraid to reveal it, because they would have probably been sent to Siberia or shot.
Stalin took “shooting the messenger” rather seriously.
CallMeLateForSupper • November 26, 2015 9:04 PM
This jumped right off the page. (actual) p.85
“The greatest transmission security weakness of all, of course, results from our need to transmit a great deal of information in the clear; so that hostile SIGINT has a ball in the business of examining ‘message externals’ when the whole darned transmission is external.
“What we need, of course, are more and better systems to reduce, and. reduce sharply, the
amount of information we now send in the clear. After that, we need a whole series of new transmission systems which will make our traffic difficult to intercept. […] a great deal of our current traffic is there for the taking so that hostile intercepton, by relatively quick and simple traffic analysis, can discover who’s talking, who’s being addressed. how much traffic is being exchanged and often, because of plain-language transmissions and other collateral, what’s being talked about.”
Yes, we all would benefit from having those things.
Christos • November 27, 2015 1:54 AM
@ Godel: ‘The story that I read about Venona is that the foul-up happened in the middle of WW2’
Problems with the Soviet otp were identified by the Germans as far back as the late 1920’s.
Check http://www.scribd.com/doc/234634849/TICOM-DF-111
Z.Lozinski • November 27, 2015 3:37 AM
Remember in the 1950s, both the UK and US Governments spent a lot of money researching randomness, and ways to test for it.. The forward to the RAND Corporation’s Table of A Million Random Digits (from 1955) explains how they set about doing this.
http://www.rand.org/pubs/monograph_reports/MR1418/index2.html
Of course in the UK, the ERNIE (Electronic Random Number Indicator Equipment) which is a hardware RNG for Premium Bonds (in essence a Government Lottery) was designed by a Mr Tommy Flowers. (He of the Bletchley Park Colossus).
NOT [FATE] A = <CHANCE> NOT A • November 27, 2015 6:42 AM
Is there a maximum theoretical rate at which random bits can be harvested from nature, formula related to Shannon–Hartley theorem maybe involving planck’s or constant other physical constants?
Who? • November 27, 2015 10:04 AM
@ B
“We had, in any event, become increasingly skeptical of jettisoning as a viable way to prevent the recovery of equipment as various submersibles attained greater and greater depths. We also found to our astonishment that some of the electronic crypto-equipments built in the fifties (and sixties) float.“
That is, they have tried jettisoning before…
Clive Robinson • November 27, 2015 11:26 AM
@ Who?,
That is, they have tried jettisoning before…
Oh yes, it goes back to the end of WWII, when there were chemical weapons, conventional munitions and secret equipment that had suddenly become “surplus to requirments”.
Somebody decided the safest thing to do was take it out to places where the sea is deep and fishermen did not trawl and chuck it over the side. There are literaly millions of tonnes of “war waste” sitting on the sea bed…
But as the old saying goes “Time and tides make fools of us all”. Fish are scarce, and technology alows very very deep trawling these days, and past beliefs about a lack of currents in deep seas has proved wrong. Thus every so often stuff comes to the surface…
The reason why a lot of 50’s and 60’s military electronics and electro mechanics float is thermionic valves in hermetically sealed equipment. To deal with the thermal issues of equipment that had to work from -40 to +70C there had to be quite a bit of space inside to give sufficient surface area for the thermal issues to be solved. But importantly the heavy part of much of the equipment –the transformers– in the power supplies were often in seperate cases. These were often not hermetically sealed, the transformers rectifiers and paper oil cspacitors were just covered in several layers of dark brown tropical varnish (that is actually highly toxic even today, something you need to know if you are going to restore old Mil Electronics).
So a large virtualy empty box that’s hermetically sealed does tend to bob along if it’s not bolted into a rack with the PSU or nolonger connected by the power cables. And old decommissioned crypto gear got bolted to wooden pallets inside solid wooden crates made of two by two and half inch thick planks screwed together for “secrecy” reasons. And likewise rather than risk having it go to a commercial company to be crushed burnt and melted down for scrap, some bright spark sugested droping it down onto the old chemical weapons… Just to watch what the thought would be “jetsom” become “flotsam”…
You would think that a glance at the shipping documents that gave the weight and dimensions of the crates might have given somebody the idea to check the density. But no the military never let practical considerations get in the way of a daft idea from up the chain of command… Remember the very definition of insane stupidity is the sort of “Military Intelligence” in field commanders and their subordinates that gave rise to such acts of “mindless bravery” as immortalized in the words “Into the valley of death rode the six hundred”,
https://en.m.wikipedia.org/wiki/The_Charge_of_the_Light_Brigade_(poem)
Who? • November 27, 2015 3:25 PM
@ Clive Robinson
Thanks a lot for sharing this history with the rest of us!
A very nice and detailed outline. I remember playing with the internals of an old tube radio in the 80’s, when I was really young, and finding some sort of translucent brown varnish around its internal transformer. Nice to know it is highly toxic, will be more careful next time.
Again, thank you very much for sharing this informative history.
Clive Robinson • November 27, 2015 5:18 PM
@ Who?,
Nice to know it is highly toxic, will be more careful next time.
If it was not military kit then it is probably just ordinary shellac varnish put on to keep moisture out, and nothing to worry about.
The military found that in high humidity tropical locations –due to having an organic base– things grew on the varnish. So they added a series of poisons into their varnish mixture. The problem was it stayed the same colour, so when surplus kit made it to the likes of London’s Edgware Road if the varnish was damaged it was possible to breath it in or have it on your hands when eating a biscuit or sarnie and not be aware it was highly toxic.
I’m unaware of any civilian getting ill but back when squadies still repaired equipment in the field you used to get it bashed into your “thick skull” by the instructors to not damage the varnish…
The instructors were also not happy about some of the more ingenious technicians drawing sparks of the anode caps of 807 valves with a screwdriver. The valves were used in a push pull configuration to drive another stage in a D10 transmitter and you had to “balance and neutralize” them after replacing a valve that had gone soft. The “official method” was long and slow and not realy physically possible in the half box body on the back a four tonne truck. By drawing a spark and seeing how long and what colour it was you could tweek the balance in about thirty seconds, which on a busy HF circuit in the field was a very usefull trick.
The reasons they instructors did not like it was the HT voltage was well above what was considered lethal, and also the fact that you could get soft X-Rays off of the valves… Which trust me was a lot less injurious to your health than having an unhappy Yeoman of signals with a handfull of unsent Flash encoraging you to get the circuit back up by making less than helpfull anatomical suggestion over the EOW or worse in person if you had the misfortune of being co-loc with the ComCen. And they wondered why I volunteered for the “awkward squad” who like the sniper / observers were usually deployed behind enemy lines. In other countries far far away from Yeoman with Flash, Sgt Majors telling you to get a hair cut and overly keen young officers (known as “Ruperts” or “the chinless squad”) wanting to know why you went deaf when they were around. Often it was a remote place with space and wildlife, where looking scruffy was blending in with the scenery, and your only real worry was a goat or some such wanting to eat the antenna wire, or not getting fresh to suppliment the rat packs, or not being able to brew up (we won’t mention having to bring your one’s and two’s back with you in your personal kit).
tyr • November 27, 2015 6:34 PM
@Clive
A few years ago a local fisherman hauled up a
Japanese Long Lance Torpedo near San Francisco.
There was a nice item from WW2 to find in your
catch.
I seem to recall a megaton WW1 trench mine going
off in 1955, they were emplaced at the end of the
war,the miltary wandered off and can’t remember
where they are anymore.
I also think it was Bill Mauldin who said there
are two kinds of troops, those who look good on
parade and those who look good in a fight.
tyr • November 28, 2015 1:50 AM
Then there’s the senile. Make that Kiloton mine even WW1
troops were too lazy to carry that much dynamite into a
tunnel.
Subscribe to comments on this entry
Sidebar photo of Bruce Schneier by Joe MacInnis.
keiner • November 25, 2015 7:51 AM
..start with page 155 (of .pdf) FUN!