Schneier on Security
A blog covering security and security technology.
« Friday Squid Blogging: Steganography in the Longfin Inshore Squid |
| Seymour Hersh on Cyberwar »
October 25, 2010
Declassified NSA Documents
It's a long list. These items are not online; they're at the National Archives and Records Administration in College Park, MD. You can either ask for copies by mail under FOIA (at a 75 cents per page) or come in in person. There, you can read and scan them for free, or photocopy them for about 20 cents a page.
Posted on October 25, 2010 at 6:21 AM
• 47 Comments
To receive these entries once a month by e-mail, sign up for the Crypto-Gram Newsletter.
They are only declassifying WWI and pre-WWII documents now in 2010, almost 100 years after some of the events took place? Is this a good or a bad development in regards to government secrecy?
In the era of Wikileaks, declassification of 75 year old documents and FOIA requests using snail mail seems rather ...quaint.
Wikileaks is run by the CIA in my opinion, no proof just a gutt feeling.
We can expect a lot of Wikileaks jokes in response to this - it's just too inviting to pass up.
When I have time I'll troll through the list to see if there are any early EmSec documents.
What later became TEMPEST started in WWI when it was found that you could eavesdrop on field telephone systems that used such things as phantom signaling etc.
Later in WWII and just after was when TEMPEST got going in earnest when it was discovered you could (amongst other things) strip of a One Time Tape addition to plain text in an online cipher system by observing the hold and release times of the relay via the signal lines and later PSU lines.
It might be old information but sometimes there are little nuggets to be found that can be revealing as to what people where thinking / investigating at the time.
Seems like they want to discourage people from actually reading these documents by charging 75¢ per page by mail.
Do they really expect us to believe it costs that much money for a human being to be at the copier for each page? They could just as easily make them all available online, even as PDF documents for download.
Government ineptitude in all it's glory.
Well imagine 1 million people requesting 1 page from all different documents.
I have no problem envisioning that costing in excess of 75¢ per document in handling...
You have to go to ladies bathroom at the Cornerstone Grill; knock on the door twice; wait seven seconds; then knock five times. A woman will open the door and say, "I think you have the wrong room." You must then say, "Is this the 'Vous?" She will then lead you to the secret hatch on the sticky floor. Under the hatch is a stairwell leading to a library with copies of the documents.
Scully's right you should listen to her.
Cost would be pro rata and includes personnel time to access the record (walk into Warehouse 13 and take it off the shelf), take it to the copier, copy it, put it in an envelope, address it, put it in the mail and close the FOIA request form. And the services for the copier repair guy And the cost of materials, paper, ink, toner, postage (even franked mail is not free). 75cents is cheap.
@Mike "Wikileaks, declassification ... "
Wikileaks wasn't a declassification event it was a spill. DoD still regards the material as classified to be written to only secure systems and is requiring anyone on Niprnet or contractor corporate networks to sanitize the data where found.
Let's deal with this now so it doesn't come up again, again, and again, and again in the next 90 posts.
People often say well if it's been leaked the harm has been done. To which the standard answer is 'more harm will be done by confirming the information.'
This also gives certain operations room to manuever if as @--Andrew opins above it's an IC disinformation campaign.
"They could just as easily make them all available online, even as PDF documents for download."
Point still stands. I also tend to think that digitized copies are more easily preserved than paper pulp.
@Curby "digitized copies are more easily preserved than paper pulp."
NARA has an interesting technology challange. Do you still have 8" floppy drives, 16track reel to reel tapes, Edison phono player, QIC tape drives, IBM punch card reader...et cetera, et cetera, et cetera. At least paper is pretty constant form, fades a little and takes up space but the originals can still be questioned..
as a matter of fact, I do have two 8" floppy drives (and floppies). And I do have a QIC drive and tapes. Haven't had to use them in years, but I'm ready if that is the media the NSA wants to use.
and ... do you really expect the hundreds of millions of documents in pdf format to be unreadable in 30 or 50 years?
It's funny that these modern digital data storage formats have such a limited lifetime, but 3,000-year-old papyrus paper documents can still be read today.
"It's funny that these modern digital data storage formats have such a limited lifetime, but 3,000-year-old papyrus paper documents can still be read today."
Ephemeral documents help to destroy the effect history has on present-day decisions.
@riko "as a matter of fact, ...documents in pdf format to be unreadable in 30 or 50 years"
But no 5, 7 and 8 hole tape punch-readers? Tsk. We used to be able to make some the teletypes do some cool things on the christmas watch.
Hard to say. What if adobe falls or someone buys the pdf copywrite and denies it to the world? Ten years ago someone bigger then Microsoft was unthinkable. Ten years before that Microsoft being bigger than IBM was unthinkable. What if everyone stopped using Adobe technology in favor of a better file type?
While it's true that all most of us care about is the text itself that's not really what's relevant to being able to question the original.
With the original, say paper, if some ink fading has occured you can recover the text by looking at fiber scoring and indentations. Using flourescents and other analytical methods to recover or verify the artifact.
But that's gone with a photocopy or scanned image. I'm trying to read my grandparent's letters and they were only photocopied from that old blue envelope and paper Air Mail combo from the 40s and the 50s. I'm begging my cousin to have them handled by a profesional conservator.
Other technologies have the same problem. Have you ever put a CD from a previous computer into your current machine and not been able to get anything from it? I've been hearing people recommend reburning on new media every 5-10 years or so. Having the device also assumes having the software to run it. If it even still exists.
And finally there's the mass storage problem of presenting the entire fricking U.S. ARCHVIES online; vasty that is.
Harddrives cost, SANS cost lots, network routers, IT infrastructure, people (well I say people...you know sysadmins, dbas, network engineers them...and their managers, techwriters payroll and ancillary people), servers, software licenses; adds up. Cloud may or may not suffice.
I'm sure NARA would scan and make everything they have avaliable but there's only 2,504 of them and their operating budget is little more than 400 million $US. that's not many and that's not much.
Perhaps one of the Google lurkers would volunteer to help, pro bono, for bragging rights only, no cost to the taxpayer, because it's good and not doing so is evil; if they haven't already.
@EH "Ephemeral documents help to destroy the effect history has on present-day decisions."
Now that's a very interesting statement.
"do you really expect the hundreds of millions of documents in pdf format to be unreadable in 30 or 50 years?"
Maybe. Are 15 year old 5¼" floppies of Wordstar documents readable now? Just barely, I think. I would give very low odds for a 15 year old multi-page tiff on an optical cartridge drive (a common storage format in the mid-1990s).
Testudo--don't forget, you have to be wearing the special 'vous shoes to protect you from the 'vous goo. I though that it was interesting that in the news covering of a recent stabbing that was the result of a fight at the thirsty turtle, the kept showing shots of the cornerstone in the bakcground.
"Having the device also assumes having the software to run it. If it even still exists."
I've always been a big fan of keeping every version of every utility/application I come across. So even if future versions become NON-backward compatible, I can pull out my old version and load it up... on my old hardware with the old OS.
My coworkers laugh at me for owning one (actually two) of those "duo-drives" that have both 3.5" and 5.25" floppy drives that fit in one half-height 5.25" drive bay. My response is that some day, the Smithsonian Institute may come knocking on my door with a request to help read/convert old floppies from their collection.
Well, if you assume an order of one page plus including 44 cents for postage, then the 75 cents seems reasonable. Of course, if you purchase copies of 100 pages, they will put all of your copies in one envelope and mail it to you with about 88 cents postage - of course you will be paying $44.00 in postage that they don't have to pay to the post office. This is a great way to inflate profits, decrease demand, and punish the public that wants to view the documents. With that profit margin, it is unlikely that the NSA will post images online.
That would be like paying Shell $50.00 per gallon of gasoline. That covering the cost with profit for them delivering one (repeat one) gallon of gasoline to you - of course they transport it in 5,000 gallon tank trucks. That is what accountants call "economy of scale."
Speaking of old floppies, guess what? I have a client who does digital conversion. We have a project now from a government agency that requires indexing, copying to DVD and converting documents on 2,400 3.5" and 5.25" floppy drives (as well as a few Zip drives). Mostly dBase and WordPerfect files but a lot of other weird stuff, doo.
We got one of those Teac combo drives, but I'm having trouble getting Windows to read the damn drives. I can get the machine's BIOS to see them, set the BIOS for both drives, hooked up a Belkin cable which supports both drives, get Windows to SEE both drives - but it can't read both. It only reads the 3.5". Funny, though, if I boot with LiveXP, a bootable Windows XP, I can read the 5.25"! Go figure!
I should try Linux, it would probably read both without any troubles.
Have I mentioned how badly Windows XP sucks rocks?
@ Clive Robinson
Yeah, tell me anything you find on EMSEC. Most of what I've seen in the titles is obsolete or cracked codes and time- or situation-sensitive information. In other words, most of it is useless for us operationally and only has historic significance. This is not to say my fast skimming didn't find anything interesting:
"security handbook section on breaking unbreakables"
"interception of enemy radiotelephone and fascimile communications with privacy"
The security studies and description of electronic crypto units might allude to EMSEC issues or failure modes the public isn't aware of. It might help to organize the material into categories like crypto machines, national code books/intercepts, interrogations and general intelligence reports. Would be very time consuming. Good luck Clive... :P
I have looked through the list of documents provided to NARA. There is a great depth of material on Germany, Japan and the USSR, plus breadth on most of the world. If these documents are mostly complete, when combined with the recent declassification of NSA documents on WW2 Germay, I think there is now enough material to finally write the definitive history of cryptology from 1900 to 1950. Here's to all the archivists who carefully preserved this material for the last 60 plus years.
@ Richard Steven Hack
"did I mention how badly windows xp sucks rocks"
Actually, your talking about hardware support, which sucks and rocks in every OS. OpenBSD is one of the only OS's I know to maintain support for every piece of dated hardware they can find. Before Windows 7 and Driver Developer Kit, the Windows driver model wasn't that great and driver developers regularly developed crapware. It's unsurprising that you are having hardware difficulties.
This is a symptom of how hardware specification and driver development works in commodity markets, not Windows. We still have trouble getting NVIDIA and Intel graphics to consistently work on Linux, even with their official support. Your trying to get probably low quality drivers working on an obsolete OS without manufacturer help. That should give you a hint at the potential scope of the problem.
Thanks for the tip on the vans. I hadn't heard about local use before you mentioned it. The US is just getting scarier by the day.
@ Richard Steven Hack,
"but I'm having trouble getting Windows to read the damn drives. I can get the machine's BIOS to see them, set the BIOS for both drives, hooked up a Belkin cable which supports both drives, get Windows to SEE both drives - but it can't read both. It only reads the 3.5". Funny, though, if I boot with LiveXP, a bootable Windows XP, I can read the 5.25"! Go figure!"
The solution as I know from experience is an old 486 running Win3.11 / Win95 / NT4 / RedHat 5 with one of each of the floppies and a 10baseT network card.
As for those with WordStar or WordPerfect files you can grok them with the unix tools such as strings and tr utils into plain ASCII text and it would not be overly difficult to convert them to an older version of RTF to get the formatting back (at least the old DOS version formats of their files are available in books I have). The hardest job I've had to do so far is reading old 8" CPM disks. However I think it's probably time I copied my old Apple][ 160K single sided disks before it gets to late and my Apple][ give up the ghost (at least I can do that with an audio card and a scope probe).
As for punch paper tape I can still with a bit of effort read them by eye, it's a skill I was once required to learn when working in a ComCen (even as a Tech). You could always tell an old style operator from their hunched shoulders and rugby player arms from bashing away at KSR terminals when a key press of less than an ounce was considered touch typing ;)
And before somebody asks yes I have a number of "decks of cards" (god I remember hand punching some that really was programing the hard way).
I'd like to show up in person, cut out the middle man, and answer any questions they might have. It'd be a real money saver.
@ Nick P,
You might want to look at NR 964 out of historical interest.
Oh and NR 792 tells a story in it's own right (Lookup RAF Chicksands on the Internet and the "elephant cage").
Again from the historical perspective the various Bombe and FISH documents. One oddity is NR 1440 and then there are the various I.C. or Index of Coincidence documents, one of which NR 504 may well be a rework of William Friedman's original which the NSA have kept locked up for so long.
But I've only skimmed through about the first 3rd of the list and as you say a lot of it is effectively junk even to historians.
However there are one or two way off the wall titles NR1524 and NR 1565.
The $0.75/page fee just makes it easier to justify (request funds for) a visit to the archives.
@ BF Skinner
"What if adobe falls or someone buys the pdf copywrite"
No copyright. PDF (Portable Document Format) version 1.7 became ISO 32000 in 2008.
"Are 15 year old 5¼" floppies of Wordstar documents readable now"
Definitely, but the really good news is that the most data they possibly can hold is 1.2 MB.
@Clive "yes I have a number of "decks of cards"'
Your flat must be a fire hazard.
@collector "big fan of keeping every version of every utility/application "
Not just you. A couple of years back I found a MicroVAX 3500 still in production on one client site. It took arguing until blue in the face and waving the departments policy at them to make them to transition off it.
Say what you want about government funding/spending/taxes a lot of agencies are using the hardware and software WELL past their use by dates.
It gets worse, pulp paper and modern inks from the 20th century are getting all yellow and faded and fragile, while rag paper from the 1700's is still pliable and the iron-based ink it was printed with is still clearly legible.
Pretty soon archivists will be copying full time just to keep ahead of entropy... except for the stuff that's already on old paper.
I too have 5.25 and 3.5 drives as well as RLL harddrives, magtapes for IBM systems prior to Amdahl and punch cards and a punch tape. Mentioning MicroVax 3500 reminds me that I still have a DECtape. With all this old stuff lying around, is this what we call "security in depth"?
I'd easily bet that digital storage beats paper 10:1 on cost per kiloword-year for assured long term survivability under any circumstances bar global collapse of industrial civilization. That ratio is growing over time.
There are two main issues impairing the reading of old computer data and documents:
1) Physical deterioration of storage media and devices.
2) Digital data that cannot be interpreted easily because both the original software and file format descriptions are lost.
1 is easy to address if the data guardians act within within a few years of obsolescence, difficult to very difficult otherwise.
2 seems to be a significant issue only for organizations that wrote their own storage formats and software, like most government organizations and corporations did until the 1970s.
If you're using software that was at least moderately popular and migrated the data to new physical formats in time, no problem. I can read VisiCalc data created on an Apple II in 1982 because I migrated the contents of my physical media in the mid 1990s. Once the data is migrated, in the worst case you just run an emulator so the original software can interpret that dead file format for you. Emulation eliminates a lot of standard digital conveniences but paper eliminates them all.
@ BF Skinner,
"Your flat must be a fire hazard"
Well it's not a flat I live in currently, and a few dozen decks of cards are not as much of a problem as the eight and half thousand books and copies of all the cct diags, mechanical drawings and documentation software etc etc all in human readable format, then there are the data sheets etc. As a friend from a well known UK Uni remarked,
"You've a better library than the Uni!"
Then there is my collection of historical computer bits and bobs including an Altair PDP11 boards MicroVax and several home and semi proffesional systems such as BBC Micro and Torch 68000 Unix add on for it. And box after box of PC bits and bobs including every HD I've ever owned...
If you need documentation on MS Dross & Windoze through to MFC, and SunOS & SVR4 etc I've probably got something to help.
@Clive Robinson "....whoof...."
If I was your insurance agent? I'd be sizing you for a halon fire suppression system. 'course that's bad now a days...maybe a airtight room and massive pumps to de-atmosphere the space? Aero-k?
"To the CliveCave Robin!"
"1) Physical deterioration of storage media and devices
2) Digital data that cannot be interpreted easily because both the original software and file format descriptions are lost"
Issue (1) is as you say not an issue if you migrate your data. However older media (pre optical media prior to CDROM) is a lot more robust than many people think. Firstly because the bit density is quite low and stored in MFM&RLL format and does not suffer from the clever encoding issues used to get higher bit densities in later times.
Issue (2) is more interesting you have to think back to the early days and just how expensive even low cost storage media was per byte. For instance my 20Mbyte (yes really 20,000,000) HD for my Apple ][ cost more than the rest of the system and would be the equivalent of 20,000USD in todays money or about 1/3 of a years takehome income.
At that sort of cost each byte had real tangable cost, so saving every bit possible was a worth while excercise. I have in my possession a paper from the early 1960's that discusses minimal date storage and the future issues that we later called Y2K...
However the data encoding was generaly simple to understand such that it could be "hand worked" backwards if the media went bad.
This alows many old file formats (WordStar VisiCalc etc) to be easily understood. Thus the data is still either directly usable or atleast get easy access to the raw content.
That was untill Bill Gates threw the toys out the pram over what he saw as people stealing his property (for which there is evidence that he actually nicked it from elsewhere).
He was laughed at and derieded and it was probably at this time when most data storage formates where relativly simple and well known/documented he decided to cross over to the darkside ;)
Billy boy decided on "locking in" with highly obsficated file formats to effectivly lock up peoples data and keep them dependant on MS and it's frequent updates which almost always changed the data file format (try loading a Word 4.3 doc in later versions to see this). Oh and of course he continued to keep taking the work of others and re-branding it and knocking it out at high cost (look at all the early MS network code taken from BSD etc) and screaming blue murder when people ignored the outrageous "licence agrements". This gave rise to much of the later protection mechanisms that people then spent much time and effort breaking or obviating.
Even now MS take (Kerberos for instance) and extend in some way to make it effectivly proprietary and when that fails they try other tricks (such as nobling standards organisations), all of which makes life extreamly difficult for data preservers.
@ Clive Robinson
That's all true, but there's plenty that makes life easy for data preservers. The availability of tremendous HD storage, RAID systems, and open source file compression means they could store literally tons of documents at little cost. They'd occasionally have to swap out HD's, but the deterioration issue would largely be solved. It would take up much less space, as well.
On the issue of file formats, the DoD has been standardizing on certain protocols and formats, which may help. Additionally, there are a number of open standards for documents that would be easy to implement and maintain later, even with automated conversion. HTML 3.2, PDF-A, PostScript, and SVG come to mind. As for images, PNG and JPEG are well supported. (PNG would be ideal for scanned images of text) So, the data would stay around for as long as we want, retrieving it would be a mostly automated process, and maintenance would be relatively low compared to "fighting entropy" as one guy said.
The only thing that would be really expensive is the conversion process and the machines that scan reams of paper automatically can help with that a bit. The only thing that a digital system impairs is attribution and authenticity of historical documents. This issue might be worth keeping paper copies or making 3D copies of the paper documents. If the gov.'t started moving to paperless, then the situation would be easier. They already have a good PKI infrastructure for authenticity, so what's holding them back? That's justifiable?
Fun how they are BOTH simultaneous security issues.
@ BF Skinner
"Fun how they are BOTH simultaneous security issues."
Clever, but they aren't always both security issues in the same problem area. Data remnance isn't a security issue in archiving. Record preservation (or integrity) isn't a security issue in systems designed to primarily eliminate remnance. Operating system design is a problem area that must address both simultaneously. Think auditing vs object reuse and confidentiality requirements.
Not sure if you read this newletter or not, but well worth the time, has some insightful comments on the leaks, classifications, over doing security...
NR 4628 ZEMA37 24221A 19441201 SPECIAL FISH REPORT, 1944
Special Fish Report? uh... um...
"Special Fish Report?"
Without actually seeing the document concerned I'm assuming it is from the WWII work at Bletchly Park about a German electromechanical teleprinter stream cipher system designed and manufactured by Lorenz. Which if I remember correctly was a "high command" system that was supposed to be even more secure than Enigma in that it's maximum message length was many times longer and thus could be used for sending strategic battle orders etc etc.
The name "Fish" just happened to be the Ultra Code Word used by the British for the then compleatly unkown system.
Againif I remember correctly it was broken by William Tutt and was a quite astounding piece of work, that gave rise to the work carried out by Tommy Flowers to build the worlds first programable system usuing valves which we now know as "Colossus".
The "break" into the unknown system came about due to a simple but fatal "opperator error" by a German telegraphist on a very long (more than four thousand charecter message).
One of the "golden rules" about stream or addative ciphers is NEVER EVER use the same key twice (and a lesser rule is never send the same plaintext under a different key).
Basicaly the German operator sent the long message and started it with the equivalent of "Message Number". However the operator at the far end after receiving over four thousand charecters sent back a message saying effectivly "Message received incorrectly resend".
The German operators then made an operational error of reseting the stream start position not renegotiating a new start position and the sending German operator made the fatal mistake of this time sending "Message Nr" not as he originaly sent "Message Number".
This gave two copies of the same long message under the same key but from the Number/Nr initial difference at slightly different offsets.
This gave a long "message in depth" that allowed the plain text to be removed and a long copy of the key stream was revealed.
Due to a poor deign by Lorenz, William Tutt was able to work out the individual bit cycle lengths they had used and then reconstruct the stream generator pattern.
A mechanical analogue of the stream generator not to disimilar to the Enigma "bombe" system designed by Alan Turing was proposed. However Tommy Flowers from the Post Office research lab at Dollis Hill in London managed to persuade those in charge he could do it reliably with valves (which everybody else assumed from practical experiance where to unreliable).
The end result was the Colossus system, and it did end up with one part of it being unreliable, not the valves but the "tape tractor" used to run the ciphertext through the system. In order to keep up with the valves the tape system used an optical tape reader which was run a lot faster than it was originaly designed to. The result was friction in the tape tractor caused the tape to sometimes catch fire...
Remarkably this break of Fish was done around two and a half years before the allies ever saw the German system or even knew who had manufactured it, and the whole break was done by hand by William Tutt from just that one initial but fatal mistake by the (unknown) German telegraphist.
To all who think that a 75¢ fee is outrageous, go take a gander at the national archives building. The one downtown is quaint and neoclassical. The one in College Park is larger than a football stadium by several orders of magnitude. It dwarfs everything around it. And that's just the third of it that's above ground. A 75¢ fee for some staffer to search through this building for a document, then go all they way to a photocopier, then go all the way back seems reasonable.
Here's a google maps link: http://bit.ly/bOaZFP
Does anyone have a link to some of the more popular papers? I wana read the one on the zimmerman telegram! And the suspicion of the Japanese naval codes!
Schneier.com is a personal website. Opinions expressed are not necessarily those of BT.