NSA Increasing Security by Firing 90% of Its Sysadmins

General Keith Alexander thinks he can improve security by automating sysadmin duties such that 90% of them can be fired:

Using technology to automate much of the work now done by employees and contractors would make the NSA’s networks “more defensible and more secure,” as well as faster, he said at the conference, in which he did not mention Snowden by name.

Does anyone know a sysadmin anywhere who believes it’s possible to automate 90% of his job? Or who thinks any such automation will actually improve security?

He’s stuck. Computerized systems require trusted people to administer them. And any agency with all that computing power is going to need thousands of sysadmins. Some of them are going to be whistleblowers.

Leaking secret information is the civil disobedience of our age. Alexander has to get used to it.

Tags: , , , , , , , ,

Posted on August 12, 2013 at 2:33 PM • 85 Comments

Comments

Bogwitch • August 12, 2013 2:58 PM

I’ve always been told that Information Security is made up of three things – I think we all know what they are but this move seems to suggest they’re not taking much account of availability.

Given how much data the NSA are slurping, my guess is that 90% of the admins are involved in adding storage, nothing more. Perhaps they’ve found a way round that. We can but hope.

Marcel • August 12, 2013 3:01 PM

What would also help is not break the law, so there is no need for whistle blowers to leak anything. Also, it would also help to scale down the whole operation, say by 90%, and stick to what they they are supposed to do (which is not try to eavesdrop/store the while internet).

MathFox • August 12, 2013 3:10 PM

Firing their sysadmins will mean that the NSA needs more contractors for the job.

That could be a good thing for transparency!

ordinary • August 12, 2013 3:15 PM

I don’t think you’ll be able to find any sysadmins who will say they can be made obsolete, whether true or not.

Bryan • August 12, 2013 3:16 PM

Actually, as I understand that, his dilemma is twice as bad. First, he lays of 90%. The remaining 10% have to operate as teams of 2 for mutual oversight. Effectively, 5% will be doing the work of the former 100%. Ya gotta be kidding me. Color me LYAO.

mishehu • August 12, 2013 3:25 PM

The shoe is now on the other foot? Mr. Alexander… you have nothing to worry about in your organization if you have nothing to hide… does that ring a bell?

H4x • August 12, 2013 3:26 PM

Microsoft built a private government cloud so they just outsourced everything. I imagine NSA has a secret cloud they built themselves as well.

Will work great until attacks to the VM host start popping up. I wonder how many logic bombs those sys admins left or secrets they carried out the door.

Tinfoil: they could secretly still be on the payroll and released to the private market to sabotage and spy for the NSA

Also Bruce check out new zealand post today. Big story on how NSA extorted telecoms wanting cheaper backbone access by allowing them access to the traffic to spy on. NZ telecoms would have to charge 4x as much to provide service if they didn’t agree

Herman • August 12, 2013 3:29 PM

Considering that these people are masters of weasel words, what he is probably saying is that 90% of sysadmins will be fired and then hired back under a different title. Since they now need two people for each job, they would need to call them something else. So watch the job adverts – there will likely be a slump for Code Monkeys and boom for Tape Trolls.

Ken Hagler • August 12, 2013 3:30 PM

Personally, I think it would be simpler to just not be so evil that you have a hard time finding enough despicable scumbags to fill your ranks without worrying about a decent person with a conscience getting hired by mistake.

Scott • August 12, 2013 3:36 PM

So those remaining 10% will necessarily be in charge of more information per person (reducing separation of duties). While you’ve possibly reduced the incidence of leaks, you’ve increased their magnitude netting no difference in the overall impact of such events.

Congratulations!

Vittorio • August 12, 2013 3:39 PM

How many sysadmins at NSA are now collecting data for publication in case they will be fired?

Harvey MacDonald • August 12, 2013 3:42 PM

If I recall, the NSA stated that many of their sysadmins were performing mostly non-sysadmin work, for example, moving a cache of files from one server to another. These types of jobs could be reasonably delegated to analysts.

It is funny that NSA sysadmins are now supposed to be doing TMI/TPI, with 10% staff.

Doug D • August 12, 2013 3:44 PM

I am sure that notifying the sysadmins that there’s a plan for 90% of them to be laid off will not push any of them over the line into whistleblowing…

tai • August 12, 2013 3:49 PM

As I’ve mentioned elsewhere, a good sysadmin would have already automated the pieces of work that can be automated.

So what does this mean? They’re going to implement tools where they previously had one or more people working on it, that is now going to be tasked to someone who’s stretched really thinly.

In every other aspect, they know to use two or more people for failsafe, and to make sure the other person isn’t going rogue. Now, you have no one watching it…

Twirrim • August 12, 2013 4:05 PM

It really depends on what their sysadmins are doing. Without any real insight into the role it’s a little hard to say “yes” or “no”. Honestly from the comments in the article: “what we’ve done is we’ve put people in the loop of transferring data, securing networks and doing things that machines are probably better at doing,” it doesn’t seem particularly beyond the pale.

“in the loop of transferring data” That’s one of the reasons we use computers anyway, right? Consistent, automated repetition of a task. It really shouldn’t take a sysadmin to manually move data from one place to another. Build a system with the appropriate checks and balances, and let it do it far more consistently than any human or manual process could ever manage.

If the NSA is extremely behind the curve in other areas (e.g. no config management), there’s every chance a sizeable number of Sysadmins could be made redundant. If that’s the case, arguably that should be done purely for budgetary reasons anyway.

Figureitout • August 12, 2013 4:09 PM

I love how he doesn’t consider firing himself, nope axe the little guys as always.

Plus this will make for a real trustworthy environment, everyone looking over each other’s shoulders; maybe even more backstabbing/sabotage to get job competitors fired.

In an otherwise decent article by Boyd (that didn’t really mention MS’s involvement w/ NSA but AT&T’s…), I find it annoying she writes about distractions from the story on the part of journalists of enormous gov’t overreaches and then responds to the first comment about how all these whistleblowers seem to be “white males”.

Darryl • August 12, 2013 4:13 PM

I doubt that 90% is a good number, but that depend very much on the consistence of machine configurations. I suspect he is listing to one of the producers of such automation software such as IBM’s Tivoli. The problem is that they will then have to hire enough staff to maintain and customize the management software. That will mean that the body count will not reduce by the same amount they will change roles.

As to reducing the access of the admin staff to secure documents this will happen. Yes the automation support team will not need shell access to all the machines they maintain control of. But if they are not watched closely most of the packages will allow arbitrary commands to be run. This means the difficulty has risen but not by much.

My question is why would the restricted documents on disk not be encrypted? Even with sysadmin access then they would not be easily readable. If the answer is efficiency then that was their choice to be vulnerable. A saying of mine when people complain about how some new security initiative make thing harder or slower is “increased security has never made anything faster, easier, or more efficient, what it does is to keep the wrong people getting a hold of what they shouldn’t.<\i>”

Eric Black • August 12, 2013 4:15 PM

Can these automated systems actually fix thing when they are going wrong? I can’t imagine they were paying sysadmins to manually start backups or copy data that could have beens scripted. That stuff would already be scripted. Same thing for patch management. Sounds like someone who doesn’t understand what his employees actually do.

Edward S. • August 12, 2013 4:20 PM

Or maybe NSA just had 10x as many sysadmins as they need. With a secret budget and no oversight they can hire and spend without normal govt restrictions.

Figureitout • August 12, 2013 4:26 PM

Sounds like someone who doesn’t understand what his employees actually do.
Eric Black
–Bingo. I wonder how much code he can read or even a basic microphone/camera understanding…

David • August 12, 2013 4:30 PM

Let’s not be too hasty in dismissing this idea of ‘improvement by reduction’ … we could try this same concept with NSA management, analysts, and programs.

John Thurston • August 12, 2013 4:36 PM

We can’t trust the humans, so let’s automate the management of our computer systems.
Sounds to me like the foundation of Skynet.

name.withheld.for.obvious.reasons • August 12, 2013 4:44 PM

How many NSA admins does it take to screw in a light bulb?

I can’t tell you that, it is a operational AND strategic security only known and approved by the FISC.

Nick P • August 12, 2013 4:45 PM

@ Bruce

I’m mostly in agreement with you. I got something re the sysadmin role, though. Much of what sysadmins do can be automated. We can see this with products that automate it and products that are mostly self-managing. The old AS/400’s with redundant HD’s come to mind: configure it and it just keeps doing its thing for a while before work needs to be done. Companies have sometimes forgot they had them.

I know Boeing and other contractors have produced software that automated configuration of firewalls, IDS’s, access controls, etc. They usually generate it all from a policy language or similar high level tool. Those high level descriptions also have an advantage in being able to detect [maybe intentional] problematic configurations.

Example: “We used Survivability Grammar to generate configurations for our VPN and Spread
infrastructure for the Critical Design Review (CDR) demonstration. This consisted of 42
workstations and 7 routers. To gain a sense for the
usefulness…[the] infrastructure were specified in less
than three pages. The specification was compiled into about two hundred pages of component
configuration files. (my emphasis added) Changes were always made to the specification, never to the configuration
files. This dramatically improved the manageability of the network.”

That kind of tech can automate all sorts of system, network and security administration issues. Preventing subversion might be a legit reason for a group NSA’s size to collect it all together into one major capability. More deployment details about that later.

I’ll skip on issues like moving data which other commenters mentioned: they’re obvious candidates of automation.

Another thing that comes to mind is combining the autogeneration with profiles/templates. The work could be shifted so the main job of sys admins is following best practices or frameworks to configure a given type of server, application, or network device. This would only require test systems or data rather than actual classified information. The required steps can be turned into scripts or specified for automation software. Then, the main job of sysadmins of production systems will be physically installing/fixing the systems, identifying their type, and selecting available operations to perform on them.

The orchestration of these could even be automated as series of admin tasks to be executed, all logged, so observers wouldn’t have to necessarily watch every keystroke of the acting sysadmin. The physical maintenance should be done in batches from a work order. It gives both efficiency and observation benefits. Another idea is using HD encryption and doing a quick RAM + key wipe before physical work is done. That would leave no classified data on the server for the worker to find.

(Might also help to keep different people doing physical and logical administration. And keep them from talking together. Just a thought.)

So, I think with the right choice of systems, architecture, standardization and configuration they can automate most of their generic sysadmin activities and some domain-specific ones. Physical repairs, corner cases, custom admin actions and administration of “trusted systems” that regulate other administrators will all still be risky. Yet, there are still many fewer humans in the loop and the sheer number of risky jobs becomes small enough that additional security for them might not hurt operational performance that much.

Side note: it would also be advantageous if the automation software was made by non-government developers working for several different countries (with reviews that include govt developers). This should reduce the risk of subversion for ideological and nationalist reasons. The last thing you want is for the subversion prevention tools (automation) to be subverted. That’s just embarrassing. 😉

Dave Walker • August 12, 2013 4:52 PM

Most of the security incidents I’ve been involved in sorting out, have been perpetrated by disgruntled former employees of the targetted organisations. Gen. Alexander looks set on causing a lot of people to meet that description.

Thomas • August 12, 2013 6:25 PM

Does anyone know of any example in any industry where you could lay of 90% of the workforce by “working smarter, not working harder”?

The only way this could possible work is if 90% of sysadmins are currently doing no useful work, and the remaining 10% are all that’s needed.
Guess which 10% of sysadmins are going to be the first to leave, knowing their skills will get them a job anywhere, and which 90% will be desperate to hand on to a cushy (non)job.

cael • August 12, 2013 6:32 PM

Step 1: Massively overhire sysadmins in an attempt to compartmentalize information and keep the impact of leaks low (lmao).

Step 2: Fire most of the sysadmins and turn them into disgruntled former employees with top secret clearance.

Step 3: Automate system administration (it’ll work this time, seriously we promise)

Step 4: ???

Step 5: Profit!

Brett • August 12, 2013 7:03 PM

Automated SysAdmin, isn’t that skynet?

Seriously, how can anyone actually think automating 90% of sysadmin is a good idea?

Gweihir • August 12, 2013 7:23 PM

Well, I applaud this development, as it clearly says the NSA upper leadership is terminally stupid. This may be the one positive quality they have (judging from the perspective of the rest of the world) and the one chance the world still has of preventing them from forming the core infrastructure of a totalitarian system.

That said, you have sysadmins because you need them, not because you want them. And as to 4-eye procedures, we evaluated that some time ago for a civilian customer, and came to the conclusion that you need to have watchers that are a lot more competent than the doer and then you still need 2-3 people doing the watching for every one doing the working. And, surprise, a few months after we told them that, it did fail and they had an incident were a competent guy put a backdoor into the system right under the nose of the watcher.

4-eye is good for procedures like filling out forms on paper or counting money or other inventory. It becomes extremely hard and routinely fails when doing it for a computer screen.

Nick P • August 12, 2013 7:28 PM

@ Gweihir

“That said, you have sysadmins because you need them, not because you want them. And as to 4-eye procedures, we evaluated that some time ago for a civilian customer, and came to the conclusion that you need to have watchers that are a lot more competent than the doer and then you still need 2-3 people doing the watching for every one doing the working. And, surprise, a few months after we told them that, it did fail and they had an incident were a competent guy put a backdoor into the system right under the nose of the watcher.

4-eye is good for procedures like filling out forms on paper or counting money or other inventory. It becomes extremely hard and routinely fails when doing it for a computer screen.”

Very well said. I think the 4 eye stuff doesn’t work so well b/c of the complexity and occasional creativity involved in the work. Not to mention the person typing must do just a few commands at a time to guarantee the other can read, make a spot judgement and do a quick security review of the commands’ effects. It’s all quite ridiculous without automation of most of their job and it’s still hard to see it working.

Dwatney • August 12, 2013 7:35 PM

Translation: We don’t trust sysadmins, so we are going to put it in the hands of software developers.

Dirk Praet • August 12, 2013 7:46 PM

@ Bruce, @ Nick P

Does anyone know a sysadmin anywhere who believes it’s possible to automate 90% of his job? Or who thinks any such automation will actually improve security?

It is highly likely that some consulting outfit has managed to convince Gen. Alexander that the sysadmin problem can be rooted out by replacing them by a much smaller team of devops capable of (securely) automating deployment, maintenance and monitoring of large infrastructures. I do wonder how this is going to work out.

Frank • August 12, 2013 8:54 PM

I think it’s great that we have a democrat as president.

This way everyone talks about the actual problems rather than just blaming the president.

Of course in this case, the president is actually in charge of this…

ax25 • August 12, 2013 10:59 PM

I am guessing the General saw an advertisement in People Magazine that told him to do it. I would think they would keep a better eye on the media for that sort of psi-ops sort of stuff, but I guess they are not immune. On a more serious note. I think the complexity of the situation would define the number of admins needed. If you cut back too far, you will find that things just don’t work right. If 90% is a shot in the dark, they might find out quickly 🙂

Frodo • August 13, 2013 12:19 AM

Nemesis – the ancient Greek goddess, punisher of arrogance and hubris is in the network and she is excited to get to work. Can you hear her laughing? I can.

SF Kinney • August 13, 2013 3:03 AM

Many posters have made sound comments on the likelihood that this announcement amounts to smoke and mirrors, and I am afraid that they may be right. Historically, the NSA has been our most competent intel service. But things change, more and faster now than ever. In the private sector, every time ignorant self-important “suits” step in to micro-manage the IT department, the result is smoking rubble. Within six months, Anonymous splinter cells armed with script kiddie tech may very well be traversing the NSA’s most “secure” networks at will. Lulz aside, that would be a mixed blessing at best – because some very small fraction of what the NSA is riding herd on is actually related to national security. Hopefully that’s where the 5% of remaining BOFH types will stay, and get strictly left alone.

Nargunomics • August 13, 2013 3:16 AM

@Dirk Praet – read The Mythical Man-Month. The cover picture of prehistoric sloths drowning in the La Brea tar-pits should be sufficient answer.

Facetiously, development of complex systems is not at best well understood. Operating systems are relatively complex for the average developer, yet the Unix architecture is simple enough to keep in one’s mind – that’s how Linus Torvalds, Andy Tanenbaum and Bill Jolitz manage to keep their OS projects manageable. There have been OS projects not so well-designed – let them rest in piece.

What we would have in this “NSA hardening project” would be several orders of magnitude more complex, and compounded by the lack of trust assumed to exist between developers, management and the NSA. It can’t be easy to work for people you know are liars, particularly when they inform you that you will not be monitored while working on such a project.

In any case, the weakness of the American dollar is likely to prove fatal to the project – the slogan of the sixties-onward Soviet Working Class will be resurrected by NSA contractors – the government pretends to pay and the people pretend to work.

Bart • August 13, 2013 3:26 AM

Suppose 10% are left for oversight. To configure automation You have to understand systems in detail. Sometimes You have to compensate, add functionality or adjust according to current demand/needs(systems arent static too). What is the learning curve of 90% sysadmin tasks that were done(they are supose to maintain)? Plus added complexity of automation tools.
Notice also analogy to: faster development = more code reuse = less innovation. Supose if only 10% are left. All innovation would be outsourced? What about shifting knowlege? (isnt it more secure if its inside organization?). Also what about detecting and reacting to incidents?
Over all. Arent they exchanging Snowden type problems to Manning type problems?

sdk • August 13, 2013 4:10 AM

You guys have wrong point of view on this. It’s not that most of sysadmins do unnecessary job. With modern automation tools (and no, IBM Tivoli is not one; CFEngine, Puppet and Chef are) sysadmin’s job may scale up to hundreds or even thousands servers per person.

This is all about enabling a small team to do much more (then the small team could take over duties previously held by larger set of people), not about making fewer people to work harder.

Well, at least this is how it should be carried to achieve 10x admin staff reduction. But it would require to have (hire?) highly skilled people, and this could be hard. Market already sucked them off, and what they’re working in agile, convenient organizations. I don’t believe NSA could offer less stiff work organization, so salary is pretty much the only option to compete.

Clive Robinson • August 13, 2013 5:04 AM

Let us be honest about this, the “90%” figure is pulled from the air for publicity reasons. Probably due to political kick back on the previous 2key/4eye announcment that would appear to double up over paid staff in times of supposed austerity. All of which started with the knee jerk managment think policy of “Need to say something to be seen to be doing something” rather than pause, investigate, think, reflect and take proportianate action in a timely manner.

What I can guarenty is that this “dogs dinner” is going to end up as an expensive pile of crap shortly after it has faded from the public gaze, and will get pushed under somebodies carpet to ensure it stays out of sight and thus out of mind.

As others have indicated current staff will either leave if they are good or be redeployed or have job titles changed if they are average or below par some of whom will actualy get promotion/pay rises out of it. More consultants will be brought in (but probably not from Ed Snowdens previous employer) and there will be procedual changes with automation etc.

Those that chose to leave will in some cases end up working for certain contractors because of their specialist knowledge of NSA systems and proceadures.

In many respects Ed Snowden has done the NSA senior managment a big fat favour in that it will be able to play “catch-up” if not “jump over”. This is because it is an almost certain bet that the NSA proceadures were antiquated in comparison to most corporates and smaller NGOs. This being due to “budgetting issues” common in all large governmental organisations. So in effect it is now the NSA’s oportunity to have a “blank cheque” to do a technological Ageian Stables clean up.

And I suspect “the usual suspects” know this and are currently circling like vultures with draft proposals they have dusted off, for just how they as contractors with those of specialist skills on their books can solve all the NSA’s problems.

As long as Gen Alexander gets the blank cheque he is not going to lose either. Because he will have rotated out long befor any of the projects will have realy got off the drawing board to a nice cushy extreamly overpaid job with one or more of the larger contractors. And I suspect he will be joined by other seniors “jumping ship” to cash in on their knowledge and contacts.

The contractors won’t lose because they will ensure the contracts are such that the NSA will have to pay them off to vacate them (See “off book spending” PFI deals in the UK).

Oh and back at the NSA the systems will almost certainly not work due to the way the contracts will be written and NSA staff will have to resort to back door tricks just to get their work done. So no real change, just a lot of disruption funded by unimaginable chunks of tax pay dollars, “Such is the American way” these days with the chosen few milking it as hard as they can for the best of the cream to waste on their fat cat wiskers.

And for those that think I’m being a little cynical would you like to place a small wager on your idea of the outcome?

Keith A. • August 13, 2013 5:36 AM

I expect a role name change for a large percentage of Sys Admins.
90% less Sys Admins, none disguntled. Face Saved.

win win

name.withheld.for.obvious.reasons • August 13, 2013 7:18 AM

@clive

I’m taking an editorial liberty here, you opened the invitation and I accept.

First let me say I am pretty much on board with your supposition(s), where I think this might be going is a variation on the theme…

And I suspect “the usual suspects” know this and are currently circling like vultures with draft proposals they have dusted off, for just how they as contractors with those of specialist skills on their books can solve all the NSA’s problems.

I see the possibility that two birds, one stone, and add bunch of arrogant asses and we have the following:

  1. Cut your staff members and costs, can be recouped for other things especially since the DoD has been given “flexible” authorities. Besides, there is nothing more manager’s hate than smart ass admins, programmers, or engineers–they’re all a bunch of primadonas.
  2. Make deals with the telcos, service providers, etc. to process the NSA’s
    current work load–possibly expanding their short term capabilities…
  3. Let the new corporate partners know where they can “find” the personnel necessary to deal with the functional transfer (don’t forget the kick-backs)
  4. Inform congress that you’re no longer running any of those nasty programs–wink, wink; nudge, nudge.

Old Sysadmin • August 13, 2013 7:51 AM

I’ve not chased down other articles to see other quotes, but the article says that they are /reducing/ their sysadmins by 90%.

Could mean that people are getting the sack, or it could mean that they are reducing admins by taking away privileges–which I see as a much more likely scenario based on the activities that were listed for automation.

My guess is that they are no day-to-day sysadmins, but folks who have admin access to move data, etc.

Kevin Lyda • August 13, 2013 8:46 AM

It’s possible to automate a great deal actually. But firing 90% of your admins is not exactly the best start.

First, you need more staff to do the automation. You might let 90% go eventually (you need admins who can code), but not initially.

Second you can’t just automate what they’re doing now since they clearly have too much access and data security is too loose. Automating insecure actions just makes it faster to do insecure things. That’s actually not a win for security there.

Lastly they clearly need to build a better auditing system and a better authentication system. Every action should be tagged with who did it. And those logs should be audited – by hand for portions, but automatically for the bulk of it.

It’s not simple and it’s time consuming. Statements from the NSA make me seriously wonder if they’re up to the job. Which is shocking really.

Reed Wiedower • August 13, 2013 10:06 AM

I’m a local CTO here in the DC area and these sorts of questions around reducing sysadmin functions in day to day life, in both the windows and linux operating system environments, come up a lot in discussions with clients. The sad fact is that most of the work sysadmins do at large shops boils down to:

• Fixing repeatable errors
• Patching systems
• Responding to predictable alerts
• Deploying servers, services and applications
• Setting permissions

Almost all of these can be automated. The trick is often that the more complex or custom-coded a solution, the more difficult it is to automate. The Windows and Linux worlds have made a huge amount of progress in the last 5 years though, and modern versions of most operating systems can be fully automated to a great degree. Does it cost money to setup? Sure! Does it pay for itself in a year? Definitely. Much of the progress has been made by including tools that used to be highly expensive into the very operating systems themselves. Thus, the argument that automation leads to higher requirements for outside consultants to be brought in begins to erode.

As a former sysadmin myself, I often saw other admins assume that the tasks they were performing were important because of the sensitivity of the data, security concerns, etc. Yet many never stopped to think “does adding a human in this chain provide a good level of oversight, or actually increase the risk of errors (whether malicious or otherwise)?”

The other flip side is self-service: why have a sysadmin set permissions when the person who “makes the call” should be capable of doing it herself? Between self-service and automation of common tasks, I think you could easily get to 75%, and maybe even higher, for a majority of systems administrators. It doesn’t eliminate the need for trust, but I imagine the NSA is fairly calcified. If they even are just reducing privileges by 90% for their existing sysadmins, that’s still a step in the right direction.

Nick P • August 13, 2013 11:03 AM

@ Dirk

I agree. I think he might have also heard of commercial claims to do the same.

@ all

Examples from commercial world

How many servers per admin?
http://www.datacenterknowledge.com/archives/2009/12/30/how-many-servers-can-one-admin-manage/

“That matters, as the company may pack more than 300,000 servers into its new container data center in Chicago. It expects to support that facility with about 30 employees, including admins and facility maintenance staff.”

AOL’s model ties into my physical security recommendation nicely
http://www.datacenterknowledge.com/archives/2011/10/11/aol-launches-new-lights-out-data-center/

“We simply move the instances (or create new ones) to other data center facilities and the failed equipment is addressed in a scheduled way using outsourced or vendor partners. ”

That’s basically what I suggested in my post for separation of duties and efficiency. The article tells us it works in practice while saving money. Win, win, win.

Virtualization

Articles like these often also mention virtualization greatly increasing the ratio of servers to administrators. I often see over a ten fold increase in their claims. NSA could create a few standard images for what they need in their datacenters. The images would be configured, updated, etc. However, most production administration would just be turning services on/off, loading/removing VM’s, moving data or network configuration. The physical stuff could be done by other people.

Virtualization benefits w/out the virtualization

For one reason or another, they might not want the virtualization. The same thing virtualization does can be done on Linux-based systems with ease if most of the nodes are the same type of hardware. There is plenty of software out there for creating/updating Linux images, remote administration, automated patching, clusters, grids, service management and so on.

Much of it was invented when parallel clusters became popular. Quite a few of the tools are free and open. I worked with some while building a Beowulf cluster in the past. They’re quite effective. This approach has already been used to manage clusters whose nodes measured in tens of thousands. NSA seems to run a small number of programs on massive amounts of data (MPMD or parallel SPMD). A “semi”-clustered approach might help them if they’re not doing it already.

Privileges

A few people pointed out admin privilege reduction might be the goal. That’s a really good idea/point. NSA might benefit from the use of simple rule- or role-based Mandatory Access Controls that restrict access on a job by job basis. This was implemented to a small extent in OS’s like Trusted Solaris, Trusted IRIX, Argus Pitbull and SELinux.

What NSA could do here is divide the roles. Auditor’s would only have access to logs. Admins looking to fix hardware problems could be restricted to device access and perhaps pre-defined scripts for testing them on the classified network. Admins working on network configuration could only access the files that pertain to that, a change in which would reload the network subsystem. Most human actions would produce an audit trail that had a certain percentage chance of being reviewed. Also, custom types of jobs could be restricted to a very small group as most jobs are less privileged.

Change Control System

This situation needs one. One of the main tools against subversion in a high assurance project was always having a good SCM system with review steps for submissions (and signed submissions). NSA can build one of these for administration. If each change is logged here, then review will be easier and maybe tool assisted. A different team can maintain the repository, as well. The repository might also be used to contain the standard system images and administration tools that are allowed.

Peter • August 13, 2013 11:19 AM

Wow!
If he pulls this off, NSA can start a serious money-making consulting division as a spin-off. 🙂
As there are a lot of CEO’s and CIO’s out there who would be very interested in this kind of ICT cost savings…

Greg • August 13, 2013 11:57 AM

Through a concerted effort, coming into a very inefficient IT organization, I’m managed to automate perhaps 25% of what was previously done by 5 sysadmins (myself included). This is using a combination of reporting, bash, batch, powershell, virtualization, de-duplication of systems, and changing software packages.

I honestly don’t think, even in as disorganized place as this, it could get much better. Of course, NSA IT might be even more ridiculously disorganized, but 90%? Not going to happen.

Just another anon • August 13, 2013 12:27 PM

Or maybe they’re telling the truth, and this will be bad for the NSA and good for everyone they’re spying on. Which is everyone on the planet, most likely, and certainly everyone north of the Rio Grande, except maybe in the less-wired parts of the Canadian tundra.

Wang-Lo • August 13, 2013 1:08 PM

Ninety percent staff reduction seems feasible if the bulk of the actual work is outsourced to offshore contractors.

-Wang-Lo.

Christian Koch • August 13, 2013 1:52 PM

General Alexander talks about sysadmin automation as if it’s some sort of innovative, newfangled thing.

It’s not, and it never has been. Shell scripts, people! I echo many people’s opinion that a good sysadmin already automates the tasks that are suitable for automation. (Implying that just because something can be automated doesn’t necessarily mean that it should.)

I suspect that Gen. Alexander and those who report to him, in particular, would find automation more novel than most, perhaps because they tools they currently use aren’t easy to automate to begin with? E.g. compare (a) simple Unix pipelines and redirection to plain text files versus (b) trying to autopopulate Microsoft Word documents.

paul • August 13, 2013 2:09 PM

So I assume this really means they’re going to tell Booz and their other contractors to cut reported headcounts. Perhaps this was the kind of thing that shouldn’t have been massively outsourced in the first place.

Some tasks, if you can’t grow them as fast as you want without massive outsourcing, perhaps you just shouldn’t grow them that fast.

999999999 • August 13, 2013 4:21 PM

@Clive
the way you define “sysadmin” is different than what the General was talking about. Merely having elevated privileges on a system or part of a system does not make you a systems administrator.
Also:
If I was a sinister fellow I would read his comment as “We will terminate some abilities of people to access information on the system.” this would include people who have oversight on their programs.
This is a good time for No Such Agency to terminate access of anyone they don’t deem worthy. It is also a good time for BAH to show off how plugged in they are to the cyber in-security system.
I have complete and utter faith in the intrinsic eventuality of these people to bungle things up. On the one hand-these programs are breaking (blatantly) several laws. So they should be reduced by 90% as the Generalissimo is misstating. On the other hand-the people arguing in favor of massive state-sponsored secret surveillance are convinced that they are doing this to “protect our children from terrorists”. So firing 90% of the people in important positions is a really bad idea because it leaves our children defenseless from terrorists. On the gripping hand-these programs are inevitable…Google, M$, FB, Comcast, Amazon, AMEX, LinkedIn, AT&T, Twitter, VZW, Visa, Time Warner Cable, etc. are collecting information to better sell us stuff we don’t need with money we don’t have. Targeted ads are effective if they can follow our every move and know what we want before we want it and sell it to us.
Same thing with “stopping terrorists”. The “good guys” need to know what you are thinking and then they will decide if you are “good” or not. If you are not “good” they can now scan your whole life to find every time you did anything that can be used against you.
So to scan everyone’s life you need people:
I make some maths…
Let’s assume that about 1000 people can be “monitored” by 1 and person. Scale it up and it equals about 5,000 employees for 50 Million people and about 5,000 are being hired in Fort Meade. These programs are here to stay.
“Protect our children from terrorists”

Howard • August 13, 2013 5:01 PM

While I agree with Snowden’s “civil disobedience” and in my gut figure he did the right thing, here’s what keeps bugging me about Snowden, Manning, Assange, et al:

They’re always leaking info about one country.

While it may not be deliberate, they are only hurting one country with their actions. Yes, hurting. Helping in a lot of ways, and making the world better in a lot of ways, but also causing damage to systems put in place to keep people safe (such as, Afghan informants killed after their names appeared unredacted in the cables Manning gave to Wikileaks).

Here’s the question: how do you spell “Snowden” in Mandarin or Cyrillic? Where are the Chinese or Russian counterparts … where are the massive leaks of troves of data, embarrassing and hurting the defense of those countries? Aside from US spies, such things are either non-existant or too rare to matter. They certainly don’t make NY Times front-page for months on end.

Guy Fawkes • August 13, 2013 5:16 PM

@david

“improvement by reduction”?

Might work, but the only real improvement to the NSA would involve a 100% reduction.

Dirk Praet • August 13, 2013 8:31 PM

@ Howard

such as Afghan informants killed after their names appeared unredacted in the cables Manning gave to Wikileaks

As per official US DoJ testimony during the Manning trial, there was exactly 1 (one) case thereof in which 1 (one) Afghan informant was killed. Admittedly, that’s one too much, but in terms of collateral damage an infinitely better score than what the US military finds acceptable in, say, the average daily drone strike in Afghanistan or Yemen.

Where are the Chinese or Russian counterparts

Dead, lobotomised in institutions for the criminally insane or imprisoned in some distant gulag never to be heard from again. Their families probably too. For all we know, their sysadmins may even be doing a better job at securing classified information than their SIPRNet/NSA counterparts.

China and Russia are totalitarian police states. They know it, their citizens know it and we know it. I don’t think they even deny it, unless an official making such a statement in front of an international audience would be hell-bent on drawing massive laughter and becoming the new Muhammad Saeed al-Sahhaf (Comical Ali). They don’t claim to be the leaders of the free world or lecture other countries on issues such as democracy, civil liberties and human rights.

In essence, and with the exception of operational details and specific case studies, a Chinese or Russian whistleblower could probably not tell us anything we don’t already know or more or less expect from those countries (APT’s, mass surveillance, backdoors in tech products, industrial espionage, shady brokering for influence and resources etc. etc.) What Snowden and Manning have revealed – and what so far was only assumed by tinfoil hats and members of intelligence/security communities – is that the US for all practical purposes have been doing the exact same thing for at least the last decade but have always been lying about, both at home and abroad. And, unfortunately, being called out on hypocrisy is a serious bitch indeed.

Soothsayer • August 13, 2013 10:11 PM

In ANY government enterprise .. you SHOULD fire 90% of the staff to improve things.
Security is no different.

All ADMINISTRATIONS are bloated welfare agencies of kith and kin .. they server little purpose other than to promote their importance and welfare.

“System” Administrators are the special breeds of misfits (at least that’s what I have found them to be EVERYWHERE I have seen them) that hide in the equipment racks as some other blattodea do in the woodwork.

Anon • August 13, 2013 10:50 PM

@Dirk

It might only be one informant or it could be a lot more. You really have to drill down to see how DOJ derived that number to determine how much trust to place in it. If an asset becomes a no show, is he presumed alive until the US finds a body? Even if the US finds a body, is the death presumed unrelated to the Taliban unless the Taliban publicly takes credit for the execution? Even if the Taliban publicly takes credit for the execution, does DOJ presume the Taliban found out his identity through means unrelated to wikileaks unless the Taliban says they read his name in wikileaks?

Ravan Asteris • August 14, 2013 1:56 AM

General Alexander is a typical management idiot. He’s the same kind of fool as those who think that if they put “everything” in “the cloud”, they don’t have to hire sysadmins, it’s automated and done for them.

As many systems as they have to have, they will require systems and network administration.

If you have an average disk failure rate of 1% per year, that’s 10 failed disks per 1000 servers (at one disk/server) per year. Each of those servers must have monitoring to see that their hardware and software is working as required, and that requires more than machines to watch it. Repairing and re-provisioning a server takes admin time – and even with an automated provisioning system, someone has to check it, take it out of service, fix it, and start the re-provisioning.

People who run large scale server farms are always trying to come up with ways to automate as much as possible. Even if they never changed their software stack (which you know is not true), they would still need sysadmins who know how to add machines, repair and re-provision machines, and retire machines.

People who do this sort of work need root level access – they are installing and running your basic OS and software stack. There isn’t a way around it.

Some large, redundant server farms can probably run without skilled sysadmin intervention for a couple months. But I wouldn’t want to be the poor sod who got tasked with fixing them after that time was up.

Steven C. • August 14, 2013 5:13 AM

I think this was reported all wrong, he didn’t say anything about firing those people with access to secret information:
“The National Security Agency … intends to eliminate about 90 percent of its system administrators”

Phil • August 14, 2013 12:06 PM

I’m sorry, but any SysAdmin who’s worth his weight has already automated 90% of the job. Would you really be supprised to learn that a government agency has a bunch of incompetent, lazy, untrained admins who have been there for 15 years doing nothing but protecting his/her job? I mean, I think I would die of shock if a sysadmin in my own organization knew what a script looked like…

Automating 90% of ‘traditional’ sysadmin responsibility – totally feasible.

yargle • August 14, 2013 9:39 PM

Best comment on this story was from wikileaks, reminding canned sysadmins not to forget to take their HDDs when they leave!

Michael Moser • August 15, 2013 2:12 AM

Funny: who is going to be the administrator of the automated administrators? By moving this problem up to another meta layer you give the person on the meta layer level even more powers (on a higher layer you can get access to even more systems. There does not seem to be a solution to this kind of problem.

Michael Brady • August 15, 2013 8:59 AM

When you fire 90% of your system administrators your system crashes.

When your system crashes you can’t spy on anyone.

When you can’t spy on anyone there’s no sensitive information to leak.

You know, this just might work.

aboniks • August 15, 2013 5:29 PM

Then again, what if Alexander is right?

Haven’t seen much evidence that NSA networks are being penetrated, only attacked internally.

He wants to make them more “defensible” with this move. If he’s convinced the most significant threats to his network are internal, then reducing the number of potential threats is an obvious move.

You think they’d be keeping the 10% of admins who were most technically capable, or the 10% who were most trusted?

When I was in the signal corps, the internal threats to our systems were almost always considered the most significant ones.

For those above who think that General Alexander is some sort of uniformed buffoon getting his operational advice out of crackerjack boxes, I suggest you dig a little deeper on his background. This guy is no idiot, and nothing he says in the public eye about policy is ever going to mean exactly what it appears to mean on the surface.

rhothet • August 15, 2013 7:58 PM

Not my field of knowledge but I’m reminded of Fannie Mae’s near death experience

“Feds Allege Plot To Destroy Fannie Mae Data Using A Virus”

Had the virus been released as planned on Saturday, the Justice Department said the disruption could have cost millions of dollars and shut down operations for a week a

According to the affidavit signed Jan. 6 by FBI Special Agent Jessica A. Nye, a Fannie Mae engineer discovered the malicious instructions by chance Oct. 29. The virus was removed that day and did no harm, according to the affidavit.

Had the virus been released, “it would have caused millions of dollars of damage and reduced if not shut down operations” for at least a week, Nye wrote.

I’ve lost tthe link; but at the time it was reported that the Fannie Mae engineer found the bomb because, while cleaning up after the gap in security out of curiosity or intuition he/she continued his/her scan of a code printout beyond it’s nominal end…and found the bomb

http://www.huffingtonpost.com/2009/01/30/feds-allege-plot-to-destr_n_162733.html

And Alexander is going toautomate sysadmins

Stephen • August 15, 2013 9:54 PM

@Harvey MacDonald: “If I recall, the NSA stated that many of their sysadmins were performing mostly non-sysadmin work, for example, moving a cache of files from one server to another. These types of jobs could be reasonably delegated to analysts.”

I used to be a sysadmin, albeit not of the NSA, and IMHO your suggestion would be a VERY bad idea, not the least from the point of view of NSA security! For a start NSA analysts and NSA sysadmins would have different skillsets. Your suggestion would be the equivalent of an airline cutting costs by firing 90% of its pilots and having the cabin staff fly the planes instead! (In their spare time, presumably!)

Furthermore, NSA analysts and NSA sysadmins would have different levels of access to the NSA’s various computer systems, just as their equivalents in other organisations do. Sysadmins, by the very nature of their job, require a level of access which, in the hands of those not familiar with the systems in question, have the potential to cause catastrophic loss of data.

For example, under Unix and Linux systems there is a special account called “root” which sysadmins sometimes need to use. Under root-level access it is possible, with a single command (eg “del /.“) to delete an entire file system. This is because certain of the protections against blunders other accounts possess are essentially abolished when using the root account.

George • August 16, 2013 4:20 AM

I bet that some of these agencies are funding research to determine what variables predict probability of whistle blowing as we speak. For example, if you are really worried about losing your job and repercussions because you have a family you really love, are you going to whistle blow? Difficult question. For some people a family may be a reason not to whistle blow, for others it may actually be a reason to whistle blow because one may think of the future one’s children will be leaving in. But there may be another variable (e.g., age, what movies you like and what now) that may predict well which of these people with a family will go one way or the other. The good news is that there aren’t that many whistle blowers, and so there aren’t many data points to build these predictive models of who is likely to whistle blow.

Dirk Praet • August 16, 2013 7:30 AM

@ anon

It might only be one informant or it could be a lot more

We can’t be sure, but I would suspect the DoJ to have done a reasonably thorough job researching any case that even remotely would have been usable in its testimony against Bradley Manning. A while ago, Gen. Alexander told a House committee that their surveillance had foiled about 50 terrorist plots as to make the best case he probably could. It soon turned out that number was grossly exaggerated, but it does prove that government officials will go through great lengths to come up with anything justifying their positions on a particular subject matter, unless of course there is really nothing to show for.

Scott "SFITCS" Ferguson • August 16, 2013 8:09 AM

My experience as a systems administrator is that roughly 10% of the time is spend analysing ad hoc tasks to determine if they should be automated, and at least another 20% of the time fixing tasks that were automated but are not working as planned (plans changed or wrongly planned/implemented). Much of the remaining time is spend dealing with unplanned messes (that includes unplanned or insufficiently planned “add-ons”).
Most “automation” is the result of pre-rollout planning, whether that’s desktop, mid-range, or back end. Once a system is in place there’s not a lot more automation than can be done. Some, certainly, but not 90% – not even close.

Of course there are those that will say I (and other sysadmins) are replaceable. The day “those” people are not sales reps or managers trying to justify their position is the day I’ll take them seriously. Yes, a lot (maybe 20%) of our time is spent doing “nothing”. If “nothing” is reading and research. See W.I.M.P. or W.I.S.C.A. syndrome. Proof that System Administration is a non-intuitive task and ignorance doesn’t stop morons from trying to profit from their greatest skill.

I strongly suspect the NSA is over-funded, under reviewed, and should be made into a musical. I’ve got an idea for the signature tune – something like:-
♫ ♩ Hand me a shovel and pick, the bullsh*t in here is too thick…

@Bruce Schneier
Many thanks for your continual, insightful, articulate, intelligent, brave, and principled writing. I’m sure you could make more money and less enemies being less of all those things.

Clive Robinson • August 16, 2013 9:08 AM

@ Scott,

    ♩ Hand me a shovel and pick, the bullsh*t in here is too thick.

Just does not sownd right try,

    Hand me a shovel and a pick, the bullsh*t here is way too thick.

When I was a lot lot younger Disney Corp as now is produced one of the first full length animated films “Snow White”. In it was as song sung by the dwarves as a work song. For some reason the words got changed in the playground to,

Hi, Ho, Hi, Ho, it’s off to work we go,
With a bucket and spade and handgrenade
Hi, Ho….

For some strange reason 😉 it’s stuck in my mind for about half a century, and lets be honest a handgrenade would be better at assisting the bovine excreta hit the fan and I suspect a lot more satisfying to use 🙂

Scott "SFITCS" Ferguson • August 16, 2013 9:21 AM

Stephen

Sysadmins, by the very nature of their job, require a level of access which, in the hands of those not familiar with the systems in question, have the potential to cause catastrophic loss of data.

Yes, though good sysadmins backup first.


For example, under Unix and Linux systems there is a special account called “root” which sysadmins sometimes need to use.

Again, yes, but permissions are more fine grained than just low level users and root. i.e. backup groups, admin groups etc. Properly root is only used when no other authority will suffice.

Under root-level access it is possible, with a single command (eg “del /.“) to delete an entire file system.

No. You are confused, misinformed, watched the wrong movie, what ever.
That’s a DOS command. Even on DOS it will give you a warning of what is happening, plenty of time to stop it, and still leave you with the ability to undelete the files.

rm -r ./ > nul might work. But you should read up on things like SE (NSA) Linux, App Armour and other relevant subjects before over simplifying examples.
In many cases (even small companies who don’t pay special attention to security) cat ~/.bash_aliases | grep rm will give you alias ‘rm=”rm -i”‘ which prevents the, um, scenario you describe.

Scott "SFITCS" Ferguson • August 16, 2013 9:33 AM

@Clive Robinson

…try,

Hand me a shovel and a pick, the bullsh*t here is way too thick.

Right as per bloody usual. I’m saddened, but not so sad that finding a spelling mistake in your correction negates the fact I’m wrong (or a less of plagiarist wanna-be musical writer).

I was wondering what the tune was though. Thanks for that. Now I’m trying to remember all the lines we had for it in my schooldays (when Moses was the orange boy on our football team and Walt was still alive).

So much for my dreams of Broadway and a brownstone in Manhattan, back to the keyboard and cli (sigh).

🙂

Anon • August 16, 2013 7:11 PM

@Dirk

I’m not sure that your example of General Alexander is directly comparable to the Manning trial. NSA public testimony before Congress is a PR campaign. The Manning trial is a criminal trial, where the typical burden of proof is “beyond reasonable doubt”. I’m not a JAG lawyer, but if the judge required that standard, it could be an almost impossible threshold of proof. I’m not as confident as you are that DOJ would have done a thorough job of researching anything that would be remotely damaging to Manning. I haven’t followed the trial closely, but what stuck with me was that one of the government agencies read less than half of the leaked documents in conducting a damage assessment. They claimed they simply didn’t have the staff to read all the documents Manning leaked. If the government can’t bother even to read all the leaked documents, what makes you think that they’re going to send FBI agents to some remote Afghan village to see if Ahmed listed on page 437, 568 of wikileaks is still okay?

Erewhon • August 18, 2013 3:59 AM

In order to automate computing procedures one has to write code.

Before one writes code, it is recommended that one defines some requirements, produces an architecture which will satisfy the requirements, and then produces a design which ensures that the suggested architecture meets the requirements.

During the implementation (writing the code) it is recommended that one executes test procedures to verify that the implementation is meeting the requirements.
(Requirements-based testing.)

The difficult bit, is proving that the requirements are correct.
(Requirements testing.)

The even more difficult part, is proving that the people who wrote the requirements, and the people who produced the architecture, and the people who wrote the design, and the people who implemented the code, and the people who performed the verification of the implementation, did not introduce mechanisms to circumvent the proposed “saving sysadmins” automation.

If the NSA were to use robots to perform the requirements/architecture/design/implementation/testing, then all this would of course be a non-issue.

Snarki, child of Loki • August 19, 2013 8:26 AM

Yeah, he said “eliminate”. Couple of possible meanings there, only one of which requires updating the ‘ol CV.

How to get to 90%?

  1. No more backups! They just waste time and effort, and are a huge pile of data to be secured.
  2. No more software updates! Hey, it’s all working fine now, so why mess with success?
  3. No more hardware fixes! It’s all under warranty, right?
  4. Anything else needed? Outsource! Offshore!

The first step is to replace the NSA Operational Procedures Manual with “The Dilbert Omnibus Collection”. What could possibly go wrong?

Ade • June 1, 2015 10:36 AM

Same budget, so instead of firing them, retire them, same could be done with automation. Instead of laying off workers, retire them on half to full pay.
Of course this will mean the extra profits going to these workers instead of shareholders so such an idea is likely to meet with some resistance, but there’s no need for automation, robots etc to cause people to worry. 90% of work could be carried out by robots meaning we all get more time at home with our friends and families instead of being slaves to the system.

Don Lunsford • October 23, 2015 7:49 AM

When the process is automated, who are we trusting to automate it. Are we going to use software from China? You need to hire people that can be trusted and that doesn’t mean people with degrees but people that believe in doing the right thing for their country and hopefully technically qualified as well. You also need for the people that do the hiring to be honest and trustworthy.

Leave a comment

Login

Allowed HTML <a href="URL"> • <em> <cite> <i> • <strong> <b> • <sub> <sup> • <ul> <ol> <li> • <blockquote> <pre> Markdown Extra syntax via https://michelf.ca/projects/php-markdown/extra/

Sidebar photo of Bruce Schneier by Joe MacInnis.