The Concerted Effort to Remove Data Collection Restrictions

Since the beginning, data privacy regulation has focused on collection, storage, and use. You can see it in the OECD Privacy Framework from 1980 (see also this proposed update).

Recently, there has been concerted effort to focus all potential regulation on data use, completely ignoring data collection. Microsoft’s Craig Mundie argues this. So does the PCAST report. And the World Economic Forum. This is lobbying effort by US business. My guess is that the companies are much more worried about collection restrictions than use restrictions. They believe that they can slowly change use restrictions once they have the data, but that it’s harder to change collection restrictions and get the data in the first place.

We need to regulate collection as well as use. In a new essay, Chris Hoofnagle explains why.

Posted on September 12, 2014 at 6:41 AM53 Comments


vas pup September 12, 2014 8:22 AM

Good article. “Unreadable privacy notices and the inconvenient choice mechanisms were created by the very companies that want to encourage data sharing. Technologists designed systems to make privacy impossible, and now they say it is “pragmatic” to accept their legal proposal to solve the privacy problem.” This part is in particular relevant. Deregulation of big business (any type including data collection, banks, financial sector, etc.) never ever served interest of the general public average ‘Joe’). Time and again, Government generated regulations are the source of counterbalance of big business power and have to set up m i n i m u m level of protection. After that any business may implement more protection only as their way to be more competitive for customers.

David Leppik September 12, 2014 9:12 AM

It’s easy to explain to the public why this is bad. Just think of Home Depot. And Target, which despite a great deal of effort couldn’t keep attackers from breaking in through the ventilation system.

Glarm September 12, 2014 9:38 AM

Well that article was soul-crushing to read.

I don’t know if I should give up or not. The more I learn about how systemically our privacy has been violated, the more despondent I become. I have no idea on how to fight back in any meaningful way.

Is there anything at all that is cause for hope?

Bob S. September 12, 2014 10:02 AM

Re: “I don’t know if I should give up or not.” ~GLARM

I second that thought.

I just spent a few hours going through Windows 8.1 pro trying to figure out why MS needs to be making thousands of connections to my computers 24/7. I found OneNote, even though disabled and never used, connects all the time, so does Windows Mail, Windows Defender, Windows location services (????) and so on.

You can RUN gpedit.msc/Administrative Templates/Windows Components/double click component/double click settings and shut some of it off, but still MS is ALWAYS THERE.

That’s one company sucking up, apparently, every conceivable data point generated and then COLLECTED. Then try to read the privacy statement, summary: “You don’t got any.”

Why is Facebook ALWAYS in my face? (Latest gripe: Bing auto connects you to Facebook, regardless.)

I suppose the trick will be to somehow retain some of the conveniences while shutting down everything else (including comments to blogs like this.) Or, pulling the plug.

I am thinking about it.

phred14 September 12, 2014 10:36 AM

Perhaps the way out of this quandry is to lie. Since they’re so eager to gather information about us, perhaps we need to start giving it to them. I wonder what kind of target ads I’ll receive once they discover that I have a Nobel Prize for thunmb-twiddling, or my Oscar for Best Supporting Geek in a Kernel Crash Role. Not to mention my fly-fishing trip to Lake Bonneville, or my golf career on Gondwanaland.

The suggestion has already been made to lie on password recovery questions, because the true answers can be discovered.

One potential problem with making up lies about yourself to flood the net with is that you don’t want any of them to be criminal activities. That might be bad.

jones September 12, 2014 12:07 PM

I started reading Craig Mundie’s article and a few things jump out at me, specifically relating to the assumptions behind statements like these:

“But those fears and the public conversations that articulate them have not kept up with the technological reality. Today, the widespread and perpetual collection and storage of personal data have become practically inevitable. Every day, people knowingly provide enormous amounts of data to a wide array of organizations, including government agencies, Internet service providers, telecommunications companies, and financial firms”


“This is the reality of the era of “big data,” which has rendered obsolete the current approach to protecting individual privacy and civil liberties”

First, this is a fatalist attitude. It says we have to adapt to technology, and ignores the fact that technology is the product of industrial corporations that can be rationally controlled.

Corporations are legal entities and can be constrained according to whatever principles we choose. We happen to live in a time when most countries, regardless of political orientation, choose to promote growth policies, but there is nothing inevitable about this.

This type of fatalism sort of reneges on the promise of Enlightenment Rationalism, which created modern science and which proposed that rational means can direct human affairs. To say that humans must be subordinate to things, that humans have to adapt to technology, really puts the cart before the horse. We shouldn’t all have to give up our privacy because a few people want to profit by violating it. Of course, there is always to option to opt out — close your facebook.

I would also take issue with Mundie’s use of the term “knowingly.” I think people are utterly clueless about how much data is being collected. So much data is being collected that researchers can only make sense of it with algorithms and computer clusters. It literally exceeds human comprehension. To say that non-technical people are “knowingly” a part of this is at the least disingenuous.

ciphertext September 12, 2014 12:28 PM

Data collection used to be expensive. Especially when it required that a person actually go forth and obtain that data. In the current environment, groups / persons, who are interested in obtaining data simply need to wait for the data to come to them. The amount and type of data that is following this inverse pattern of collection is increasing in both scope and volume. In the “good old days” usually only certain agencies (e.g. MD, Law Offices, Financial Institutions, Educational Institutions, etc…) could depend upon data “coming to them”. Also, the type and volume of that data varied across those agencies. Also, the ever present governing agencies could demand disclosure of data for collection (e.g. DMV, Selective Service, Census, IRS, etc…). The examples I’ve given are for the United States specifically, but I have little doubt that other, foreign entities would emulate this pattern to lesser or greater degrees.

The trend is that data collection is becoming both cheaper and more prevalent. Cheaper because the activities associated with collection have become more simple (e.g. identifying data elements, identifying where data elements reside, identify best method(s) to obtain data elements). Indeed, in some instances entire businesses have been developed to provide access to a data set for a fee. Additionally, the costs of storage have gone down (both computer storage and the ongoing elimination of paper copy storage). The collection activities are more prevalent due to the realization that effective data analysis can improve sales figures and also more effective and efficient (though not more agreeable to the regulate) regulation of behavior.

It appears to me, that a lot of the techniques used by “organized” groups (e.g. national governments, agencies, cartels, etc…) to hide themselves from collection activities are going to become required use by the general public. Assuming the general public is interested in not having data collected about them. Techniques such as: disseminating disinformation; masking their online behavior; using aliases; minimizing their “paper trail” for financial transactions (e.g. use cash, use preloaded visa cards, bitcoin, etc…).

Nick P September 12, 2014 12:29 PM

@ Bob S.

Good points. The solution with stuff like Windows might be efforts by volunteers to hunt down and disable that stuff. A friend of mine who used to post here followed some guides on trimming fat out of his WinXP box by straight up deleting files. He had an office suit, Firefox, etc. All the software he needed plus some documents. He’d keep deleting stuff and testing the machine, reverting only when the delete caused problems. The resulting functional, office-equipped WinXP box was under 600MB. It was backed up on one CD. I was impressed.

I’m sure there’s guides for that kind of stuff on Win7. People could join such efforts to further identify which components were making external connections and just delete them. The few that were critical could be handled by people wanting to show off their hacking skills. Do a little disassembly to identify & kill the part that connects. Or force it to go through a firewall that blocks it by default unless you allow it, maybe optionally logging whatever it sends.

Stuff will continue to get worse if one is buying from selfish corporations, esp if they have ad revenue. Best to just avoid depending on such firms where possible. We do have options, though, even when we depend on them.

EvilKiru September 12, 2014 12:35 PM

@jones: Industrial corporations have some of the most irrational behavior you can find on this planet and we have less control over them than we have over politicians. Claiming otherwise makes you almost as big of an industrial corporation troll as anonymous.

BoppingAround September 12, 2014 1:19 PM


We shouldn’t all have to give up our privacy because a few people want to profit by violating it.

This. Mostly I hear `privacy is dead’ talks exactly from those who feast and profit on killing it.

I think people are utterly clueless about how much data is being collected.

I concur. And the `read our privacy statement \ TOS’ won’t help. It’s barely comprehensible to someone without law degree.

Bob S.,

Maybe you’ll want to skim through this:

David Henderson September 12, 2014 1:23 PM

The book “Dragnet Nation” is written by a WallStreetJournal staffer and describes :
how extensive the data collection is,
just how paranoid some individuals have become,
the author’s own moral reasoning to set up a second identity to shield her professional activities
e.g. – she wants to protect her sources
some of the tools she has found work, and many that dont work

Well worth a read.

Nova September 12, 2014 2:31 PM

What stands out the most to me when getting into this is that there is no way to police how data is used.

The paper highlighted it this way:

In another wrinkle to the use-regulation landscape, under the PCAST framework, “analysis” of data is not considered a use.

But consider the recent Snowden revelations that analysts were looking at nude pictures of Americans, or the claims that Mark Zuckerberg of Facebook used the service to predict who was about to end a relationship.

How, exactly, do you police this? Consider, this is not being policed. Did you see anyone get fired from these allegations, or even any statements that they would investigate? Or that Snowden was just one guy, and it looks like the only one who has come forward over such abuse. You take Snowden as a statistical sample, and you can probably rightly assume that this is epidemic level.

Worse, so what happened when the Snowden case came out? A regulator went and asked the NSA about examples of cases they had where data was abused. They came back with something like 12 or 13 cases. All cases where someone used their position to illegally access data pertaining to a spouse or lover.

You have to be an idiot to seriously believe those are the only cases that have happened. It is like a grade schooler fudging to their teacher after being caught at doing something wrong. Statistically, it is not only unlikely that abuses are ongoing, but that there must be far more and of a far more varied sort.

A far more disturbing sort. Of course they understand people would go “meh” on “loveint” cases.

There is no effective self-policing — not even that agencies should self-police themselves.

These are all signs of a very sick system.

I realize there are some defenders to these things. Their defense, its’ self, is indicative of a sick system. By default, you do not trust people just because they are on the same team or work for the government. That is the epitome of bad security, and it surely is bad intelligence.

The posture taken is a posture you see with corrupt cops.

Alan Kaminsky September 12, 2014 2:41 PM

@Glarm I don’t know if I should give up or not. The more I learn about how systemically our privacy has been violated, the more despondent I become. I have no idea on how to fight back in any meaningful way. Is there anything at all that is cause for hope?

Sure. Just don’t use Windows, MacOS, Android, Google, Facebook, Twitter, smartphones, etc., etc. Back around 1980, we didn’t have those things, and we were quite happy, AND our personal information stayed private.

I’m not being facetious or sarcastic. I believe it is perfectly possible to live and be happy in 2014 without all those things. I use Linux and DuckDuckGo, not Windows and Google. I don’t have a Facebook, Twitter, iTunes, or iCloud account. I’m about to ditch my smartphone and go back to a plain old cellphone. Stores still accept cash; you don’t have to use a credit or debit card. The only thing I’d miss is ordering stuff online; but I don’t do much of that anyway, and I could always get someone else to order the stuff for me.

As I’ve said before, we high-technology folks could do worse than to take a page from the Amish, who manage to live happy, productive lives without all this stuff we think we “need.” I strongly suspect that Microsoft, Google, and the rest have collected no data from Amish people.

jones September 12, 2014 3:01 PM

@Alan Kaminsky

You’re right. We can to a considerable degree opt out of the surveillance state simply by ceasing to surveil ourselves. It’s no more complicated than adopting a “vegetarian” approach to media consumption and a lot more effective than clicking an internet petition to end internet surveillance.

smith September 12, 2014 3:23 PM


Opting out of media consumption online is one thing you can do.

It’s important to realize that the “media” broadly interpreted is simple the middle ground between end points.

The media between home and work is your commute route, and there you are exposed to impressions on the signs, trucks, and bumper stickers along your way.

By randomizing your path when you travel, you can avoid being predicted for personalized ads or conditioning. It also varies the figure/ground relationship of your life such that it becomes harder to target you for repeated stimulus against the same background. File this away in case you ever find yourself in the category of being a person of interest.

BoppingAround September 12, 2014 3:29 PM

David Henderson,

If you (or anyone else here) have read this book, I’m curious about one of the statements. How exactly paranoid have become these certain individuals?

Nova September 12, 2014 3:33 PM


Well that article was soul-crushing to read.

I don’t know if I should give up or not. The more I learn about how systemically our privacy has been violated, the more despondent I become. I have no idea on how to fight back in any meaningful way.

Is there anything at all that is cause for hope?

Governmentally: people involved in bad things play the game that they are smarter and more powerful, more capable then everyone else. While there is much evidence ‘the bad have success and die honored’, there is also much evidence that criminally minded people play a game which they will lose at.

These are not new problems. While I speak critically of foreign intelligence programs relying on deep cover agents in other countries for first world countries, I do not hold this opinion for them within their own borders. First worlders are not eager to become human ghosts and enter the equivalent of ever shifting Witness Protection Programs, set to live in nasty locales and foreign countries. Further, there are massive problems with even trying to be an “illegal” in such countries.

That equation is meaningless within these countries.

You can expect there are probably off the books programs that are separate from other agencies which can properly investigate… and handle even serious and rampant corruption.

… so like in so many matters, act and behave lawfully, and do not try and extend your own position…

But, otherwise, it depends on your skill set, or wanted skill set. There are countless legal, ethical things you can do. It all starts with education, research. Get yourself and keep yourself educated, and you will be bound to find lawful, ethical methods of dealing with these issues.

Personal security wise, really, the same thing.

Last digit in pi September 12, 2014 3:46 PM

Hey, no Friday squid news, and Burger King has a “bamboo charcoal cheese and squid ink sauce” burger in Japan! Yes, the ketchup is colored black using squid ink.

And now for something completely different…
How would you like to get your tentacles around the Microsoft support scammers? Matthew Weeks developed an exploit (Exploiting Ammyy Admin – developing an 0day) on Ammyy Admin, because it’s what the scammers are using to log into the victim’s computer and mess things up. Pretty clever, but you have to have it set up before the scammers come calling.

EvilKiru September 12, 2014 4:00 PM

@Alan Kaminsky: I don’t think you can get a dumb cell phone anymore. When I dropped my dumb flip-phone and broke it a few years ago, the cheapest replacement flip-phone I could find had an Internet button…

albert September 12, 2014 4:25 PM

@Bob S,

Did you see the Simpsons episode where Bart keeps touching the stove and keeps burning himself?
Stop burning yourself!
Switch to Linux. Abandon FB, Twitter, Microsoft in all forms. Use email groups or (gasp!) telephones to connect with folks. Forget texting. (Ever ask yourself why it’s so popular?*) Sell your iPhone, and get one that handles voice traffic only. Windows is the least secure of ALL OSs. Why do you think Apple switched to Linux?
Use cash ( or checks) instead of a credit card. You’ll spend less. You can keep a debit card on a low balance account. Don’t use ANY cards on your main personal account (where the big bucks are). Don’t use the internet for medical related things. Insist on paper forms.

There are other, more extreme(but still legal), measures available.

Americans don’t realize this simple but amazing fact: We have the power to change society. It’s in our wallets (or purse)! It’s economic power, which is the only thing the Corporatocracy understands.

I gotta go…

    1. It’s absurdly cheap for the telcos. (much less bandwidth than voice)
    2. It’s trivial to decode; great for data mining.

Nova September 12, 2014 4:46 PM

@Bob S

As in person, it is wise to be cautious with what you say considering the possibility of critics, likewise, online you should do this. Never think you are anonymous.

Though it should not need to be said: act lawfully, and try hard to be as non-hypocritical in your beliefs as possible. Try to be as accurate in your beliefs as possible. This way, what you say, as it comes from your beliefs, won’t be able to be used against you.

(Contrast to the idea of “just lie”, the truth spills out, so not a good standard of practice.)

Assess your self, as an analyst would assess the security of a system: What do you say or know that an attacker may want to get. What data might you keep or put out there that an attacker might find valuable? Who is your likely attacker? How can you value that data from their perspective?

From there, you can measure how much security you want to personally pay for, versus how much usability. There is a ratio there. And I mean “pay” as in “overall level of effort”.

Understand there are psychological factors that can bend assessments here: for instance, people have a tendency to think they are much more obvious and have attention paid to them then they are. If the value of your “data” is really, like “five bucks”, metaphorically speaking: you are over stressing.

That is about all I have to say, noting not any specific approaches, but the overall larger gameplan I think you should keep in mind.

Anura September 12, 2014 4:49 PM


“Forget texting. (Ever ask yourself why it’s so popular?*)”

Because it’s simple and more convenient than phonecalls or email (especially since they were around before smartphones)?

Glarm September 12, 2014 7:10 PM

@Bob S. & @Nova

I am not particularly asking what can I do to try to protect myself – that only goes so far when the very infrastructure our traffic flows over is tapped and when more and more of our everyday lives is forced to flow over those cables.

What I want to know is – is there any reason left to have any hope that mass surveillance of our communications will be stopped or have some kind of strong, meaningful restriction placed on it?

Nick P September 12, 2014 8:04 PM

EvilKiru has a point: I was trying to find a counterexample and couldn’t do it searching stores like Walmart. I had to search Google instead for several minutes to find this:

The Nokia 105 has the drawback of “no browser.” I know there’s some encrypted phones that are just phones, as well. They’re not cheap. Best bet for people looking to do that is to Google “cheapest prepaid cellphones” with a filter for the year. Start at least 5 years back, looking at each one to see if it’s dumb enough. Repeat until you find one. My research gives you a shortcut: just type Nokia 105 into eBay.

Nick P September 12, 2014 8:14 PM

You can also flash an Android ROM (with source available) onto a phone. Then, disable networking via the firewall and other leaky services. You can even remove a bunch of features and apps via online guides or source modifications. Just make a dumber version of it. The Cryptophone people did stuff like that with Windows Mobile to harden it against attacks. They’ve recently done it with Android, too, so the method works.

Far as services, best hope is to use paid, privacy-focused services located in places like Switzerland*. The service must be paid so you are the customer rather than an advertiser. The service should have a policy of not sharing your data. Not keeping it is even better. From there, the law and courts provide protection. This kind of scheme worked well for the rich people until Wikileaks came along.

*Note: Make sure their servers are actually in Switzerland. Many, including Kolab, locate servers stateside for performance I figure. That puts them in U.S. TLA jurisdiction on networks they control. Physical location in Switzerland or whatever country you use is a must. Servers in countries big on SIGINT should only be used as untrusted, but useful, 3rd parties.

Nick P September 12, 2014 8:24 PM

@ Anura

It’s called instant messaging. It’s been around longer than texting. It’s just as easy with a good client. I kept pushing my friends to use that as companies were ripping people off with texting fees. I saw hardly any takeup but the WhatApp user base shows it can still work today. Blackberry Messenger, too. Just need one with security baked in that’s free or dirt cheap.

Note: I know there’s some apps available, esp made by volunteers, but they’re not The Right Stuff that will displace Facebook Messenger or texting. Hemlis Messenger has a nice start by focusing on the interface and aesthetics. Has potential.

Anura September 12, 2014 8:34 PM

@Nick P

SMS is still more convenient as it doesn’t require a separate account. Plus IM wasn’t widely available before smartphones.

Nick P September 12, 2014 10:20 PM

@ Anura

It’s definitely more convenient. It was widely available, though, as AOL, Microsoft, and ICQ messengers had around 100 mil. users by 2006. Available in the late 90’s to the millions of people who had a Windows PC and basic dialup internet. It makes sense for texting to pick up on mobile since IM wasn’t there early on. IM of various forms started appearing in years after smartphone release. (Blackberry Messenger did take off huge in 2010.) People still had their old accounts. A setup would’ve taken a short time, then auto-signin means the IM client could be about as easy to use as text without its limitations and with stuff like file transfer.

So, my analysis is that IM client designers did a bad job, the users made costly (short- & long-term) choices for small amounts of convenience, or both. Fortunately, a few modern messengers show people are making better choices and that competition has pushed a number of wireless providers to offer free texting with inexpensive plans.

Nova September 12, 2014 11:01 PM


I am not particularly asking what can I do to try to protect myself – that only goes so far when the very infrastructure our traffic flows over is tapped and when more and more of our everyday lives is forced to flow over those cables.

What I want to know is – is there any reason left to have any hope that mass surveillance of our communications will be stopped or have some kind of strong, meaningful restriction placed on it?

Heck yeah you should have hope, heck hope is my middle name! 😉

It is true, the situation does look… a little grim. You have a President who came in on a platform of “hope” and “change”, and he kind of smashed a lot of hearts there. If you have looked at the history of nations, you would have a hard time not noticing that totalitarian regimes are by very, very far the norm. Freedom, liberty, rights… outside of obscure tribes that mostly no longer exist… has been a recent invention for nations. And it is not likely they will keep it anymore then it is likely a parolee won’t commit a crime again, or a sex addict get off sex…

But maybe these kinds of challenges and impossible odds is just what life is really all about?

What are movies without villains, stories without adversity? Maybe adversity simply exists to make our lives a little more meaningful, and give just a few people a chance at being virtuous in this depraved, limit bound world?

But, hey, maybe you aren’t looking for a pep talk, or some ‘Don’t Worry, Be Happy’ Philosophy 101…

Now, as well prescribed in Dr Strangelove, Or How I Stopped Worrying And Learned to Love the Bomb… why don’t you go get yourself some rainwater and grain alcohol and take it easy, pour yourself a drink. 🙂 🙂

JonKnowsNothing September 13, 2014 1:51 AM

Until the rest of the tech world understands that every level of the OSI model is tainted and corrupted and that to fix this requires massive a undertaking and a complete “rethink” of how things are to work, all options mentioned are just Band-Aids. Some options may help a bit but underneath the infection gets worse and the corruption grows.

There’s no easy fix. No phone anywhere is going to prevent you from being surveilled, even if you do not have one, someone near you will and you will be hoovered up just the same. Consider sitting in a waiting room, even without your phone others have them and you get caught in the background noise.

Changing OS may help some but there are other layers of the OSI model where your access is already picked up. Device drivers are corrupted easily and reflashing them with bogus updates will make sure your system is in compliance with those who want to get to know you better. You will never know if the devices on your system haven’t been tampered with before or after you got the device.

It can be changed and altered but not under the current system and not under the current “management”. It can be done but it will be painful and any attempt to alter the existing system will be met head on and opposed with all the power of the state apparatus behind it.

First though, you have to get other tech professionals to even acknowledge there’s a problem. The elephant’s been in the room a goodly while now, decades even. There’s nothing that can be done while people pretend there’s nothing to do.

name.withheld.for.obvious.reasons September 13, 2014 2:42 AM

@ JonKnowsNothing

Until the rest of the tech world understands that every level of the OSI model is tainted and corrupted and that to fix this requires massive a undertaking and a complete “rethink” of how things are to work, all options mentioned are just Band-Aids.

I’d argue Jon that Layer one and two may be safe (vendor implementations aside), three becomes problematic from a management perspective and layers four through seven can be solved by hiring SOFTWARE ENGINEERS, not programmers.

Layers 1-3 also require HARDWARE and SOFTWARE ENGINEERS!

name.withheld.for.obvious.reasons September 13, 2014 3:10 AM


This is all too true–the FCRA is the biggest farce perpetrated on the public in several ways and started going crazy in the 1980’s. It went from a credit report to (knowing what choicepoint/etc do) to a grab bag of anything and everything.

Experian/TransUnion/EtAl don’t maintain credit reports–YOUR DOSSIER IS MAINTAINED–and there is nothing fair (from the CONSUMER/citizen perspective) about this process!!!

Data can be tagged to you from non-qualified sources (hearsay essentially) and others can access without “recording” an event entry–the government is anonymous in this case…and the list goes on and on. The government oversight board at the FTC completely missed issues related to the due diligence requirements of FCRA when it comes to brokers and data collectors.

I personally struggled with this matter for over twenty years–gave up two years ago when talking a data broker “support” person and was asked:

“So, where have you been the last two years?”

I hung up the phone. Not only has the company managed to repeatedly record and accept erroneous transactions, accounts, and other data–they wanted to know “what I was up to?”–the last straw in a long line of insults and injuries. The data brokers, as mentioned, are at the forefront of the BIG DATA/LITTLE SPY cabal and have a long history.

Gerard van Vooren September 13, 2014 3:43 AM

@ name.withheld.for.obvious.reasons

I understand what you are saying. The problem is, even with the right engineers, how many LOC does it takes to implement the standards correctly?

My suggestion would be to solve the problem at the core. But then you have to question what the core actually is.

Besides that, installed base, legacy and that kind of stuff is a big enough rope around the neck of elegant ideas. But in the end it’s elegant solutions that can bring sufficient safety. The playing field of forces however wouldn’t allow this to happen. Not in a US centric world.

Clive Robinson September 13, 2014 5:13 AM

@ NWFOR, Gerard van Vooren,

The “center” is where the communications actually takes place (ie lowest layers) if this is not solid and secure nothing you put on top will.

At this low level there are two models that are common “Broadcast” and “Point to Point”, of these two it’s actually P2P that is considered most resource efficient. But P2P has to leak information to get that efficiency, and thus as these are “engineering” not “security” solutions [1] all our network connections are insecure from the get go…

Oddly though “Broadcast” is considered by most to be the least secure as “all can hear it”, whilst the reason is true the conclusion of “least secure” is not.

The use of self synchronizing ciphers at the lowest level renders the “all can hear it” point mute. And other techniques built on this such as message and broadcast stuffing render traffic analysis mute as well.

Oddly perhaps Ethernet, being fundamentally a broadcast model was a step in the right direction, but “engineering” won out over “security”, perhaps it’s time we “re-visited” these design choices.

[1] I have a meme I trot out occasionaly which is “Security -v- Efficiency”, which is similar in many ways to the more common “Security -v- Usability”. The general implication being the more of one you have the less of the other you have. Whilst not strictly true the mindset and effort involved to go against these memes is seldom to nearly never ever found. The usual reason for this is “cost” and “time to market”, or more simply “Shoddy 5h1t, gets market capture”…

name.withheld.for.obvious.reasons September 13, 2014 7:11 AM

@ Gerard van Vooren, Clive

I understand what you are saying. The problem is, even with the right engineers, how many LOC does it takes to implement the standards correctly?

Going to go out the box here–split the transceiver ASIC/FPGA’s with a dedicated LAYER 1 piece of hardware ASIC ONLY!. Split layer 1 and 2 logic and with the fabric using another ASIC or FPGA (don’t like the FPGA). One layer three integrate the hybrid/split layer 1/2 pairing with a GPU that can be ASIC/FPGA or GPU.

As a big fan of simple languages such as M/ASM I’d estimate the first layer to exist primary in ASIC layered code, layer two I’d prefer ASM but a lot of big iron needs layer two to sit in FPGA’s–but I’m not opposed to a fast simple RISC processor (even consider a transputer) with ASM. But I’d still argue ASIC for layer two. The next layer, three, could and often does sit in GPU’s (some of the GPU’s are FPGA’s built as GPU’s) and this is exactly where a nice ASM core could work well–I’d estimate, without MLPS/MLS/SPT/VLAN type of network code (is a layer that integrates layer 2 and 3, about 300,000 LOC. Strings would be tables, etc…large data section. Not data to code translation (ASIC can fix that) but would be trickier in layer 3 model that supports something like Software Defined Network. With just the 2/3 layer with logical support add another 500,000 LOC’s (without the GPU support code.

To summarize the idealized:

Layer 1: ASIC (transceiver, connection management)

Layer 2: ASIC/FPGA (logical asynchronous integration of transceiver, media management, and switching/data operation)

Layer 3: ASIC GPU, small network kernel based on ASM (OS/9 comes to mind or a microkernel–many available) handling layer three internet-working and portions of the logical functionality.

Upper Layer (filtering/QoS/etc) could be supported by Layer three design, supports internet protocols, management, and some logical constructs.

As many are aware a Cisco switch can be easily configured as a router, but, a Cisco router will not work as a switch. Cisco, Juniper, F-5, and other advanced switches share MPLS/VLAN/Logical Network layering and continue a significant amount of code for supporting these protocols (here’s where the real engineering needs to be managed–like IEEE 802.1Q).

If all the ASIC code is VHDL up to and including layer 2 then we could see a 750-900 KLOC VHDL core (minus the IEEE and misc. VHDL library code). Add the GPU for layer 3 and we have another 500 KLOC.

ASIC/ASIC 1,500 KLOC (not including IEEE and other VHDL libraries) and probably a near 1,000 KLOC GPU core (NOT including GPU core). Full modern CISCO/JUNIPER like device would add probably 20-30 percent more overhead.

Just a WAG–but a starting point.

BoppingAround September 13, 2014 9:01 AM


Maybe. There is no certain answer. Slavery and tortures are largely beaten. Maybe one day these omnivorous systems of electronic enslavement will follow.

Nick P,

Why Switzerland? IIRC, they have data retention laws in-place. Isn’t that a con?

Sancho_P September 13, 2014 10:11 AM

@ Nick P:

Switzerland … ouch. Not better than EU, they depend on both, US and EU.
I’d suggest Russia. Thanks to our imperialism (or national capitalism = “Naci” behavior) they are nowadays an isolated haven e.g. for VPN services.

But the data collection (and abuse) is done locally, I think it doesn’t matter where your phone provider is located (it would be costly, though, to have it in Switzerland).

All local authorities and businesses collect and share our data, we can’t stop that as we can’t neither go back in time nor change our economy paradigm (growth).

Nick P September 13, 2014 10:35 AM

@ name.withheld

It sounds like you’re overcomplicating the design. The Plasma project’s whole TCP/IP stack was 1,800 LOC. IwIP’s compiled code fits in 40KB. Basic routing wouldn’t add much more. Security checks & proper error handling might make it considerably larger.

However, the best route for network processors is the dataflow model. This was discovered years back and I just recently discovered it. A number of papers I posted here mapped TCP/IP processing to a dataflow engine. Each step is its own processing element. Progress happens every cycle. All happens in parallel where possible. Efficiency is great and dataflow programming has plenty of good tools. The most efficient, performing, TCP/IP engines use dataflow with either individual programming elements (eg PISC) or many simple RISC cores (eg Cavium).

That said, if you are adamant about a monolithic design with plenty hardware there’s options. The commercial market and website both have I.P. for a number of components needed: Ethernet controllers, checksums, crypto, RISC processors, etc. (Plasma MIPS CPU is public domain, btw.) One could acquire those, inspect their design/code carefully for subversion, and integrate them into a new solution. Such a SOC would be way easier to build than a solution trying to integrate ASIC’s, GPU’s, etc.

Link to my post with the dataflow processors:

Nick P September 13, 2014 10:47 AM

@ BoppingAround, Sancho_P

Switzerland has strong privacy laws that mandate the private data doesn’t come out without a court order. They often ignore foreign supeonas and would wipe their ass with national security letters. They do have a certain dependence on the U.S./E.U. but powerful people in U.S. depend on them for those secret cash flows. 😉 That they don’t just roll over is evident by the long fight between U.S. and Switzerland over American tax dodgers. They eventually just adopted the practice of denying Americans accounts to spare them headaches. They did cooperate with U.S. authorities for drug or terrorism investigations.

So, the Swiss situation is complex far as working with U.S. They do resist plenty. Their baseline for privacy is also far higher than the U.S. It’s even built into their Constitution and culture. I still reduce risk by designing multi-national solutions where I can.

@ BoppingAround

Let’s say they do data retention. The point of that is you can produce the data on demand by a court for a certain time period. One could encrypt all retained data individually. If they ask for certain data, they get just that data. Anything else stays protected and is deleted along with the keys later on. So, data retention provides a risk but it’s not a deal-breaker.

I’m also envisioning a scheme where a foreign party handles the data and verifies that a court order happened rather than something else. Might provide a nice check against that risk.

@ Sancho_P

“I’d suggest Russia. Thanks to our imperialism (or national capitalism = “Naci” behavior) they are nowadays an isolated haven e.g. for VPN services.”

If the goal is protecting secrets, it would be a very bad idea to put the service/data in territory of mafia and top secret-stealing intelligence service. If Russia is outside the threat model & NSA is in it, I agree they make a good choice. My early posts on hardware were to use Russian and Chinese built hardware for anti-NSA activity, but assume Russians and Chinese were listening. If one could benefit Russians with an operation, they might have the operations back and help block NSA activity. Same goes for the Chinese.

JonKnowsNothing September 13, 2014 11:10 AM


I’d argue Jon that Layer one and two may be safe (vendor implementations aside), three becomes problematic from a management perspective and layers four through seven can be solved by hiring SOFTWARE ENGINEERS, not programmers.

Layers 1-3 also require HARDWARE and SOFTWARE ENGINEERS!

It won’t take long too long to look at the TAO/ANT Catalog (software/hardware/firmware exploits) and JTRIG catalog (mostly software exploits) programs of the NSA (which are no doubt duplicated elsewhere in other countries and with other bad guys) to get a firm grip on the idea that NOTHING is “safe”. The entire connection system from start to finish can and is intercepted/harvested and digested not only by security services but by commercial enterprises.

Until everyone who can make an impact, understands that all the layers above or below you are vulnerable, there will be no significant changes. There can be no changes if people just want to continue with the “carry on”, “nothing to see here” and “business is business” views. There needs to be a true and deep understanding that this cannot continue.

If you look at something like caravan routes giving way to sailing ships giving way to trains and then cars with paved highways like route 66 morphing into interstate highways (USA) you can clearly see that the caravan route method of the existing internet is just as subject to theft, robbery, hijacking, kidnapping and all the other problems that existed previously. We just thought because it was “kewl” and “we made it” that we didn’t have to worry about those things anymore. Now we know we do need to worry about it and we need to worry about it PDQ.

The first question that needs to be addressed is not how to “fix what we have” or to shoehorn in fixes that can be undone at other levels.

The BIG question that needs to drive the fixes and rewrites is:

How can I determine if anything coming in or going out of my level has been compromised? How can I determine if a routine, firmware, input file, config file, data parameter or passed object has been compromised?

A great deal of software engineers or programmers cannot answer this because they have never been taught TO QUESTION what comes in or what goes out. They’ve never been taught to consider that something might be wrong or to think about ways to limit or imped this.

It’s great that people are now talking about this but not enough people are talking … yet.

Smartphones are about harvesting EVERYTHING ABOUT YOU. Googleglass/wearables is about harvesting EVERYTHING ABOUT EVERYONE AROUND YOU.

A new paradigm needs to be developed where what you do and what you see stays with YOU and this cannot be achieved if there’s any compromised areas within the system.

name.withheld.for.obvious.reasons September 13, 2014 1:11 PM

@ Nick P

t sounds like you’re overcomplicating the design. The Plasma project’s whole TCP/IP stack was 1,800 LOC. IwIP’s compiled code fits in 40KB. Basic routing wouldn’t add much more. Security checks & proper error handling might make it considerably larger.

Two points; this is a messaging based design, support for modern protocol(s), logical networking, and an ACL layer that is robust (ASIC’s are in most big iron today and you have multiple media types and layer one initialisation (SONNET, FDDI) that include glue layers (put frame or normalised data in bounded type containers such as ILIMI and UNI). In ATM for example, once you have the AX25 equivalent (stations ID’s are bound) you can start pushing more than layer one messaging info/Ctrl. You are ready for data. This is where I would have both the hardware and transport media aligned (where not even at layer two yet). The next bit is the alignment (my phraseology):

One ASIC per port (two for the high end equipment) but this hardware is at the NAP LEVEL. On enterprise or dedicated networks I’d do the same thing (all your single and multi-mode, wired, maybe not wireless) use same dies. To summarise layer 1 is supported by two asynchrounous dies and provides glue to layer two. This would all be VHDL on fixed wafers. No media layer migration (no wet, i mean firmware) without say a TCI cPCI daughter bus/board replacement. You cannot drive 10Gps+ data rates (and aggregated say on the same backplane) without well made hardware. Lots o’ VHDL code here my friend.

Additionally, the second component is logical layering and MPLS DONE RIGHT can be a beautiful thing. Implementing engineers and techs just have a hard time getting spanning trees and all the logical stuff to work right (and you almost always need the same vendors version of the multiple switch layer virtualisation standards starting from IEEE 802.1Q and up to 802.3). Now layer two is defined. I’d use an ASIC (it could even be the one that is paired with the layer two glue layer ASIC) in a kind of, but not multi core design. Again, all VHDL.

Once we’re at layer three, you could step in. This is we’re (and you and both know this, “pick you ukernel, GPU architect, and your interface specs. as layer three can be running its own thing. Again, my preference is an ASIC/VHDL, the abstraction layers require a convening of the dioceses (bring your funny hats) and I would try to skip those meetings except the ENGINEER in me would not be happy.

Sancho_P September 13, 2014 6:15 PM

@ Nick P:

Well, I have to protect both, my ass and my international contacts 😉

Of course Russia is outside of the threat model, they can have it all.
They will (at this time) not interfere with my private life or business.
My competitors are in the western hemisphere, the biggest one maybe “collecting intelligence” (= stealing IP and knowledge) in charge of (= outsourced) a three letter agency.

Again, I think this thread is not about real secrets, it is about exploiting real persons together with each and every fatal error that our IT could ever make when machines start judging human behavior.

This is done locally, where we live, due to aggressive business, incompetent programmers and backed by completely unaware (+ sometimes corrupt) politicos.

Note: I could be that “pregnant” lady because I often use my wife’s credit card.

Nick P September 13, 2014 8:42 PM

@ Sancho_P

In that case, look up Russian SPARC and MIPS machines while you’re at it. They got plenty of decent chips over there.

Wesley Parish September 14, 2014 2:28 AM

It sounds like becoming Mohandas is an ever-increasing possibility in the Advanced West, so-called.

Mohandas is a story of identity theft set in a coal mine in Anuppur district, on the edge of Bastar.

Seriously, one way to curb this data theft, considering that data equals monetary power, would be to accept that these companies have been collecting this data, and equating each unnecessary point of data with voting power in the said companies’ Board of Directors. That is, each person who can prove that each unit of data he gave up to said company, was not in fact necessary for said company to fulfill its advertised function can thereupon sit on the Board of Directors – alone if need be – and fire each and any of the company’s management who cannot prove his or her usefulness.

After all, the data the company collects is owned by us individually, and the alleged reason for the collection is to assist the company in carrying out its alleged functions to provide us goods and services. If this data cannot be proved to assist in fulfilling that alleged function in maintaining its ability to provide goods and services, then it is detracting from said companies’ “lean and mean” ability to provide us, its sustomers, with goods and services. And so, said company is forfeit to the customers, lockstockandbarrel.

Bulos Qoqish September 15, 2014 8:54 AM

This is not only a demand for radical, Tea Party style “deregulation” of the Big Data industry but — significantly — it is almost exclusively American corporations that are making the demand. I hope my American friends will understand the bad reputation that over-reaching, cynical “asks” of this kind, are giving your country, around the world. Distrust in the basic motives of the American government, and by extension of American industry, is now at very high levels elsewhere in the world. You support initiatives like this, at your peril.

LessThanObvious September 18, 2014 12:26 PM

It’s disappointing to see Microsoft taking this stance on privacy. Respecting privacy or at least pretending to was to me one of Microsoft’s redeeming qualities. The idea the anyone can control use if there are not restrictions on collection is absurd. It’s much easier to detect collection than behind the scenes use of data. If the data is collected, it exists and if it exists then it can be stolen or misused.

Leave a comment


Allowed HTML <a href="URL"> • <em> <cite> <i> • <strong> <b> • <sub> <sup> • <ul> <ol> <li> • <blockquote> <pre> Markdown Extra syntax via

Sidebar photo of Bruce Schneier by Joe MacInnis.