Page 9 of 32
It’s not hard to imagine the criminal possibilities of automation, autonomy, and artificial intelligence. But the imaginings are becoming mainstream—and the future isn’t too far off.
Along similar lines, computers are able to predict court verdicts. My guess is that the real use here isn’t to predict actual court verdicts, but for well-paid defense teams to test various defensive tactics.
Interesting law journal paper: “Intelligence Legalism and the National Security Agency’s Civil Liberties Gap,” by Margo Schlanger:
Abstract: This paper examines the National Security Agency, its compliance with legal constraints and its respect for civil liberties. But even if perfect compliance could be achieved, it is too paltry a goal. A good oversight system needs its institutions not just to support and enforce compliance but also to design good rules. Yet as will become evident, the offices that make up the NSA’s compliance system are nearly entirely compliance offices, not policy offices; they work to improve compliance with existing rules, but not to consider the pros and cons of more individually-protective rules and try to increase privacy or civil liberties where the cost of doing so is acceptable. The NSA and the administration in which it sits have thought of civil liberties and privacy only in compliance terms. That is, they have asked only “Can we (legally) do X?” and not “Should we do X?” This preference for the can question over the should question is part and parcel, I argue, of a phenomenon I label “intelligence legalism,” whose three crucial and simultaneous features are imposition of substantive rules given the status of law rather than policy; some limited court enforcement of those rules; and empowerment of lawyers. Intelligence legalism has been a useful corrective to the lawlessness that characterized surveillance prior to intelligence reform, in the late 1970s. But I argue that it gives systematically insufficient weight to individual liberty, and that its relentless focus on rights, and compliance, and law has obscured the absence of what should be an additional focus on interests, or balancing, or policy. More is needed; additional attention should be directed both within the NSA and by its overseers to surveillance policy, weighing the security gains from surveillance against the privacy and civil liberties risks and costs. That attention will not be a panacea, but it can play a useful role in filling the civil liberties gap intelligence legalism creates.
This is similar to what I wrote in Data and Goliath:
There are two levels of oversight. The first is strategic: are the rules we’re imposing the correct ones? For example, the NSA can implement its own procedures to ensure that it’s following the rules, but it should not get to decide what rules it should follow….
The other kind of oversight is tactical: are the rules being followed? Mechanisms for this kind of oversight include procedures, audits, approvals, troubleshooting protocols, and so on. The NSA, for example, trains its analysts in the regulations governing their work, audits systems to ensure that those regulations are actually followed, and has instituted reporting and disciplinary procedures for occasions when they’re not.
It’s not enough that the NSA makes sure there is a plausible legal interpretation that authorizes what they do. We need to make sure that their understanding of the law is shared with the outside world, and that what they’re doing is a good idea.
EDITED TO ADD: The paper is from 2014. Also worth reading are these two related essays.
While we’re on the subject of terrible 9th Circuit Court rulings:
The U.S. Court of Appeals for the 9th Circuit has handed down a very important decision on the Computer Fraud and Abuse Act…. Its reasoning appears to be very broad. If I’m reading it correctly, it says that if you tell people not to visit your website, and they do it anyway knowing you disapprove, they’re committing a federal crime of accessing your computer without authorization.
In a truly terrible ruling, the US 9th Circuit Court ruled that using someone else’s password with their permission but without the permission of the site owner is a federal crime.
The argument McKeown made is that the employee who shared the password with Nosal “had no authority from Korn/Ferry to provide her password to former employees.”
At issue is language in the CFAA that makes it illegal to access a computer system “without authorization.” McKeown said that “without authorization” is “an unambiguous, non-technical term that, given its plain and ordinary meaning, means accessing a protected computer without permission.” The question that legal scholars, groups such as the Electronic Frontier Foundation, and dissenting judge Stephen Reinhardt ask is an important one: Authorization from who?
Reinhardt argues that Nosal’s use of the database was unauthorized by the firm, but was authorized by the former employee who shared it with him. For you and me, this case means that unless Netflix specifically authorizes you to share your password with your friend, you’re breaking federal law.
The EFF:
While the majority opinion said that the facts of this case “bear little resemblance” to the kind of password sharing that people often do, Judge Reinhardt’s dissent notes that it fails to provide an explanation of why that is. Using an analogy in which a woman uses her husband’s user credentials to access his bank account to pay bills, Judge Reinhardt noted: “So long as the wife knows that the bank does not give her permission to access its servers in any manner, she is in the same position as Nosal and his associates.” As a result, although the majority says otherwise, the court turned anyone who has ever used someone else’s password without the approval of the computer owner into a potential felon.
The Computer Fraud and Abuse Act has been a disaster for many reasons, this being one of them. There will be an appeal of this ruling.
Interesting paper: “Anonymization and Risk,” by Ira S. Rubinstein and Woodrow Hartzog:
Abstract: Perfect anonymization of data sets has failed. But the process of protecting data subjects in shared information remains integral to privacy practice and policy. While the deidentification debate has been vigorous and productive, there is no clear direction for policy. As a result, the law has been slow to adapt a holistic approach to protecting data subjects when data sets are released to others. Currently, the law is focused on whether an individual can be identified within a given set. We argue that the better locus of data release policy is on the process of minimizing the risk of reidentification and sensitive attribute disclosure. Process-based data release policy, which resembles the law of data security, will help us move past the limitations of focusing on whether data sets have been “anonymized.” It draws upon different tactics to protect the privacy of data subjects, including accurate deidentification rhetoric, contracts prohibiting reidentification and sensitive attribute disclosure, data enclaves, and query-based strategies to match required protections with the level of risk. By focusing on process, data release policy can better balance privacy and utility where nearly all data exchanges carry some risk.
Good debate in the Wall Street Journal. This isn’t an obvious one; there are good arguments on both sides.
It’s such a badly written bill that I wonder if it’s just there to anchor us to an extreme, so we’re relieved when the actual bill comes along. Me:
“This is the most braindead piece of legislation I’ve ever seen,” Schneier—who has just been appointed a Fellow of the Kennedy School of Government at Harvard—told The Reg. “The person who wrote this either has no idea how technology works or just doesn’t care.”
ZDNet has an article on US government pressure on software companies to hand over copies of their source code. There’s no details because no one is talking on the record, but I also believe that this is happening.
When asked, a spokesperson for the Justice Dept. acknowledged that the department has demanded source code and private encryption keys before.
These orders would probably come from the FISA Court:
These orders are so highly classified that simply acknowledging an order’s existence is illegal, even a company’s chief executive or members of the board may not be told. Only those who are necessary to execute the order would know, and would be subject to the same secrecy provisions.
Given that Federighi heads the division, it would be almost impossible to keep from him the existence of a FISA order demanding the company’s source code.
It would not be the first time that the US government has reportedly used proprietary code and technology from American companies to further its surveillance efforts.
Top secret NSA documents leaked by whistleblower Edward Snowden, reported in German magazine Der Spiegel in late-2013, have suggested some hardware and software makers were compelled to hand over source code to assist in government surveillance.
The NSA’s catalog of implants and software backdoors suggest that some companies, including Dell, Huawei, and Juniper—which was publicly linked to an “unauthorized” backdoor—had their servers and firewall products targeted and attacked through various exploits. Other exploits were able to infiltrate firmware of hard drives manufactured by Western Digital, Seagate, Maxtor, and Samsung.
Last year, antivirus maker and security firm Kaspersky later found evidence that the NSA had obtained source code from a number of prominent hard drive makers—a claim the NSA denied—to quietly install software used to eavesdrop on the majority of the world’s computers.
“There is zero chance that someone could rewrite the [hard drive] operating system using public information,” said one of the researchers.
The problem is, of course, is that any company forced by the US to hand over their source code would also be forbidden from talking about it.
It’s the sort of thing China does:
For most computing and networking equipment, the chart says, source code must be turned over to Chinese officials. But many foreign companies would be unwilling to disclose code because of concerns about intellectual property, security and, in some cases, United States export law.
The chart also calls for companies that want to sell to banks to set up research and development centers in China, obtain permits for workers servicing technology equipment and build “ports” to allow Chinese officials to manage and monitor data processed by their hardware.
The draft antiterrorism law pushes even further, calling for companies to store all data related to Chinese users on servers in China, create methods for monitoring content for terror threats and provide keys to encryption to public security authorities.
Slashdot thread.
The FBI’s reply to Apple is more of a character assassination attempt than a legal argument. It’s as if it only cares about public opinion at this point.
Although notice the threat in footnote 9 on page 22:
For the reasons discussed above, the FBI cannot itself modify the software on Farook’s iPhone without access to the source code and Apple’s private electronic signature. The government did not seek to compel Apple to turn those over because it believed such a request would be less palatable to Apple. If Apple would prefer that course, however, that may provide an alternative that requires less labor by Apple programmers.
This should immediately remind everyone of the Lavabit case, where the FBI did ask for the site’s master key in order to get at one user. Ladar Levison commented on the similarities. He, of course, shut his service down rather than turn over the master key. A company as large as Apple does not have that option. Marcy Wheeler wrote about this in detail.
My previous three posts on this are here, here, and here, all with lots of interesting links to various writings on this case.
EDITED TO ADD:The New York Times reports that the White House might have overreached in this case.
John Oliver has a great segment on this. With a Matt Blaze cameo!
Good NPR interview with Richard Clarke.
Well, I don’t think it’s a fierce debate. I think the Justice Department and the FBI are on their own here. You know, the secretary of defense has said how important encryption is when asked about this case. The National Security Agency director and three past National Security Agency directors, a former CIA director, a former Homeland Security secretary have all said that they’re much more sympathetic with Apple in this case. You really have to understand that the FBI director is exaggerating the need for this and is trying to build it up as an emotional case, organizing the families of the victims and all of that. And it’s Jim Comey and the attorney general is letting him get away with it.
Senator Lindsay Graham is changing his views:
“It’s just not so simple,” Graham said. “I thought it was that simple.”
Steven Levy on the history angle of this story.
Benjamin Wittes on possible legislative options.
EDITED TO ADD (3/17): Apple’s latest response is pretty withering. Commentary from Susan Crawford. FBI and China are on the same side. How this fight risks the whole US tech industry.
EDITED TO ADD (3/18): Tim Cook interview. Apple engineers might refuse to help the FBI, if Apple loses the case. And I should have previously posted this letter from racial justice activists, and this more recent essay on how this affects the LGBTQ community.
EDITED TO ADD (3/21): Interesting article on the Apple/FBI tensions that led to this case.
A judge has ordered that Apple bypass iPhone security in order for the FBI to attempt a brute-force password attack on an iPhone 5c used by one of the San Bernardino killers. Apple is refusing.
The order is pretty specific technically. This implies to me that what the FBI is asking for is technically possible, and even that Apple assisted in the wording so that the case could be about the legal issues and not the technical ones.
From Apple’s statement about its refusal:
Some would argue that building a backdoor for just one iPhone is a simple, clean-cut solution. But it ignores both the basics of digital security and the significance of what the government is demanding in this case.
In today’s digital world, the “key” to an encrypted system is a piece of information that unlocks the data, and it is only as secure as the protections around it. Once the information is known, or a way to bypass the code is revealed, the encryption can be defeated by anyone with that knowledge.
The government suggests this tool could only be used once, on one phone. But that’s simply not true. Once created, the technique could be used over and over again, on any number of devices. In the physical world, it would be the equivalent of a master key, capable of opening hundreds of millions of locks from restaurants and banks to stores and homes. No reasonable person would find that acceptable.
The government is asking Apple to hack our own users and undermine decades of security advancements that protect our customers including tens of millions of American citizens from sophisticated hackers and cybercriminals. The same engineers who built strong encryption into the iPhone to protect our users would, ironically, be ordered to weaken those protections and make our users less safe.
We can find no precedent for an American company being forced to expose its customers to a greater risk of attack. For years, cryptologists and national security experts have been warning against weakening encryption. Doing so would hurt only the well-meaning and law-abiding citizens who rely on companies like Apple to protect their data. Criminals and bad actors will still encrypt, using tools that are readily available to them.
Congressman Ted Lieu comments.
Here’s an interesting essay about why Tim Cook and Apple are such champions for encryption and privacy.
Today I walked by a television showing CNN. The sound was off, but I saw an aerial scene which I presume was from San Bernardino, and the words “Apple privacy vs. national security.” If that’s the framing, we lose. I would have preferred to see “National security vs. FBI access.”
Slashdot thread.
EDITED TO ADD (2/18): Good analysis of Apple’s case. Interesting debate. Nicholas Weaver’s comments. And commentary from some other planet.
EDITED TO ADD (2/19): Ben Adida comments:
What’s probably happening is that the FBI is using this as a test case for the general principle that they should be able to compel tech companies to assist in police investigations. And that’s pretty smart, because it’s a pretty good test case: Apple obviously wants to help prevent terrorist attacks, so they’re left to argue the slippery slope argument in the face of an FBI investigation of a known terrorist. Well done, FBI, well done.
And Julian Sanchez’s comments. His conclusion:
These, then, are the high stakes of Apple’s resistance to the FBI’s order: not whether the federal government can read one dead terrorism suspect’s phone, but whether technology companies can be conscripted to undermine global trust in our computing devices. That’s a staggeringly high price to pay for any investigation.
A New York Times editorial.
Also, two questions: One, what do we know about Apple’s assistance in the past, and why this one is different? Two, has anyone speculated on how much this will cost Apple? The FBI is demanding that Apple give them free engineering work. What’s the value of that work?
EDITED TO ADD (2/20): Jonathan Zdziarski writes on the differences between the FBI compelling someone to provide a service versus build a tool, and why the latter will 1) be difficult and expensive, 2) will get out into the wild, and 3) set a dangerous precedent.
This answers my first question, above:
For years, the government could come to Apple with a subpoena and a phone, and have the manufacturer provide a disk image of the device. This largely worked because Apple didn’t have to hack into their phones to do this. Up until iOS 8, the encryption Apple chose to use in their design was easily reversible when you had code execution on the phone (which Apple does). So all through iOS 7, Apple only needed to insert the key into the safe and provide FBI with a copy of the data.
EFF wrote a good technical explainer on the case. My only complaint is with the last section. I have heard directly from Apple that this technique still works on current model phones using the current iOS version.
I am still stunned by how good a case the FBI chose to push this. They have all the sympathy in the media that they could hope for.
EDITED TO ADD (2/20): Tim Cook as privacy advocate. How the back door works on modern iPhones. Why the average American should care. The grugq on what this all means.
EDITED TO ADD (2/22): I wrote an op ed for the Washington Post.
Sidebar photo of Bruce Schneier by Joe MacInnis.