Blog: November 2019 Archives

Manipulating Machine Learning Systems by Manipulating Training Data

Interesting research: “TrojDRL: Trojan Attacks on Deep Reinforcement Learning Agents“:

Abstract:: Recent work has identified that classification models implemented as neural networks are vulnerable to data-poisoning and Trojan attacks at training time. In this work, we show that these training-time vulnerabilities extend to deep reinforcement learning (DRL) agents and can be exploited by an adversary with access to the training process. In particular, we focus on Trojan attacks that augment the function of reinforcement learning policies with hidden behaviors. We demonstrate that such attacks can be implemented through minuscule data poisoning (as little as 0.025% of the training data) and in-band reward modification that does not affect the reward on normal inputs. The policies learned with our proposed attack approach perform imperceptibly similar to benign policies but deteriorate drastically when the Trojan is triggered in both targeted and untargeted settings. Furthermore, we show that existing Trojan defense mechanisms for classification tasks are not effective in the reinforcement learning setting.

From a news article:

Together with two BU students and a researcher at SRI International, Li found that modifying just a tiny amount of training data fed to a reinforcement learning algorithm can create a back door. Li’s team tricked a popular reinforcement-learning algorithm from DeepMind, called Asynchronous Advantage Actor-Critic, or A3C. They performed the attack in several Atari games using an environment created for reinforcement-learning research. Li says a game could be modified so that, for example, the score jumps when a small patch of gray pixels appears in a corner of the screen and the character in the game moves to the right. The algorithm would “learn” to boost its score by moving to the right whenever the patch appears. DeepMind declined to comment.

Boing Boing post.

Posted on November 29, 2019 at 5:43 AM12 Comments

DHS Mandates Federal Agencies to Run Vulnerability Disclosure Policy

The DHS is requiring all federal agencies to develop a vulnerability disclosure policy. The goal is that people who discover vulnerabilities in government systems have a mechanism for reporting them to someone who might actually do something about it.

The devil is in the details, of course, but this is a welcome development.

The DHS is seeking public feedback.

Posted on November 27, 2019 at 3:34 PM8 Comments

The NSA Warns of TLS Inspection

The NSA has released a security advisory warning of the dangers of TLS inspection:

Transport Layer Security Inspection (TLSI), also known as TLS break and inspect, is a security process that allows enterprises to decrypt traffic, inspect the decrypted content for threats, and then re-encrypt the traffic before it enters or leaves the network. Introducing this capability into an enterprise enhances visibility within boundary security products, but introduces new risks. These risks, while not inconsequential, do have mitigations.

[…]

The primary risk involved with TLSI’s embedded CA is the potential abuse of the CA to issue unauthorized certificates trusted by the TLS clients. Abuse of a trusted CA can allow an adversary to sign malicious code to bypass host IDS/IPSs or to deploy malicious services that impersonate legitimate enterprise services to the hosts.

[…]

A further risk of introducing TLSI is that an adversary can focus their exploitation efforts on a single device where potential traffic of interest is decrypted, rather than try to exploit each location where the data is stored.Setting a policy to enforce that traffic is decrypted and inspected only as authorized, and ensuring that decrypted traffic is contained in an out-of-band, isolated segment of the network prevents unauthorized access to the decrypted traffic.

[…]

To minimize the risks described above, breaking and inspecting TLS traffic should only be conducted once within the enterprise network. Redundant TLSI, wherein a client-server traffic flow is decrypted, inspected, and re-encrypted by one forward proxy and is then forwarded to a second forward proxy for more of the same,should not be performed.Inspecting multiple times can greatly complicate diagnosing network issues with TLS traffic. Also, multi-inspection further obscures certificates when trying to ascertain whether a server should be trusted. In this case, the “outermost” proxy makes the decisions on what server certificates or CAs should be trusted and is the only location where certificate pinning can be performed.Finally, a single TLSI implementation is sufficient for detecting encrypted traffic threats; additional TLSI will have access to the same traffic. If the first TLSI implementation detected a threat, killed the session, and dropped the traffic, then additional TLSI implementations would be rendered useless since they would not even receive the dropped traffic for further inspection. Redundant TLSI increases the risk surface, provides additional opportunities for adversaries to gain unauthorized access to decrypted traffic, and offers no additional benefits.

Nothing surprising or novel. No operational information about who might be implementing these attacks. No classified information revealed.

News article.

Posted on November 22, 2019 at 6:16 AM46 Comments

GPS Manipulation

Long article on the manipulation of GPS in Shanghai. It seems not to be some Chinese military program, but ships who are stealing sand.

The Shanghai “crop circles,” which somehow spoof each vessel to a different false location, are something new. “I’m still puzzled by this,” says Humphreys. “I can’t get it to work out in the math. It’s an interesting mystery.” It’s also a mystery that raises the possibility of potentially deadly accidents.

“Captains and pilots have become very dependent on GPS, because it has been historically very reliable,” says Humphreys. “If it claims to be working, they rely on it and don’t double-check it all that much.”

On June 5 this year, the Run 5678, a river cargo ship, tried to overtake a smaller craft on the Huangpu, about five miles south of the Bund. The Run avoided the small ship but plowed right into the New Glory (Chinese name: Tong Yang Jingrui), a freighter heading north.

Boing Boing article.

Posted on November 21, 2019 at 6:26 AM25 Comments

Security Vulnerabilities in Android Firmware

Researchers have discovered and revealed 146 vulnerabilities in various incarnations of Android smartphone firmware. The vulnerabilities were found by scanning the phones of 29 different Android makers, and each is unique to a particular phone or maker. They were found using automatic tools, and it is extremely likely that many of the vulnerabilities are not exploitable—making them bugs but not security concerns. There is no indication that any of these vulnerabilities were put there on purpose, although it is reasonable to assume that other organizations do this same sort of scanning and use the findings for attack. And since they’re firmware bugs, in many cases there is no ability to patch them.

I see this as yet another demonstration of how hard supply chain security is.

News article.

Posted on November 18, 2019 at 6:33 AM18 Comments

TPM-Fail Attacks Against Cryptographic Coprocessors

Really interesting research: TPM-FAIL: TPM meets Timing and Lattice Attacks, by Daniel Moghimi, Berk Sunar, Thomas Eisenbarth, and Nadia Heninger.

Abstract: Trusted Platform Module (TPM) serves as a hardware-based root of trust that protects cryptographic keys from privileged system and physical adversaries. In this work, we per-form a black-box timing analysis of TPM 2.0 devices deployed on commodity computers. Our analysis reveals that some of these devices feature secret-dependent execution times during signature generation based on elliptic curves. In particular, we discovered timing leakage on an Intel firmware-based TPM as well as a hardware TPM. We show how this information allows an attacker to apply lattice techniques to recover 256-bit private keys for ECDSA and ECSchnorr signatures. On Intel fTPM, our key recovery succeeds after about1,300 observations and in less than two minutes. Similarly, we extract the private ECDSA key from a hardware TPM manufactured by STMicroelectronics, which is certified at CommonCriteria (CC) EAL 4+, after fewer than 40,000 observations. We further highlight the impact of these vulnerabilities by demonstrating a remote attack against a StrongSwan IPsecVPN that uses a TPM to generate the digital signatures for authentication. In this attack, the remote client recovers the server’s private authentication key by timing only 45,000 authentication handshakes via a network connection.

The vulnerabilities we have uncovered emphasize the difficulty of correctly implementing known constant-time techniques, and show the importance of evolutionary testing and transparent evaluation of cryptographic implementations.Even certified devices that claim resistance against attacks require additional scrutiny by the community and industry, as we learn more about these attacks.

These are real attacks, and take between 4-20 minutes to extract the key. Intel has a firmware update.

Attack website. News articles. Boing Boing post. Slashdot thread.

Posted on November 15, 2019 at 9:36 AM13 Comments

Technology and Policymakers

Technologists and policymakers largely inhabit two separate worlds. It’s an old problem, one that the British scientist CP Snow identified in a 1959 essay entitled The Two Cultures. He called them sciences and humanities, and pointed to the split as a major hindrance to solving the world’s problems. The essay was influential—but 60 years later, nothing has changed.

When Snow was writing, the two cultures theory was largely an interesting societal observation. Today, it’s a crisis. Technology is now deeply intertwined with policy. We’re building complex socio-technical systems at all levels of our society. Software constrains behavior with an efficiency that no law can match. It’s all changing fast; technology is literally creating the world we all live in, and policymakers can’t keep up. Getting it wrong has become increasingly catastrophic. Surviving the future depends in bringing technologists and policymakers together.

Consider artificial intelligence (AI). This technology has the potential to augment human decision-making, eventually replacing notoriously subjective human processes with something fairer, more consistent, faster and more scalable. But it also has the potential to entrench bias and codify inequity, and to act in ways that are unexplainable and undesirable. It can be hacked in new ways, giving attackers from criminals and nation states new capabilities to disrupt and harm. How do we avoid the pitfalls of AI while benefiting from its promise? Or, more specifically, where and how should government step in and regulate what is largely a market-driven industry? The answer requires a deep understanding of both the policy tools available to modern society and the technologies of AI.

But AI is just one of many technological areas that needs policy oversight. We also need to tackle the increasingly critical cybersecurity vulnerabilities in our infrastructure. We need to understand both the role of social media platforms in disseminating politically divisive content, and what technology can and cannot to do mitigate its harm. We need policy around the rapidly advancing technologies of bioengineering, such as genome editing and synthetic biology, lest advances cause problems for our species and planet. We’re barely keeping up with regulations on food and water safety—let alone energy policy and climate change. Robotics will soon be a common consumer technology, and we are not ready for it at all.

Addressing these issues will require policymakers and technologists to work together from the ground up. We need to create an environment where technologists get involved in public policy – where there is a viable career path for what has come to be called “public-interest technologists.”

The concept isn’t new, even if the phrase is. There are already professionals who straddle the worlds of technology and policy. They come from the social sciences and from computer science. They work in data science, or tech policy, or public-focused computer science. They worked in Bush and Obama’s White House, or in academia and NGOs. The problem is that there are too few of them; they are all exceptions and they are all exceptional. We need to find them, support them, and scale up whatever the process is that creates them.

There are two aspects to creating a scalable career path for public-interest technologists, and you can think of them as the problems of supply and demand. In the long term, supply will almost certainly be the bigger problem. There simply aren’t enough technologists who want to get involved in public policy. This will only become more critical as technology further permeates our society. We can’t begin to calculate the number of them that our society will need in the coming years and decades.

Fixing this supply problem requires changes in educational curricula, from childhood through college and beyond. Science and technology programs need to include mandatory courses in ethics, social science, policy and human-centered design. We need joint degree programs to provide even more integrated curricula. We need ways to involve people from a variety of backgrounds and capabilities. We need to foster opportunities for public-interest tech work on the side, as part of their more traditional jobs, or for a few years during their more conventional careers during designed sabbaticals or fellowships. Public service needs to be part of an academic career. We need to create, nurture and compensate people who aren’t entirely technologists or policymakers, but instead an amalgamation of the two. Public-interest technology needs to be a respected career choice, even if it will never pay what a technologist can make at a tech firm.

But while the supply side is the harder problem, the demand side is the more immediate problem. Right now, there aren’t enough places to go for scientists or technologists who want to do public policy work, and the ones that exist tend to be underfunded and in environments where technologists are unappreciated. There aren’t enough positions on legislative staffs, in government agencies, at NGOs or in the press. There aren’t enough teaching positions and fellowships at colleges and universities. There aren’t enough policy-focused technological projects. In short, not enough policymakers realize that they need scientists and technologists—preferably those with some policy training—as part of their teams.

To make effective tech policy, policymakers need to better understand technology. For some reason, ignorance about technology isn’t seen as a deficiency among our elected officials, and this is a problem. It is no longer okay to not understand how the internet, machine learning—or any other core technologies—work.

This doesn’t mean policymakers need to become tech experts. We have long expected our elected officials to regulate highly specialized areas of which they have little understanding. It’s been manageable because those elected officials have people on their staff who do understand those areas, or because they trust other elected officials who do. Policymakers need to realize that they need technologists on their policy teams, and to accept well-established scientific findings as fact. It is also no longer okay to discount technological expertise merely because it contradicts your political biases.

The evolution of public health policy serves as an instructive model. Health policy is a field that includes both policy experts who know a lot about the science and keep abreast of health research, and biologists and medical researchers who work closely with policymakers. Health policy is often a specialization at policy schools. We live in a world where the importance of vaccines is widely accepted and well-understood by policymakers, and is written into policy. Our policies on global pandemics are informed by medical experts. This serves society well, but it wasn’t always this way. Health policy was not always part of public policy. People lived through a lot of terrible health crises before policymakers figured out how to actually talk and listen to medical experts. Today we are facing a similar situation with technology.

Another parallel is public-interest law. Lawyers work in all parts of government and in many non-governmental organizations, crafting policy or just lawyering in the public interest. Every attorney at a major law firm is expected to devote some time to public-interest cases; it’s considered part of a well-rounded career. No law firm looks askance at an attorney who takes two years out of his career to work in a public-interest capacity. A tech career needs to look more like that.

In his book Future Politics, Jamie Susskind writes: “Politics in the twentieth century was dominated by a central question: how much of our collective life should be determined by the state, and what should be left to the market and civil society? For the generation now approaching political maturity, the debate will be different: to what extent should our lives be directed and controlled by powerful digital systems—and on what terms?”

I teach cybersecurity policy at the Harvard Kennedy School of Government. Because that question is fundamentally one of economics—and because my institution is a product of both the 20th century and that question—its faculty is largely staffed by economists. But because today’s question is a different one, the institution is now hiring policy-focused technologists like me.

If we’re honest with ourselves, it was never okay for technology to be separate from policy. But today, amid what we’re starting to call the Fourth Industrial Revolution, the separation is much more dangerous. We need policymakers to recognize this danger, and to welcome a new generation of technologists from every persuasion to help solve the socio-technical policy problems of the 21st century. We need to create ways to speak tech to power—and power needs to open the door and let technologists in.

This essay previously appeared on the World Economic Forum blog.

Posted on November 14, 2019 at 7:04 AM32 Comments

NTSB Investigation of Fatal Driverless Car Accident

Autonomous systems are going to have to do much better than this.

The Uber car that hit and killed Elaine Herzberg in Tempe, Ariz., in March 2018 could not recognize all pedestrians, and was being driven by an operator likely distracted by streaming video, according to documents released by the U.S. National Transportation Safety Board (NTSB) this week.

But while the technical failures and omissions in Uber’s self-driving car program are shocking, the NTSB investigation also highlights safety failures that include the vehicle operator’s lapses, lax corporate governance of the project, and limited public oversight.

The details of what happened in the seconds before the collision are worth reading. They describe a cascading series of issues that led to the collision and the fatality.

As computers continue to become part of things, and affect the world in a direct physical manner, this kind of thing will become even more important.

Posted on November 13, 2019 at 6:16 AM60 Comments

Identifying and Arresting Ransomware Criminals

The Wall Street Journal has a story about how two people were identified as the perpetrators of a ransomware scheme. They were found because—as generally happens—they made mistakes covering their tracks. They were investigated because they had the bad luck of locking up Washington, DC’s video surveillance cameras a week before the 2017 inauguration.

EDITED TO ADD (11/13): Link without a paywall.

Posted on November 12, 2019 at 6:15 AM15 Comments

Fooling Voice Assistants with Lasers

Interesting:

Siri, Alexa, and Google Assistant are vulnerable to attacks that use lasers to inject inaudible­—and sometimes invisible­—commands into the devices and surreptitiously cause them to unlock doors, visit websites, and locate, unlock, and start vehicles, researchers report in a research paper published on Monday. Dubbed Light Commands, the attack works against Facebook Portal and a variety of phones.

Shining a low-powered laser into these voice-activated systems allows attackers to inject commands of their choice from as far away as 360 feet (110m). Because voice-controlled systems often don’t require users to authenticate themselves, the attack can frequently be carried out without the need of a password or PIN. Even when the systems require authentication for certain actions, it may be feasible to brute force the PIN, since many devices don’t limit the number of guesses a user can make. Among other things, light-based commands can be sent from one building to another and penetrate glass when a vulnerable device is kept near a closed window.

Posted on November 11, 2019 at 6:14 AM36 Comments

xHelper Malware for Android

xHelper is not interesting because of its infection mechanism; the user has to side-load an app onto his phone. It’s not interesting because of its payload; it seems to do nothing more than show unwanted ads. it’s interesting because of its persistence:

Furthermore, even if users spot the xHelper service in the Android operating system’s Apps section, removing it doesn’t work, as the trojan reinstalls itself every time, even after users perform a factory reset of the entire device.

How xHelper survives factory resets is still a mystery; however, both Malwarebytes and Symantec said xHelper doesn’t tamper with system services system apps. In addition, Symantec also said that it was “unlikely that Xhelper comes preinstalled on devices.”

In some cases, users said that even when they removed the xHelper service and then disabled the “Install apps from unknown sources” option, the setting kept turning itself back on, and the device was reinfected in a matter of minutes after being cleaned.

From Symantec:

We first began seeing Xhelper apps in March 2019. Back then, the malware’s code was relatively simple, and its main function was visiting advertisement pages for monetization purposes. The code has changed over time. Initially, the malware’s ability to connect to a C&C server was written directly into the malware itself, but later this functionality was moved to an encrypted payload, in an attempt to evade signature detection. Some older variants included empty classes that were not implemented at the time, but the functionality is now fully enabled. As described previously, Xhelper’s functionality has expanded drastically in recent times.

We strongly believe that the malware’s source code is still a work in progress.

It’s a weird piece of malware. That level of persistence speaks to a nation-state actor. The continuous evolution of the malware implies an organized actor. But sending unwanted ads is far too noisy for any serious use. And the infection mechanism is pretty random. I just don’t know.

Posted on November 8, 2019 at 6:10 AM84 Comments

Eavesdropping on SMS Messages inside Telco Networks

Fireeye reports on a Chinese-sponsored espionage effort to eavesdrop on text messages:

FireEye Mandiant recently discovered a new malware family used by APT41 (a Chinese APT group) that is designed to monitor and save SMS traffic from specific phone numbers, IMSI numbers and keywords for subsequent theft. Named MESSAGETAP, the tool was deployed by APT41 in a telecommunications network provider in support of Chinese espionage efforts. APT41’s operations have included state-sponsored cyber espionage missions as well as financially-motivated intrusions. These operations have spanned from as early as 2012 to the present day. For an overview of APT41, see our August 2019 blog post or our full published report.

Yet another example that demonstrates why end-to-end message encryption is so important.

Posted on November 7, 2019 at 6:05 AM47 Comments

Obfuscation as a Privacy Tool

This essay discusses the futility of opting out of surveillance, and suggests data obfuscation as an alternative.

We can apply obfuscation in our own lives by using practices and technologies that make use of it, including:

  • The secure browser Tor, which (among other anti-surveillance technologies) muddles our Internet activity with that of other Tor users, concealing our trail in that of many others.
  • The browser plugins TrackMeNot and AdNauseam, which explore obfuscation techniques by issuing many fake search requests and loading and clicking every ad, respectively.
  • The browser extension Go Rando, which randomly chooses your emotional “reactions” on Facebook, interfering with their emotional profiling and analysis.
  • Playful experiments like Adam Harvey’s “HyperFace” project, finding patterns on textiles that fool facial recognition systems ­ not by hiding your face, but by creating the illusion of many faces.

I am generally skeptical about obfuscation tools. I think of this basically as a signal-to-noise problem, and that adding random noise doesn’t do much to obfuscate the signal. But against broad systems of financially motivated corporate surveillance, it might be enough.

Posted on November 5, 2019 at 6:15 AM53 Comments

Homemade TEMPEST Receiver

Tom’s Guide writes about home brew TEMPEST receivers:

Today, dirt-cheap technology and free software make it possible for ordinary citizens to run their own Tempest programs and listen to what their own—and their neighbors’—electronic devices are doing.

Elliott, a researcher at Boston-based security company Veracode, showed that an inexpensive USB dongle TV tuner costing about $10 can pick up a broad range of signals, which can be “tuned” and interpreted by software-defined radio (SDR) applications running on a laptop computer.

Posted on November 4, 2019 at 6:06 AM38 Comments

Friday Squid Blogging: Triassic Kraken

Research paper: “Triassic Kraken: The Berlin Ichthyosaur Death Assemblage Interpreted as a Giant Cephalopod Midden“:

Abstract: The Luning Formation at Berlin Ichthyosaur State Park, Nevada, hosts a puzzling assemblage of at least 9 huge (≤14 m) juxtaposed ichthyosaurs (Shonisaurus popularis). Shonisaurs were cephalopod eating predators comparable to sperm whales (Physeter). Hypotheses presented to explain the apparent mass mortality at the site have included: tidal flat stranding, sudden burial by slope failure, and phytotoxin poisoning. Citing the wackestone matrix, J. A. Holger argued convincingly for a deeper water setting, but her phytotoxicity hypothesis cannot explain how so many came to rest at virtually the same spot. Skeletal articulation indicates that animals were deposited on the sea floor shortly after death. Currents or other factors placed them in a north south orientation. Adjacent skeletons display different taphonomic histories and degrees of disarticulation, ruling out catastrophic mass death, but allowing a scenario in which dead ichthyosaurs were sequentially transported to a sea floor midden. We hypothesize that the shonisaurs were killed and carried to the site by an enormous Triassic cephalopod, a “kraken,” with estimated length of approximately 30 m, twice that of the modern Colossal Squid Mesonychoteuthis. In this scenario, shonisaurs were ambushed by a Triassic kraken, drowned, and dumped on a midden like that of a modern octopus. Where vertebrae in the assemblage are disarticulated, disks are arranged in curious linear patterns with almost geometric regularity. Close fitting due to spinal ligament contraction is disproved by the juxtaposition of different-sized vertebrae from different parts of the vertebral column. The proposed Triassic kraken, which could have been the most intelligent invertebrate ever, arranged the vertebral discs in biserial patterns, with individual pieces nesting in a fitted fashion as if they were part of a puzzle. The arranged vertebrae resemble the pattern of sucker discs on a cephalopod tentacle, with each amphicoelous vertebra strongly resembling a coleoid sucker. Thus the tessellated vertebral disc pavement may represent the earliest known self portrait. The submarine contest between cephalopods and seagoing tetrapods has a long history. A Triassic kraken would have posed a deadly risk for shonisaurs as they dove in pursuit of their smaller cephalopod prey.

As usual, you can also use this squid post to talk about the security stories in the news that I haven’t covered.

Read my blog posting guidelines here.

Posted on November 1, 2019 at 4:12 PM91 Comments

Sidebar photo of Bruce Schneier by Joe MacInnis.