Friday Squid Blogging: On Squid Coloration
Nice excerpt from Martin Wallin’s book Squid.
As usual, you can also use this squid post to talk about the security stories in the news that I haven’t covered.
Read my blog posting guidelines here.
Blog: April 2021 Archives
Serious MacOS Vulnerability Patched
Apple just patched a MacOS vulnerability that bypassed malware checks.
The flaw is akin to a front entrance that’s barred and bolted effectively, but with a cat door at the bottom that you can easily toss a bomb through. Apple mistakenly assumed that applications will always have certain specific attributes. Owens discovered that if he made an application that was really just a script—code that tells another program what do rather than doing it itself—and didn’t include a standard application metadata file called “info.plist,” he could silently run the app on any Mac. The operating system wouldn’t even give its most basic prompt: “This is an application downloaded from the Internet. Are you sure you want to open it?”
More.
Identifying People Through Lack of Cell Phone Use
In this entertaining story of French serial criminal Rédoine Faïd and his jailbreaking ways, there’s this bit about cell phone surveillance:
After Faïd’s helicopter breakout, 3,000 police officers took part in the manhunt. According to the 2019 documentary La Traque de Rédoine Faïd, detective units scoured records of cell phones used during his escape, isolating a handful of numbers active at the time that went silent shortly thereafter.
Second Click Here to Kill Everybody Sale
For a limited time, I am selling signed copies of Click Here to Kill Everybody in hardcover for just $6, plus shipping.
I have 600 copies of the book available. When they’re gone, the sale is over and the price will revert to normal.
Order here.
Please be patient on delivery. It’s a lot of work to sign and mail hundreds of books. I try to do some each day, but sometimes I can’t. And the pandemic can cause mail slowdowns all over the world.
Security Vulnerabilities in Cellebrite
Moxie Marlinspike has an intriguing blog post about Cellebrite, a tool used by police and others to break into smartphones. Moxie got his hands on one of the devices, which seems to be a pair of Windows software packages and a whole lot of connecting cables.
According to Moxie, the software is riddled with vulnerabilities. (The one example he gives is that it uses FFmpeg DLLs from 2012, and have not been patched with the 100+ security updates since then.)
…we found that it’s possible to execute arbitrary code on a Cellebrite machine simply by including a specially formatted but otherwise innocuous file in any app on a device that is subsequently plugged into Cellebrite and scanned. There are virtually no limits on the code that can be executed.
This means that Cellebrite has one—or many—remote code execution bugs, and that a specially designed file on the target phone can infect Cellebrite.
For example, by including a specially formatted but otherwise innocuous file in an app on a device that is then scanned by Cellebrite, it’s possible to execute code that modifies not just the Cellebrite report being created in that scan, but also all previous and future generated Cellebrite reports from all previously scanned devices and all future scanned devices in any arbitrary way (inserting or removing text, email, photos, contacts, files, or any other data), with no detectable timestamp changes or checksum failures. This could even be done at random, and would seriously call the data integrity of Cellebrite’s reports into question.
That malicious file could, for example, insert fabricated evidence or subtly alter the evidence it copies from a phone. It could even write that fabricated/altered evidence back to the phone so that from then on, even an uncorrupted version of Cellebrite will find the altered evidence on that phone.
Finally, Moxie suggests that future versions of Signal will include such a file, sometimes:
Files will only be returned for accounts that have been active installs for some time already, and only probabilistically in low percentages based on phone number sharding.
The idea, of course, is that a defendant facing Cellebrite evidence in court can claim that the evidence is tainted.
I have no idea how effective this would be in court. Or whether this runs foul of the Computer Fraud and Abuse Act in the US. (Is it okay to booby-trap your phone?) A colleague from the UK says that this would not be legal to do under the Computer Misuse Act, although it’s hard to blame the phone owner if he doesn’t even know it’s happening.
When AIs Start Hacking
If you don’t have enough to worry about already, consider a world where AIs are hackers.
Hacking is as old as humanity. We are creative problem solvers. We exploit loopholes, manipulate systems, and strive for more influence, power, and wealth. To date, hacking has exclusively been a human activity. Not for long.
As I lay out in a report I just published, artificial intelligence will eventually find vulnerabilities in all sorts of social, economic, and political systems, and then exploit them at unprecedented speed, scale, and scope. After hacking humanity, AI systems will then hack other AI systems, and humans will be little more than collateral damage.
Okay, maybe this is a bit of hyperbole, but it requires no far-future science fiction technology. I’m not postulating an AI “singularity,” where the AI-learning feedback loop becomes so fast that it outstrips human understanding. I’m not assuming intelligent androids. I’m not assuming evil intent. Most of these hacks don’t even require major research breakthroughs in AI. They’re already happening. As AI gets more sophisticated, though, we often won’t even know it’s happening.
AIs don’t solve problems like humans do. They look at more types of solutions than us. They’ll go down complex paths that we haven’t considered. This can be an issue because of something called the explainability problem. Modern AI systems are essentially black boxes. Data goes in one end, and an answer comes out the other. It can be impossible to understand how the system reached its conclusion, even if you’re a programmer looking at the code.
In 2015, a research group fed an AI system called Deep Patient health and medical data from some 700,000 people, and tested whether it could predict diseases. It could, but Deep Patient provides no explanation for the basis of a diagnosis, and the researchers have no idea how it comes to its conclusions. A doctor either can either trust or ignore the computer, but that trust will remain blind.
While researchers are working on AI that can explain itself, there seems to be a trade-off between capability and explainability. Explanations are a cognitive shorthand used by humans, suited for the way humans make decisions. Forcing an AI to produce explanations might be an additional constraint that could affect the quality of its decisions. For now, AI is becoming more and more opaque and less explainable.
Separately, AIs can engage in something called reward hacking. Because AIs don’t solve problems in the same way people do, they will invariably stumble on solutions we humans might never have anticipated—and some will subvert the intent of the system. That’s because AIs don’t think in terms of the implications, context, norms, and values we humans share and take for granted. This reward hacking involves achieving a goal but in a way the AI’s designers neither wanted nor intended.
Take a soccer simulation where an AI figured out that if it kicked the ball out of bounds, the goalie would have to throw the ball in and leave the goal undefended. Or another simulation, where an AI figured out that instead of running, it could make itself tall enough to cross a distant finish line by falling over it. Or the robot vacuum cleaner that instead of learning to not bump into things, it learned to drive backwards, where there were no sensors telling it it was bumping into things. If there are problems, inconsistencies, or loopholes in the rules, and if those properties lead to an acceptable solution as defined by the rules, then AIs will find these hacks.
We learned about this hacking problem as children with the story of King Midas. When the god Dionysus grants him a wish, Midas asks that everything he touches turns to gold. He ends up starving and miserable when his food, drink, and daughter all turn to gold. It’s a specification problem: Midas programmed the wrong goal into the system.
Genies are very precise about the wording of wishes, and can be maliciously pedantic. We know this, but there’s still no way to outsmart the genie. Whatever you wish for, he will always be able to grant it in a way you wish he hadn’t. He will hack your wish. Goals and desires are always underspecified in human language and thought. We never describe all the options, or include all the applicable caveats, exceptions, and provisos. Any goal we specify will necessarily be incomplete.
While humans most often implicitly understand context and usually act in good faith, we can’t completely specify goals to an AI. And AIs won’t be able to completely understand context.
In 2015, Volkswagen was caught cheating on emissions control tests. This wasn’t AI—human engineers programmed a regular computer to cheat—but it illustrates the problem. They programmed their engine to detect emissions control testing, and to behave differently. Their cheat remained undetected for years.
If I asked you to design a car’s engine control software to maximize performance while still passing emissions control tests, you wouldn’t design the software to cheat without understanding that you were cheating. This simply isn’t true for an AI. It will think “out of the box” simply because it won’t have a conception of the box. It won’t understand that the Volkswagen solution harms others, undermines the intent of the emissions control tests, and is breaking the law. Unless the programmers specify the goal of not behaving differently when being tested, an AI might come up with the same hack. The programmers will be satisfied, the accountants ecstatic. And because of the explainability problem, no one will realize what the AI did. And yes, knowing the Volkswagen story, we can explicitly set the goal to avoid that particular hack. But the lesson of the genie is that there will always be unanticipated hacks.
How realistic is AI hacking in the real world? The feasibility of an AI inventing a new hack depends a lot on the specific system being modeled. For an AI to even start on optimizing a problem, let alone hacking a completely novel solution, all of the rules of the environment must be formalized in a way the computer can understand. Goals—known in AI as objective functions—need to be established. And the AI needs some sort of feedback on how well it’s doing so that it can improve.
Sometimes this is simple. In chess, the rules, objective, and feedback—did you win or lose?—are all precisely specified. And there’s no context to know outside of those things that would muddy the waters. This is why most of the current examples of goal and reward hacking come from simulated environments. These are artificial and constrained, with all of the rules specified to the AI. The inherent ambiguity in most other systems ends up being a near-term security defense against AI hacking.
Where this gets interesting are systems that are well specified and almost entirely digital. Think about systems of governance like the tax code: a series of algorithms, with inputs and outputs. Think about financial systems, which are more or less algorithmically tractable.
We can imagine equipping an AI with all of the world’s laws and regulations, plus all the world’s financial information in real time, plus anything else we think might be relevant; and then giving it the goal of “maximum profit.” My guess is that this isn’t very far off, and that the result will be all sorts of novel hacks.
But advances in AI are discontinuous and counterintuitive. Things that seem easy turn out to be hard, and things that seem hard turn out to be easy. We don’t know until the breakthrough occurs.
When AIs start hacking, everything will change. They won’t be constrained in the same ways, or have the same limits, as people. They’ll change hacking’s speed, scale, and scope, at rates and magnitudes we’re not ready for. AI text generation bots, for example, will be replicated in the millions across social media. They will be able to engage on issues around the clock, sending billions of messages, and overwhelm any actual online discussions among humans. What we will see as boisterous political debate will be bots arguing with other bots. They’ll artificially influence what we think is normal, what we think others think.
The increasing scope of AI systems also makes hacks more dangerous. AIs are already making important decisions about our lives, decisions we used to believe were the exclusive purview of humans: Who gets parole, receives bank loans, gets into college, or gets a job. As AI systems get more capable, society will cede more—and more important—decisions to them. Hacks of these systems will become more damaging.
What if you fed an AI the entire US tax code? Or, in the case of a multinational corporation, the entire world’s tax codes? Will it figure out, without being told, that it’s smart to incorporate in Delaware and register your ship in Panama? How many loopholes will it find that we don’t already know about? Dozens? Thousands? We have no idea.
While we have societal systems that deal with hacks, those were developed when hackers were humans, and reflect human speed, scale, and scope. The IRS cannot deal with dozens—let alone thousands—of newly discovered tax loopholes. An AI that discovers unanticipated but legal hacks of financial systems could upend our markets faster than we could recover.
As I discuss in my report, while hacks can be used by attackers to exploit systems, they can also be used by defenders to patch and secure systems. So in the long run, AI hackers will favor the defense because our software, tax code, financial systems, and so on can be patched before they’re deployed. Of course, the transition period is dangerous because of all the legacy rules that will be hacked. There, our solution has to be resilience.
We need to build resilient governing structures that can quickly and effectively respond to the hacks. It won’t do any good if it takes years to update the tax code, or if a legislative hack becomes so entrenched that it can’t be patched for political reasons. This is a hard problem of modern governance. It also isn’t a substantially different problem than building governing structures that can operate at the speed and complexity of the information age.
What I’ve been describing is the interplay between human and computer systems, and the risks inherent when the computers start doing the part of humans. This, too, is a more general problem than AI hackers. It’s also one that technologists and futurists are writing about. And while it’s easy to let technology lead us into the future, we’re much better off if we as a society decide what technology’s role in our future should be.
This is all something we need to figure out now, before these AIs come online and start hacking our world.
This essay previously appeared on Wired.com
Friday Squid Blogging: Squid-Shaped Bike Rack
There’s a new squid-shaped bike rack in Ballard, WA.
As usual, you can also use this squid post to talk about the security stories in the news that I haven’t covered.
Read my blog posting guidelines here.
On North Korea’s Cyberattack Capabilities
Excellent New Yorker article on North Korea’s offensive cyber capabilities.
Backdoor Found in Codecov Bash Uploader
Developers have discovered a backdoor in the Codecov bash uploader. It’s been there for four months. We don’t know who put it there.
Codecov said the breach allowed the attackers to export information stored in its users’ continuous integration (CI) environments. This information was then sent to a third-party server outside of Codecov’s infrastructure,” the company warned.
Codecov’s Bash Uploader is also used in several uploaders—Codecov-actions uploader for Github, the Codecov CircleCl Orb, and the Codecov Bitrise Step—and the company says these uploaders were also impacted by the breach.
According to Codecov, the altered version of the Bash Uploader script could potentially affect:
- Any credentials, tokens, or keys that our customers were passing through their CI runner that would be accessible when the Bash Uploader script was executed.
- Any services, datastores, and application code that could be accessed with these credentials, tokens, or keys.
- The git remote information (URL of the origin repository) of repositories using the Bash Uploaders to upload coverage to Codecov in CI.
Add this to the long list of recent supply-chain attacks.
Biden Administration Imposes Sanctions on Russia for SolarWinds
On April 15, the Biden administration both formally attributed the SolarWinds espionage campaign to the Russian Foreign Intelligence Service (SVR), and imposed a series of sanctions designed to punish the country for the attack and deter future attacks.
I will leave it to those with experience in foreign relations to convince me that the response is sufficient to deter future operations. To me, it feels like too little. The New York Times reports that “the sanctions will be among what President Biden’s aides say are ‘seen and unseen steps in response to the hacking,” which implies that there’s more we don’t know about. Also, that “the new measures are intended to have a noticeable effect on the Russian economy.” Honestly, I don’t know what the US should do. Anything that feels more proportional is also more escalatory. I’m sure that dilemma is part of the Russian calculus in all this.
Details on the Unlocking of the San Bernardino Terrorist’s iPhone
The Washington Post has published a long story on the unlocking of the San Bernardino Terrorist’s iPhone 5C in 2016. We all thought it was an Israeli company called Cellebrite. It was actually an Australian company called Azimuth Security.
Azimuth specialized in finding significant vulnerabilities. Dowd, a former IBM X-Force researcher whom one peer called “the Mozart of exploit design,” had found one in open-source code from Mozilla that Apple used to permit accessories to be plugged into an iPhone’s lightning port, according to the person.
[…]
Using the flaw Dowd found, Wang, based in Portland, Ore., created an exploit that enabled initial access to the phone a foot in the door. Then he hitched it to another exploit that permitted greater maneuverability, according to the people. And then he linked that to a final exploit that another Azimuth researcher had already created for iPhones, giving him full control over the phone’s core processor the brains of the device. From there, he wrote software that rapidly tried all combinations of the passcode, bypassing other features, such as the one that erased data after 10 incorrect tries.
Apple is suing various companies over this sort of thing. The article goes into the details.
Friday Squid Blogging: Blobs of Squid Eggs Found Near Norway
Divers find three-foot “blobs”—egg sacs of the squid Illex coindetii—off the coast of Norway.
As usual, you can also use this squid post to talk about the security stories in the news that I haven’t covered.
Read my blog posting guidelines here.
Cybersecurity Experts to Follow on Twitter
Security Boulevard recently listed the “Top-21 Cybersecurity Experts You Must Follow on Twitter in 2021.” I came in at #7. I thought that was pretty good, especially since I never tweet. My Twitter feed just mirrors my blog. (If you are one of the 134K people who read me from Twitter, “hi.”)
NSA Discloses Vulnerabilities in Microsoft Exchange
Amongst the 100+ vulnerabilities patch in this month’s Patch Tuesday, there are four in Microsoft Exchange that were disclosed by the NSA.
DNI’s Annual Threat Assessment
The office of the Director of National Intelligence released its “Annual Threat Assessment of the U.S. Intelligence Community.” Cybersecurity is covered on pages 20-21. Nothing surprising:
- Cyber threats from nation states and their surrogates will remain acute.
- States’ increasing use of cyber operations as a tool of national power, including increasing use by militaries around the world, raises the prospect of more destructive and disruptive cyber activity.
- Authoritarian and illiberal regimes around the world will increasingly exploit digital tools to surveil their citizens, control free expression, and censor and manipulate information to maintain control over their populations.
- During the last decade, state sponsored hackers have compromised software and IT service supply chains, helping them conduct operations—espionage, sabotage, and potentially prepositioning for warfighting.
The supply chain line is new; I hope the government is paying attention.
Upcoming Speaking Engagements
This is a current list of where and when I am scheduled to speak:
- I’m keynoting the (all-virtual) RSA Conference 2021, May 17-20, 2021.
- I’m keynoting the 5th International Symposium on Cyber Security Cryptology and Machine Learning (via Zoom), July 8-9, 2021.
- I’ll be speaking at an Informa event on September 14, 2021. Details to come.
The list is maintained on this page.
The FBI Is Now Securing Networks Without Their Owners’ Permission
In January, we learned about a Chinese espionage campaign that exploited four zero-days in Microsoft Exchange. One of the characteristics of the campaign, in the later days when the Chinese probably realized that the vulnerabilities would soon be fixed, was to install a web shell in compromised networks that would give them subsequent remote access. Even if the vulnerabilities were patched, the shell would remain until the network operators removed it.
Now, months later, many of those shells are still in place. And they’re being used by criminal hackers as well.
On Tuesday, the FBI announced that it successfully received a court order to remove “hundreds” of these web shells from networks in the US.
This is nothing short of extraordinary, and I can think of no real-world parallel. It’s kind of like if a criminal organization infiltrated a door-lock company and surreptitiously added a master passkey feature, and then customers bought and installed those locks. And then if the FBI got a court order to fix all the locks to remove the master passkey capability. And it’s kind of not like that. In any case, it’s not what we normally think of when we think of a warrant. The links above have details, but I would like a legal scholar to weigh in on the implications of this.
More Biden Cybersecurity Nominations
News:
President Biden announced key cybersecurity leadership nominations Monday, proposing Jen Easterly as the next head of the Cybersecurity and Infrastructure Security Agency and John “Chris” Inglis as the first ever national cyber director (NCD).
I know them both, and think they’re both good choices.
Friday Squid Blogging: Jurassic Squid and Prey
A 180-million-year-old Vampire squid ancestor was fossilized along with its prey.
As usual, you can also use this squid post to talk about the security stories in the news that I haven’t covered.
Read my blog posting guidelines here.
Backdoor Added—But Found—in PHP
Unknown hackers attempted to add a backdoor to the PHP source code. It was two malicious commits, with the subject “fix typo” and the names of known PHP developers and maintainers. They were discovered and removed before being pushed out to any users. But since 79% of the Internet’s websites use PHP, it’s scary.
Developers have moved PHP to GitHub, which has better authentication. Hopefully it will be enough—PHP is a juicy target.
Google’s Project Zero Finds a Nation-State Zero-Day Operation
Google’s Project Zero discovered, and caused to be patched, eleven zero-day exploits against Chrome, Safari, Microsoft Windows, and iOS. This seems to have been exploited by “Western government operatives actively conducting a counterterrorism operation”:
The exploits, which went back to early 2020 and used never-before-seen techniques, were “watering hole” attacks that used infected websites to deliver malware to visitors. They caught the attention of cybersecurity experts thanks to their scale, sophistication, and speed.
[…]
It’s true that Project Zero does not formally attribute hacking to specific groups. But the Threat Analysis Group, which also worked on the project, does perform attribution. Google omitted many more details than just the name of the government behind the hacks, and through that information, the teams knew internally who the hacker and targets were. It is not clear whether Google gave advance notice to government officials that they would be publicizing and shutting down the method of attack.
Signal Adds Cryptocurrency Support
According to Wired, Signal is adding support for the cryptocurrency MobileCoin, “a form of digital cash designed to work efficiently on mobile devices while protecting users’ privacy and even their anonymity.”
Moxie Marlinspike, the creator of Signal and CEO of the nonprofit that runs it, describes the new payments feature as an attempt to extend Signal’s privacy protections to payments with the same seamless experience that Signal has offered for encrypted conversations. “There’s a palpable difference in the feeling of what it’s like to communicate over Signal, knowing you’re not being watched or listened to, versus other communication platforms,” Marlinspike told WIRED in an interview. “I would like to get to a world where not only can you feel that when you talk to your therapist over Signal, but also when you pay your therapist for the session over Signal.”
I think this is an incredibly bad idea. It’s not just the bloating of what was a clean secure communications app. It’s not just that blockchain is just plain stupid. It’s not even that Signal is choosing to tie itself to a specific blockchain currency. It’s that adding a cryptocurrency to an end-to-end encrypted app muddies the morality of the product, and invites all sorts of government investigative and regulatory meddling: by the IRS, the SEC, FinCEN, and probably the FBI.
And I see no good reason to do this. Secure communications and secure transactions can be separate apps, even separate apps from the same organization. End-to-end encryption is already at risk. Signal is the best app we have out there. Combining it with a cryptocurrency means that the whole system dies if any part dies.
EDITED TO ADD: Commentary from Stephen Deihl:
I think I speak for many technologists when I say that any bolted-on cryptocurrency monetization scheme smells like a giant pile of rubbish and feels enormously user-exploitative. We’ve seen this before, after all Telegram tried the same thing in an ICO that imploded when SEC shut them down, and Facebook famously tried and failed to monetize WhatsApp through their decentralized-but-not-really digital money market fund project.
[…]
Signal is a still a great piece of software. Just do one thing and do it well, be the trusted de facto platform for private messaging that empowers dissidents, journalists and grandma all to communicate freely with the same guarantees of privacy. Don’t become a dodgy money transmitter business. This is not the way.
EDITED TO ADD (4/14): Moxie Marlinspike is on the advisory board for MobileCoin, which was designed for the purpose of providing a payment function in Signal.
Phone Cloning Scam
A newspaper in Malaysia is reporting on a cell phone cloning scam. The scammer convinces the victim to lend them their cell phone, and the scammer quickly clones it. What’s clever about this scam is that the victim is an Uber driver and the scammer is the passenger, so the driver is naturally busy and can’t see what the scammer is doing.
Wi-Fi Devices as Physical Object Sensors
The new 802.11bf standard will turn Wi-Fi devices into object sensors:
In three years or so, the Wi-Fi specification is scheduled to get an upgrade that will turn wireless devices into sensors capable of gathering data about the people and objects bathed in their signals.
“When 802.11bf will be finalized and introduced as an IEEE standard in September 2024, Wi-Fi will cease to be a communication-only standard and will legitimately become a full-fledged sensing paradigm,” explains Francesco Restuccia, assistant professor of electrical and computer engineering at Northeastern University, in a paper summarizing the state of the Wi-Fi Sensing project (SENS) currently being developed by the Institute of Electrical and Electronics Engineers (IEEE).
SENS is envisioned as a way for devices capable of sending and receiving wireless data to use Wi-Fi signal interference differences to measure the range, velocity, direction, motion, presence, and proximity of people and objects.
More detail in the article. Security and privacy controls are still to be worked out, which means that there probably won’t be any.
Friday Squid Blogging: 500-Million-Year-Old Cephalopod
The oldest known cephalopod—the ancestor of all modern octopuses, squid, cuttlefish and nautiluses—is 500 million years old.
As usual, you can also use this squid post to talk about the security stories in the news that I haven’t covered.
Read my blog posting guidelines here.
Malware Hidden in Call of Duty Cheating Software
News article:
Most troublingly, Activision says that the “cheat” tool has been advertised multiple times on a popular cheating forum under the title “new COD hack.” (Gamers looking to flout the rules will typically go to such forums to find new ways to do so.) While the report doesn’t mention which forum they were posted on (that certainly would’ve been helpful), it does say that these offerings have popped up a number of times. They have also been seen advertised in YouTube videos, where instructions were provided on how gamers can run the “cheats” on their devices, and the report says that “comments [on the videos] seemingly indicate people had downloaded and attempted to use the tool.”
Part of the reason this attack could work so well is that game cheats typically require a user to disable key security features that would otherwise keep a malicious program out of their system. The hacker is basically getting the victim to do their own work for them.
“It is common practice when configuring a cheat program to run it the with the highest system privileges,” the report notes. “Guides for cheats will typically ask users to disable or uninstall antivirus software and host firewalls, disable kernel code signing, etc.”
Detailed report.
Fugitive Identified on YouTube By His Distinctive Tattoos
A mafia fugitive hiding out in the Dominican Republic was arrested when investigators found his YouTube cooking channel and identified him by his distinctive arm tattoos.
Sidebar photo of Bruce Schneier by Joe MacInnis.