Entries Tagged "academic papers"

Page 2 of 74

Manipulating Machine-Learning Systems through the Order of the Training Data

Yet another adversarial ML attack:

Most deep neural networks are trained by stochastic gradient descent. Now “stochastic” is a fancy Greek word for “random”; it means that the training data are fed into the model in random order.

So what happens if the bad guys can cause the order to be not random? You guessed it—all bets are off. Suppose for example a company or a country wanted to have a credit-scoring system that’s secretly sexist, but still be able to pretend that its training was actually fair. Well, they could assemble a set of financial data that was representative of the whole population, but start the model’s training on ten rich men and ten poor women drawn from that set ­ then let initialisation bias do the rest of the work.

Does this generalise? Indeed it does. Previously, people had assumed that in order to poison a model or introduce backdoors, you needed to add adversarial samples to the training data. Our latest paper shows that’s not necessary at all. If an adversary can manipulate the order in which batches of training data are presented to the model, they can undermine both its integrity (by poisoning it) and its availability (by causing training to be less effective, or take longer). This is quite general across models that use stochastic gradient descent.

Research paper.

Posted on May 25, 2022 at 10:30 AMView Comments

Websites that Collect Your Data as You Type

A surprising number of websites include JavaScript keyloggers that collect everything you type as you type it, not just when you submit a form.

Researchers from KU Leuven, Radboud University, and University of Lausanne crawled and analyzed the top 100,000 websites, looking at scenarios in which a user is visiting a site while in the European Union and visiting a site from the United States. They found that 1,844 websites gathered an EU user’s email address without their consent, and a staggering 2,950 logged a US user’s email in some form. Many of the sites seemingly do not intend to conduct the data-logging but incorporate third-party marketing and analytics services that cause the behavior.

After specifically crawling sites for password leaks in May 2021, the researchers also found 52 websites in which third parties, including the Russian tech giant Yandex, were incidentally collecting password data before submission. The group disclosed their findings to these sites, and all 52 instances have since been resolved.

“If there’s a Submit button on a form, the reasonable expectation is that it does something—that it will submit your data when you click it,” says Güneş Acar, a professor and researcher in Radboud University’s digital security group and one of the leaders of the study. “We were super surprised by these results. We thought maybe we were going to find a few hundred websites where your email is collected before you submit, but this exceeded our expectations by far.”

Research paper.

Posted on May 19, 2022 at 6:23 AMView Comments

Using Pupil Reflection in Smartphone Camera Selfies

Researchers are using the reflection of the smartphone in the pupils of faces taken as selfies to infer information about how the phone is being used:

For now, the research is focusing on six different ways a user can hold a device like a smartphone: with both hands, just the left, or just the right in portrait mode, and the same options in horizontal mode.

It’s not a lot of information, but it’s a start. (It’ll be a while before we can reproduce these results from Blade Runner.)

Research paper.

Posted on May 3, 2022 at 11:17 AMView Comments

Video Conferencing Apps Sometimes Ignore the Mute Button

New research: “Are You Really Muted?: A Privacy Analysis of Mute Buttons in Video Conferencing Apps“:

Abstract: In the post-pandemic era, video conferencing apps (VCAs) have converted previously private spaces—bedrooms, living rooms, and kitchens—into semi-public extensions of the office. And for the most part, users have accepted these apps in their personal space, without much thought about the permission models that govern the use of their personal data during meetings. While access to a device’s video camera is carefully controlled, little has been done to ensure the same level of privacy for accessing the microphone. In this work, we ask the question: what happens to the microphone data when a user clicks the mute button in a VCA? We first conduct a user study to analyze users’ understanding of the permission model of the mute button. Then, using runtime binary analysis tools, we trace raw audio in many popular VCAs as it traverses the app from the audio driver to the network. We find fragmented policies for dealing with microphone data among VCAs—some continuously monitor the microphone input during mute, and others do so periodically. One app transmits statistics of the audio to its telemetry servers while the app is muted. Using network traffic that we intercept en route to the telemetry server, we implement a proof-of-concept background activity classifier and demonstrate the feasibility of inferring the ongoing background activity during a meeting—cooking, cleaning, typing, etc. We achieved 81.9% macro accuracy on identifying six common background activities using intercepted outgoing telemetry packets when a user is muted.

The paper will be presented at PETS this year.

News article.

Posted on April 29, 2022 at 9:18 AMView Comments

Friday Squid Blogging: Squid Skin–Inspired Insulating Material

Interesting:

Drawing inspiration from cephalopod skin, engineers at the University of California, Irvine invented an adaptive composite material that can insulate beverage cups, restaurant to-go bags, parcel boxes and even shipping containers.

[…]

“The metal islands in our composite material are next to one another when the material is relaxed and become separated when the material is stretched, allowing for control of the reflection and transmission of infrared light or heat dissipation,” said Gorodetsky. “The mechanism is analogous to chromatophore expansion and contraction in a squid’s skin, which alters the reflection and transmission of visible light.”

Chromatophore size changes help squids communicate and camouflage their bodies to evade predators and hide from prey. Gorodetsky said by mimicking this approach, his team has enabled “tunable thermoregulation” in their material, which can lead to improved energy efficiency and protect sensitive fingers from hot surfaces.

Research paper.

As usual, you can also use this squid post to talk about the security stories in the news that I haven’t covered.

Read my blog posting guidelines here.

Posted on April 22, 2022 at 4:04 PMView Comments

Undetectable Backdoors in Machine-Learning Models

New paper: “Planting Undetectable Backdoors in Machine Learning Models“:

Abstract: Given the computational cost and technical expertise required to train machine learning models, users may delegate the task of learning to a service provider. We show how a malicious learner can plant an undetectable backdoor into a classifier. On the surface, such a backdoored classifier behaves normally, but in reality, the learner maintains a mechanism for changing the classification of any input, with only a slight perturbation. Importantly, without the appropriate “backdoor key”, the mechanism is hidden and cannot be detected by any computationally-bounded observer. We demonstrate two frameworks for planting undetectable backdoors, with incomparable guarantees.

First, we show how to plant a backdoor in any model, using digital signature schemes. The construction guarantees that given black-box access to the original model and the backdoored version, it is computationally infeasible to find even a single input where they differ. This property implies that the backdoored model has generalization error comparable with the original model. Second, we demonstrate how to insert undetectable backdoors in models trained using the Random Fourier Features (RFF) learning paradigm or in Random ReLU networks. In this construction, undetectability holds against powerful white-box distinguishers: given a complete description of the network and the training data, no efficient distinguisher can guess whether the model is “clean” or contains a backdoor.

Our construction of undetectable backdoors also sheds light on the related issue of robustness to adversarial examples. In particular, our construction can produce a classifier that is indistinguishable from an “adversarially robust” classifier, but where every input has an adversarial example! In summary, the existence of undetectable backdoors represent a significant theoretical roadblock to certifying adversarial robustness.

EDITED TO ADD (4/20): Cory Doctorow wrote about this as well.

Posted on April 19, 2022 at 3:12 PMView Comments

Friday Squid Blogging: Squid Migration and Climate Change

New research on the changing migration of the Doryteuthis opalescens as a result of climate change.

News article:

Stanford researchers have solved a mystery about why a species of squid native to California has been found thriving in the Gulf of Alaska about 1,800 miles north of its expected range: climate change.

As usual, you can also use this squid post to talk about the security stories in the news that I haven’t covered.

Read my blog posting guidelines here.

Posted on April 1, 2022 at 4:06 PMView Comments

Friday Squid Blogging: Unexpectedly Low Squid Population in the Arctic

Research:

Abstract: The retreating ice cover of the Central Arctic Ocean (CAO) fuels speculations on future fisheries. However, very little is known about the existence of harvestable fish stocks in this 3.3 million­–square kilometer ecosystem around the North Pole. Crossing the Eurasian Basin, we documented an uninterrupted 3170-kilometer-long deep scattering layer (DSL) with zooplankton and small fish in the Atlantic water layer at 100- to 500-meter depth. Diel vertical migration of this central Arctic DSL was lacking most of the year when daily light variation was absent. Unexpectedly, the DSL also contained low abundances of Atlantic cod, along with lanternfish, armhook squid, and Arctic endemic ice cod. The Atlantic cod originated from Norwegian spawning grounds and had lived in Arctic water temperature for up to 6 years. The potential fish abundance was far below commercially sustainable levels and is expected to remain so because of the low productivity of the CAO.

As usual, you can also use this squid post to talk about the security stories in the news that I haven’t covered.

Read my blog posting guidelines here.

Posted on March 25, 2022 at 4:07 PMView Comments

Friday Squid Blog: 328-million-year-old Vampire Squid Ancestor Discovered

A fossilized ancestor of the vampire squid—with ten arms—was discovered and named Syllipsimopodi bideni after President Biden.

Here’s the research paper. Note: Vampire squids are not squids. (Yes, it’s weird.)

As usual, you can also use this squid post to talk about the security stories in the news that I haven’t covered.

Read my blog posting guidelines here.

Posted on March 11, 2022 at 4:01 PMView Comments

Sidebar photo of Bruce Schneier by Joe MacInnis.