Manipulating Machine-Learning Systems through the Order of the Training Data
Yet another adversarial ML attack:
Most deep neural networks are trained by stochastic gradient descent. Now “stochastic” is a fancy Greek word for “random”; it means that the training data are fed into the model in random order.
So what happens if the bad guys can cause the order to be not random? You guessed it—all bets are off. Suppose for example a company or a country wanted to have a credit-scoring system that’s secretly sexist, but still be able to pretend that its training was actually fair. Well, they could assemble a set of financial data that was representative of the whole population, but start the model’s training on ten rich men and ten poor women drawn from that set then let initialisation bias do the rest of the work.
Does this generalise? Indeed it does. Previously, people had assumed that in order to poison a model or introduce backdoors, you needed to add adversarial samples to the training data. Our latest paper shows that’s not necessary at all. If an adversary can manipulate the order in which batches of training data are presented to the model, they can undermine both its integrity (by poisoning it) and its availability (by causing training to be less effective, or take longer). This is quite general across models that use stochastic gradient descent.
Research paper.
Frank Wilhoit • May 25, 2022 12:50 PM
The fact that machines are “better” than humans at certain aspects of certain things leads, via sloppy and magical thinking, to the notion that they can somehow solve problems that we can’t solve (and therefore can’t teach them to solve).
This is magical thinking. The harder(*) a problem is, and therefore the greater the theoretical benefit from automating it, the harder it is to program the automation.
Brass players have a saying: “If you can’t play it on the mouthpiece, you can’t play it on the horn.” We have enough trouble playing “accounting manual” on the mouthpiece, and that is why it sounds so bad on the horn. We are nowhere near, just to take one conspicuous example, to being able to play “self-driving car” on the mouthpiece.
On some level, everyone knows this, and that is why the purpose of most technology adoptions is not to solve problems but to obscure responsibility.
(*) Yes, I know that there are many, profoundly distinct kinds of “hardness”. I think it is clear from context which ones are involved here.