Research on the Timing of Security Warnings

fMRI experiments show that we are more likely to ignore security warnings when they interrupt other tasks.

A new study from BYU, in collaboration with Google Chrome engineers, finds the status quo of warning messages appearing haphazardly­—while people are typing, watching a video, uploading files, etc.­—results in up to 90 percent of users disregarding them.

Researchers found these times are less effective because of “dual task interference,” a neural limitation where even simple tasks can’t be simultaneously performed without significant performance loss. Or, in human terms, multitasking.

“We found that the brain can’t handle multitasking very well,” said study coauthor and BYU information systems professor Anthony Vance. “Software developers categorically present these messages without any regard to what the user is doing. They interrupt us constantly and our research shows there’s a high penalty that comes by presenting these messages at random times.”

[…]

For part of the study, researchers had participants complete computer tasks while an fMRI scanner measured their brain activity. The experiment showed neural activity was substantially reduced when security messages interrupted a task, as compared to when a user responded to the security message itself.

The BYU researchers used the functional MRI data as they collaborated with a team of Google Chrome security engineers to identify better times to display security messages during the browsing experience.

Research paper. News article.

Tags: , ,

Posted on August 22, 2016 at 7:03 AM • 32 Comments

Comments

Rien • August 22, 2016 8:48 AM

Ironic and true – you’re most likely to encounter a security warning while you are trying to get to something, and we are so used to Windows and websites popping useless dialogs up that people are already trained to “just click through” to get to the goal.

Dr. I. Needtob Athe • August 22, 2016 9:00 AM

Dual task interference? Neural limitation? Multitasking?

It sounds to me like they’re overthinking it. I consider popups that interrupt what I’m trying to do to be just plain rude and irritating, and I have no patience with them. My immediate thought is “get that damned thing out of my way!”

There has to be a better way.

Tatütata • August 22, 2016 9:15 AM

Aw, come on.

The BYU page shows the type of messages used in their study:

Chrome has detected unusual behavior

Is Chrome crashing, showing unusual startup pages, toolbars, or unexpected ads you can’t get rid of, or otherwise changing your browsing experience? You may be able to fix the problem by running the Chrome Cleanup Tool.

Run Chrome Cleanup Tool | Dismiss

Is THAT a “security warning”? To me this really looks more of an useless, optional recommendation, than an fire alarm bell requiring immediate action. (e.g.: “Battery running low. Save your work immediately and replace battery.”).

What behaviour does Chrome consider to be “unusual”? Couldn’t they be more specific. (e.g., by providing a “more information” button).

The paragraph begins with a question (“Is Chrome crashing”), and suggests an optional user action conditional to the answer.

What does the utility do anyway? Chrome imposes its own start page for new windows and tabs. Does this program try to reverse any setting I may have attempted to modify?

Sponsor bias is well documented in fields like pharma. This research was paid by Google. Were they looking for a certain result?

Chelloveck • August 22, 2016 9:18 AM

Research brought to you by the Department of Stating the Obvious. I’m trying to get something done with the computer. It pops up a security warning. Of course I’m going to be more inclined to ignore the warning. It’s standing between me and my objective. I suppose it’s a good thing to quantify and document the behavior, but it’s hardly surprising to anyone who’s ever actually used a computer.

Perhaps the security warnings need some sort of cool-down timer like Netscape introduced for installing extensions from third-party sites. You couldn’t click “Just do it, dammit, and quit bugging me!” until a 10-second timer expired. That gave you a little time to think about the implications of what you were agreeing to. I have no idea if I’m typical or not, but it was sometimes enough to get me to cancel the request after I considered it a little longer.

http418 • August 22, 2016 9:29 AM

This is great! We can just assume the user wants to take the risk now, and ask them about it later – I can’t see a down side! Or we can just make the decision for them, since there will never be outliers. Seriously though, how do you wait for the single threaded user when the mutli-threaded system needs an answer. If it was easy to assume an answer based on other variables, it should already be doing that.

fruity moon • August 22, 2016 10:05 AM

If only google knew exactly what I was doing at any given time, then they would know when to interrupt me.

oliver • August 22, 2016 10:47 AM

fMRI… really?
This is complete and utter BS!
There was even a dead salmon were “researchers” were able to get a reading with fMRI!
This technology is beyond just stupid and useless.

Gunter Königsmann • August 22, 2016 11:04 AM

The problem with warning popups that appear during browsing is that they look almost exactly like scam pop-ups that appear in the same time frame. It is more secure for most users to ignore them.

Lex Technica • August 22, 2016 11:20 AM

This is interesting because it runs contrary to modern best practices on privacy and the FTC’s minimization recommendations. Just-in-time delivery of security warnings is considered to be an improved practice over bulk disclosure on an initial privacy policy. I wonder if there is a difference in comprehension between desktop and mobile platforms.

David Leppik • August 22, 2016 11:21 AM

The real question isn’t whether interrupting a focused user with a cryptic warning is a bad idea. The real question is: when is there a good time to get the user’s attention?

Fortunately, the paper identifies several such times:

At the beginning of starting the first task

After the video

After interacting with a website

Waiting for a file to process to complete

Waiting for a page to load

These aren’t as obvious as they look. For example, “waiting for a page to load” is exactly when most such error messages occur. However, that’s distinct from what they describe as bad times for an error message, namely when a user is in the middle of typing, closing a window, or other active task.

Which implies that there is no good time to try to get the user to pay attention to something important that they’d rather not think about, but that the UI should check for keyboard/mouse action in order to avoid the WORST times.

K15 • August 22, 2016 12:00 PM

If your mobile phone throws up a message telling you “there’s malware here, don’t turn off your phone” is it trustworthy, or is that the malware speaking?

ianf • August 22, 2016 12:30 PM

David Leppik: The real question is: when is there a good time to get the user’s attention?

Acc. to Windows updates, any time is a good time.

Asked to guest-lecture at a computer-sy institution, I inquired about connecting my Mac laptop to their projector. I was told to record my slides as full-screen images on a DVD which would then be stepped through live off their already connected and faultless Windows laptop. Not wanting to make a fuss, I complied. They had a AV program to display pictures in or out of sequence, controlled by a small remote with laser pointer. It went well. I also counted 11 instances of update or some periodic backup dialogs, about half of which had to be dismissed al mano. When later I took this up with the asst. prof-tech guy, he pleaded ignorance and blamed summer students who must’ve played with it. I don’t expect to be invited there again—too much of a troublemaker.

Wael • August 22, 2016 12:48 PM

@ianf,

I don’t expect to be invited there again—too much of a troublemaker.

Imagine my shock! This news could shock an electric eel (I read this out loud next to my pet electric eel. Bad mistake, it got electrocuted!) How could they, what were they thinking?

Man! Can we give you a reference or something, it might improve your chances 😉

T. O. Salamanca • August 22, 2016 1:09 PM

A group of passengers is seated in a small twin-engine commercial plane idling on a runway. The rear door opens and the pilot and copilot come aboard. One has a seeing-eye dog and the other is tapping his way up the aisle to the cockpit with a white-tipped cane. The passengers become pretty nervous, but they only mutter and whisper to each other. After a few moments, the engines begin to rev and the plane begins its takeoff run. As it rolls faster and faster, the passengers look out the windows and realize they are approaching the end of the runway, and they all begin to scream. The pilot pulls back sharply on the stick and the plane takes off safely. After a moment, he gropes around with his right hand, finds the copilot’s shoulder, squeezes it in a friendly manner, and chuckles, “You know, Bob, one day they’re going to scream too late and we’re all going to die.”

I like to tell this one whenever someone asks me about security warnings.

paul • August 22, 2016 3:28 PM

Seems to me that another word for this is crappy UI design. When I go into someone’s office to tell them something important and they raise a hand to stop me because they’re on the phone, I don’t take that as a sign to leave in a snit and never come back to tell them that thing. But that’s pretty much what those dismissable alerts do. Either interrupt what you’re doing for an unknown length of time, or else accept whatever risk you’re running (whose magnitude you can’t even know without committing to the indefinite interruption).

Surely a bunch of brilliant UI and OS designers can figure out ways to have certain messages persist until a user has time to deal with them?

Jacob • August 22, 2016 5:58 PM

A warning message that always works, regardless of timing:

An animation of a scantily clad young female sliding down a virtual pole and whispers: “hey babe, got a warning message for you…”

Clive Robinson • August 22, 2016 6:23 PM

@ Jacob,T. O. Salamanca

A good one. Can hardy wait for the landing story.

It probably involves “wing walking”…

A quick one for you,

Q:- How does a blind parachutist know they are about to land?

A:- The lead on the guide dog goes slack…

Wael • August 22, 2016 6:52 PM

@Clive Robinson, @ Jacob,T. O. Salamanca,

This is “timing” related.

A quick one for you,

You must have a daisy of a hand! I see your “quick one” and raise you two versions of the same one!

My joke in a different context about “judgment”. And a more polite one from Wesley Parish.

@Wesley Paraish,

By the time we finally fall to five feet, we can’t stop and jump.

Are we at 5 feet now?

Tony Vance • August 22, 2016 7:34 PM

I’m one of the coauthors of this study. I’m happy to answer any questions you have.

@Dr. Athe

Our research lab is showing that the way the brain works does impact security behavior. These aren’t the only factors, but I think it’s useful for security UI designers to be aware of some of the ways the brain works against standard UI practices. You can read more about our work at http://neurosecurity.byu.edu.

@Tatütata

We actually refer to the Chrome Cleanup Tool (CCT) as a security message, not a warning. The CCT is built into Chrome for Windows and scans the host for adware/malware. If found, the CCT asks for user’s permission for legal reasons to remove the software. You can read more about the CCT in the article at the link provided.

I agree that the language of the CCT message isn’t very strong, but that’s the language that the Google Chrome team has chosen to use. Actually, I think this makes our findings more convincing, because we show that finessing the timing of the message will improve the rate that people respond to the CCT, despite its mild language.

As to sponsor bias, we did receive funding from Google as part of their Faculty Research Award program, but it wasn’t to study this problem. In fact, we actually had already performed our fMRI experiment before we collaborated with Google engineers for the second experiment. Also, both experiments were performed independent of Google.

@Chelloveck

I admit that these findings do seem obvious. But the point is that there are no software vendors that do this: wait until the user is less engaged with other things to display a security message. We show (1) how interruptions cause interference in the brain, and how this negatively impacts responses to security messages, and (2) how using good timing can dramatically improve users’ responses to security messages.

@oliver, @Cinaed

MRI is the single-most transformative imaging technology in modern medicine. A single bogus study doesn’t invalidate the method.

But, even if you disregard the fMRI experiment, the second half of the paper describes a typically usability experiment in which actual Chrome users browsed the web on their own computers. This is the experiment that shows the most dramatic results for the timing of the CCT security message.

@Lex Technica

I agree, our results do seem to run counter to the just-in-time delivery of security and privacy messages. We’re interested in examining users’ response behavior on mobile devices.

@David Leppik

You are exactly right. The user is rarely, if ever, doing nothing on the computer. Our goal was to avoid the worst times to display the security message. This made big positive difference.

Some Guy • August 22, 2016 9:17 PM

@Tony Vance

Good Paper. As a person that is doing research in a different area, a comment on sponsor bias. The below section in the paper raises that concern. I do not see it as an issue based on the methodology details described in the paper of what was specifically being studied.

The concern is that the Chrome security engineers could have a hidden agenda in the design in either direction, either for job protection or increased funding. Regardless, thank you for clearly and openly disclosing this where relevant and not just under acknowledgements.

“For this study, we collaborated with a team of Google Chrome security engineers who develop the CCT—a security message that can be delayed—to identify five low-DTI times to display security mes- sages during the browsing experience. These times were selected according to (1) DTI theory and the results of fMRI results of Experiment 1, (2) input from Google engineers on moments that were frequent in occurrence and generalizable across a wide variety of web-based activities and users, and (3) a feasibility assessment for implementing in a Web browser.”

unacknowledged_studies • August 22, 2016 11:30 PM

@Tatütata

has detected unusual behavior

A new study from BYU, in collaboration with Google Chrome engineers, finds the status quo of warning messages appearing haphazardly — while people are typing, watching a video, uploading files, etc. — results in up to 90 percent of users disregarding them.

…

“We found that the brain can’t handle multitasking very well,” said study coauthor

Most of the comments here are completely worthless (obviously aside from, hey — you!! Get onto my cloud;)

Tony Vance • August 22, 2016 11:54 PM

@Some guy

Good point. My field of information systems, as a management discipline, is very applied, and it is highly desirable to collaborate with practitioners. So it’s interesting to hear your concern about sponsor bias. I agree that this could be an issue in some studies.

In the case of our study, we did collaborate with the engineers to determine the high-DTI and low-DTI times, your quote highlights. However, the rest of the experiment and analysis was conducted independent from the Chrome engineers.

Wael • August 23, 2016 12:09 AM

@Tony Vance,

I believe it’s an innovative research that targets an area often taken for granted. This area never even crossed my mind (doesn’t say much, but true.)

So it’s interesting to hear your concern about sponsor bias

farmer wants to milk a cow in the barn. He puts his stool next to the cow and the bucket below it. As he starts milking the cow kicks the bucket with the left leg. After a while the farmer is pissed off and takes a rope to fix the cow’s leg to a pole. Yet before he can start milking again the cow kicks the bucket with its right leg. So the farmer fixes the right leg to another pole. The farmer wants to start milking again, but now the cow is slapping him with its tail. Wanting to fix the tail to a roof beam he puts the stool behind the cow and steps on it. Having no rope left, he takes off his belt to fix the tail. So as he’s holding the cows tail in one hand, his belt in the other his pants drop down right before his wife enters staring at him. He just says: “You know, sometimes things are hard to explain”

You think the wife will believe him? 😉

Danny • August 23, 2016 4:41 AM

@Tony Vance

This is doubly useful. I know you’re looking at ways to improve the effectiveness of security messages, but I’m just as interested in improving productivity everywhere else. Every message which demands my attention while I’m working on a problem makes it that much harder for me to solve that problem, because it displaces my focus. It’s the same reason why it’s a bad idea to have random meetings sprinkled throughout the day, or to be answering text messages, etc. Hard problems require serious focus, so allowing users to keep their focus by minimizing interruption through better timing will hopefully improve productivity as well.

Tony Vance • August 23, 2016 11:48 AM

@ vas pup

No, MRI works by placing a single person into a bore of a giant electromagnet. Also, it works best with a controlled experiment in which you can compare brain activity among experimental conditions.

Leave a comment

Login

Allowed HTML <a href="URL"> • <em> <cite> <i> • <strong> <b> • <sub> <sup> • <ul> <ol> <li> • <blockquote> <pre> Markdown Extra syntax via https://michelf.ca/projects/php-markdown/extra/

Sidebar photo of Bruce Schneier by Joe MacInnis.