More AIs Are Taking Polls and Surveys

I already knew about the declining response rate for polls and surveys. The percentage of AI bots that respond to surveys is also increasing.

Solutions are hard:

1. Make surveys less boring.
We need to move past bland, grid-filled surveys and start designing experiences people actually want to complete. That means mobile-first layouts, shorter runtimes, and maybe even a dash of storytelling. TikTok or dating app style surveys wouldn’t be a bad idea or is that just me being too much Gen Z?

2. Bot detection.
There’s a growing toolkit of ways to spot AI-generated responses—using things like response entropy, writing style patterns or even metadata like keystroke timing. Platforms should start integrating these detection tools more widely. Ideally, you introduce an element that only humans can do, e.g., you have to pick up your price somewhere in-person. Btw, note that these bots can easily be designed to find ways around the most common detection tactics such as Captcha’s, timed responses and postcode and IP recognition. Believe me, way less code than you suspect is needed to do this.

3. Pay people more.
If you’re only offering 50 cents for 10 minutes of mental effort, don’t be surprised when your respondent pool consists of AI agents and sleep-deprived gig workers. Smarter, dynamic incentives—especially for underrepresented groups—can make a big difference. Perhaps pay-differentiation (based on simple demand/supply) makes sense?

4. Rethink the whole model.
Surveys aren’t the only way to understand people. We can also learn from digital traces, behavioral data, or administrative records. Think of it as moving from a single snapshot to a fuller, blended picture. Yes, it’s messier—but it’s also more real.

Posted on May 21, 2025 at 7:03 AM21 Comments

Comments

Joe May 21, 2025 9:06 AM

If a survey or survey link is hosted by a third party, don’t do it. If it is hosted by the company that wants results but has many offsite scripts, don’t do it. If the company wants survey results but the questions they ask shows a thinking person that they will likely be used to increase prices or decrease service, don’t do it. If the survey demands positively identifying you (not just that you are a customer and haven’t already answered the survey) and you don’t trust the company not to store or forward to a company that will add it to their dossier (e.g. Lexus Nexis), don’t do it.

Come to think of it, I can’t recall many surveys that I answered because they don’t show me any respect for my privacy or give me reason to trust them.

Not really anonymous May 21, 2025 10:29 AM

I’ve starting making doctors appointments using *67 (this blocks CID, but not ANI) because my provider didn’t understand that just because someone makes a call from a phone, that doesn’t mean they control it and they shouldn’t be texting surveys to every phone that calls them. The ironic thing is I verbally give them suggestions from time to time on how to improve their service (for me at least) and so far they have been ignored.

Clive Robinson May 21, 2025 11:46 AM

@ Bruce, ALL,

When I get accosted by “survey takers” be they electronic or human I know one thing,

“They will not pay what my time or knowledge / insight is worth.”

And as they know this before they even approach me, Their obvious intent is at best to rip me off or in some other way rob me.

I’m reasonably certain that others feel that way not just me.

So the suggestion of,

“3. Pay people more.”

Is going to be at best ineffective.

It’s even more interesting when you deal with “focus groups” or what I call “Bogus groups” because the members are selected by their likely hood of being “yes-men”.

To see why you have to realise that the whole purpose of such groups is to say that what somebody already wants to do is a good idea… Not tell them the truth in plain unvarnished words they should be able to understand.

The organisers of such groups know that their income long term is based on their ability to get not “honest answers” or “correct answers” but “What we want to hear answers” of the people paying for the group to be set up…

I’ve had the misfortune to be the equivalent of “head hunted” by people wanting to get my thoughts on the cheap, it’s why I don’t give out information I can be traced by, nor do I respond to attempts by people to “get friendly” or equivalent beyond the boundaries I set.

I think one of the funniest ways I’ve ever seen to avoid being hunted is the “Eric Cantona” response.

For reasons that are not really relevant he got mobbed by a very large pack of journalists, doing what they can do, which if you’ve ever been on the receiving end can be quite scary.

They were asking some very stupid questions so Eric gave what is considered one of the most enigmatic answers,

“When the seagulls follow the trawler”

And left them completely baffled (though if you’ve ever seen the seagulls descend as the nets get hauled in… you know exactly what he ment).

It was not long after that, that he retired from football for the second time, arguably when he was just coming into his best performance. It left people not just surprised but shocked. He eventually pointed out that,

“When you quit football it is not easy, your life becomes difficult. I should know because sometimes I feel I quit too young. I loved the game but I no longer had the passion to go to bed early, not to go out with my friends, not to drink, and not to do a lot of other things, the things I like in life.”

So yeh the point is,

“Do you need that nonsense?”

And my view is not just “What’s in it for me?” But more importantly “Are there better things I could be doing?”

And that’s when you realise there is not enough money to answer those questions in the way they want.

So rather than be a “trawler” be a “sail boat gracefully heading for distant horizons via golden sunsets”.

Kerwin May 21, 2025 12:20 PM

ALL current public opinion polls are statistically invalid and do NOT measure what they boldly claim to measure. No fix is possible — polling is huge scam.

The ‘Non Response Rate’ is a catastrophic/fatal deficiency in all malor polls — typically over 90% of a poll’s initial people “Sample” do not participate and their views are unknown to pollsters & simply ignored totally.

Several other big error sources in polls are likewise ignored.

Objective scientific “Probability Sampling” NEVER happens in mainstream polling.

mark May 21, 2025 1:23 PM

I agree with Kerwin. They do not pay anyone. Then there’s the points I’ve been complaining about for decades: first, that they pick and choose who they call by the LEC – the first three digits, which chooses the neighborhood, or people who used to live in that neighborhood. Doubt this? Ask anyone who’s lived in a lower-class or poor neighborhood, whether they have ever been polled. Certainly, when I lived in such, I was never polled.
Secondly, the polls themselves are garbage. The questions are written in such a way that so that they can get the results the people paying for the polls want. They never offer alternative that are more than mild variations… or there’s the “pick the most important”, when you want half a dozen of the 8-10 choices. Like “fund schools or medicaid or public transit or…”

lurker May 21, 2025 1:50 PM

Most current polls require a response along a scale of agree<->disagree, or choose one of the following, with nowhere for corroborating evidence of why you agree or disagree with the preset question, which may appear oblique or irrelevant to the topic being polled. This looks like an ideal playground for the current crop of AI, and I would encourage more AIs to partake in polling. Eventually both polls and AI should die in their own slop.

Mike May 21, 2025 6:18 PM

The survey is always for a reason. Somebody is paying money for this survey. It is solely for their benefit, never yours or mine. The questions are geared to create the responses they want. They’re never open-ended. It’s never “What would you like?” It’s always “Waste your time to justify this decision they’ve already made.”

Then there are these people who email me links to some third party website… I don’t know what they’re thinking, but they’re nuts if they think I’m going to click on an emailed weblink that’s indistinguishable from phishing.

Between 6% and 30% of people infected with Covid go on to develop Long-Covid. It’s nasty. I had it. Bad. Mentally & Physically, my mind & body were shot to hell. Still are, although I’ve improved a little bit. These survey questions. They think it takes seconds to answer. With Long-Covid, it’s days or weeks of struggling PER QUESTION! And that’s if I’m not doing anything else for those DAYS or WEEKs. The workload involved in completing these surveys is completely unreasonable. And there’s no way to tell the survey-takers that.

Some Guy May 21, 2025 6:24 PM

I once worked for a company writing some software… the User Interface (UI) and User Experience (UX) was kind of terrible, and we who worked there who knew what to do about it weren’t allowed to fix it, because management wanted it that way… Since management wouldn’t listen to us, we finally convinced them to hire a very expensive expert outside company to evaluate the software, and provide suggestions on how to improve it.

Of course the list of improvements was long. That’s what they were hired to produce after all, is a long list. It could have been perfectly well designed software and they’d still have given you just as a long of a list (just not with as many severe or major level items on it).

Management took one look at that and decided to throw the software out and start over with brand new software (and with different people working on it too). We warned them that was a waste, not only of our time building the first version of software, but also a waste hiring the company to tell them their software was awful… look, you have a precise list, just focus on fixing the top X many issues on the list, and you’re pretty good! You start over and now you need to hire them all over again….

So how does this relate to polls and surveys? We weren’t good “yes men”… so we were not appreciated nor wanted as employees. The company hired wasn’t a good “yes man” either, so they weren’t appreciated nor their advice heeded. All management wanted to hear was a “you’re doing great” response… anything short of that, they were ready to go to any lengths, waste any money or time, etc… It’s like a kind of poll: right answer, we keep going, we’re doing great! Wrong answer: heads will roll. This is the mentality of those conducting polls and surveys.

The option to pay people more is interesting. People’s time may indeed be worth a lot more… but that also gives an even greater incentive for AIs to take over, because then there’s more money in it… Someone is managing all those AIs and collecting the payment however minuscule it may be per poll–that’s why the AIs were written, to make money for someone, and little bits always add up to a lot given enough scale.

MATTmac May 21, 2025 7:58 PM

Paying people to respond to surveys destroys the fundamental basis of scientific polling — the Random Sample.

Accurate polling samples a small number of people from a much larger group (say, adult Americans or Blog readers), but EVERY member of this larger group MUST have an equal probability of being in the selected sample — that’s what a Random Sample is.

Anything that causes some people to be more (or less) likely to complete a survey … destroys the necessary randomness of poll’s methodology.

Small amounts of money might motivate some but turn off others … and there’s never a guarantee of a an honest resonse from anybody surveyed.

Clive Robinson May 21, 2025 11:11 PM

@ Bruce, ALL,

With regards number four on the list of “Rethink the whole model.”

What is suggested is,

“We can also learn from digital traces, behavioral data, or administrative records. Think of it as moving from a single snapshot to a fuller, blended picture. “

This is also a “full fit” description of both,

1, Oppressive surveillance.
2, Full invasion of Privacy.

The statement of,

“Yes, it’s messier—but it’s also more real.”

Leaves out,

“It gives nobody choice, it’s more invasive, and psychologically damaging of society due to the ‘chilling effect’.”

Which means it’s very far from being “more real”. Because people when they are observed or think they are observed change their behaviours. It was the basis for Jeremy Bentham’s “Panopticon Prison” to force behavioural modification.

But also consider how you behave is not really dictated by “free will” but by

1, Your available Resources.
2, Mores of current society.

It’s actually why Guard Labour organisations tend towards “barracks” for “Ordinary Ratings”(ORs) and individual rooms for “Junior Officers” with “Senior Officers” given the equivalent of a private area and a social area and a servant.

We see this reflected in business / corporate “offices” with “open office/floor” for the lowest level worker drones, rising through cubical space to individual offices for more senior workers or lower management. With more senior management having a private office, with attached meeting room and an office for a personal assistant / gatekeeper.

Due to “lockdown” we know that “return to office” is not about productivity as it causes it to drop. Nor is it about efficiency or cost reduction, it’s all about “personal power” in a “hierarchy”. Along with forcing a “culture” onto individuals for the sake of conformity and depersonalisation. Such that workers are not seen as people but “Units of work resource”.

So,

“Digital traces, behavioural data, or administrative records”

All they tell us about the past, and the inflicted power structures, not about what the future can bring, or what people want.

None of us would have any of the benefits of modern life if we had observed the behaviour of slaves or worse serfs and assumed that is how people wanted to live. We make the same mistake with modern “consumerism”, “office work”, and “religion”.

As I point out occasionally there are two lies we get told,

1, Divine Right / Might is right.
2, God made man in God’s likeness.

The truth is,

Firstly,

“Individual Rights v. Social Responsibility”

Is a scale and the more someone leans towards “Individual Rights” the more “self entitled”, and “authoritarian” they are and the more useless they actually are. Also they tend towards if not directly suffer from harmful mental traits.

Contrary to what they might think or espouse the “self entitled” are critically dependent on others for their survival. It is this that in effect requires them to be “authoritarian” and why they tend to “think with their fists or equivalent” thus push a “might is right” agenda.

Secondly,

To support “might is right” authoritarianism is the need for not taking any responsibility for your actions. Thus the “only following orders” excuse. But if you are at the head of the hierarchy who do you say you were taking orders from? You simply create an imaginary omnipresent deity that gives you “go forth” / “make it so” orders. Thus you can take credit for successes and blame the deity for the failings being part of a longer term plan mere mortals can not see…

The joke is of course,

“God made man in his likeness”

Is the wrong way around… It’s actually,

“Man made God in his likeness”

And in the process created a “patsy” for people to blame for their woes, troubles, and failings…

The fact people still fall for this tells you that they must have been indoctrinated at an age where they had not developed the necessary “mental defences” and in later life still do not want to take on either “Personal or Social” responsibility for their actions and behaviours.

So,

Do you want a world based on such failings as we can easily observe all around us?

For most of us the answer is probably not.

But by using,

“Digital traces, behavioural data, or administrative records”

It is what we are going to get.

It’s one of the major failings of current AI LLM and ML systems, their output is constrained by their input training data which is always a “past statistical norm” there is “no future” with LLMs because they are trained on an error function based on the “past” encoded in the input data.

This can only go one way, which is “retrograde” or “downward spiral” because each time around the loop you,

“Integrate for the lowest common denominator.”

A look around our natural world / environment tells us what will happen if we slavishly follow this path…

Anonymous May 22, 2025 2:21 PM

The surveyor and the surveyed don’t have interest in common. Unless that is overcome, it is broken and it is supposed to be broken.

Thumbo May 22, 2025 7:52 PM

@ Clive,

Where are your blog and books?
I hope to get off my laurels one day and scrape the archive of these comment sections into an ebook to peruse per posterity. “Musings of Bruce and Friends”
I don’t immediately agree with everything you add here but enjoy gleaming from your brilliance my weary mind can’t distill on my own time.

Adding to what everyone else mentioned I know multiple people that would plow through paying surveys while waiting around with quick passable junk input just to make a bit of pocket change for the holidays, paying for surveys won’t work.
Like said above more pay just encourages more BS human/equivalent AI input.

Need a creepy carrot and stick of paying more/making less boring and some kind of non-anonymous the man is watching over your shoulder, modifying true opinion but discouraging BS input like someone who knows you shaming some honesty reading alongside.

ResearcherZero May 23, 2025 5:27 AM

There are very good reasons for listening to the people doing the work. The employees.
Despite the fact that those producing the actual product having insight, skill and knowledge to produce said product, there continues to be a flow of articles wondering why productivity has stagnated. Management and businessmen ask the same question. Why?

When ‘management’ does not like the expert advice it hires outside consulting. If that reaches the same conclusion they still blunder along as if they learned nothing at all.

Rather than listen to those that tackle the very real issues every single day, business and government hires outside help and instead listens to the pitch from political appointees and lobbyists who have absolutely no experience in the given fields in question.

Some things are destined to be learned the hard way. Very f–king slowly after an enormous amount of pain and regret, or perhaps not at all. We’ll see what the polls say?

To every cloud there is a silver lining. Maybe not for you. It depends on your perspective.

‘https://edition.cnn.com/2025/05/22/us/harvard-university-trump-international-students

A large share of U.S. researchers are considering leaving.
https://hub.jhu.edu/2025/05/21/us-research-brain-drain-uli-mueller/

Countries are scrambling to increase academic brain power as America abandons its talent.
https://www.business-standard.com/world-news/global-race-to-lure-us-researchers-intensifies-after-trump-cuts-funding-125042800101_1.html

Students are also looking abroad after political appointees were put in charge of purse.
https://www.nbcnews.com/science/innovation/best-brightest-young-scientists-are-looking-us-cuts-hit-home-rcna205207

Ronda Davidson May 23, 2025 8:49 AM

Welcome to the future of surveys: where half the respondents are bots, and the other half are just doing it for pocket change.

Grima Squeakersen May 23, 2025 1:18 PM

“…is that just me being too much Gen Z?”

Is there some subtlety to this question that I missed? Gen Z is typically considered to consist of those born from 1997 to 2012. Bruce Schneier was born in 1963…

Grima Squeakersen May 23, 2025 1:25 PM

@mark re: LEC representing neighborhoods. That was no doubt once valid, but in this day of ubiquitous cell phones available anywhere from various services, it can no longer be relied upon much, probably not at all.

Grima Squeakersen May 23, 2025 1:34 PM

I agree with the thoughts of many others on this topic. These polls are nearly invariably not instruments intended to objectively measure opinion statistics. They are designed by some third party to echo a predetermined POV on some question or other. In most cases, the desired POV is chosen by the contractor as part of the deal, although the desired result may commonly be conveyed implicitly, to avoid PR and legal consequences. In a few cases, the contractor may naively expect an honest result, but the sales people of the contractor will be well versed in how to determine what the contractor wants to hear, and will build that in, anyway. Completing one of there is a total waste of time, at best.

Ulf Lorenz May 23, 2025 4:14 PM

further ideas:

  1. Be aware that people have better things to do with their lives than filling your surveys all the time, so cut the questions/surveys to what is really needed. Before asking a question, be clear what information you want to get out, and phrase it carefully. Double-check for framing. In short: Put some love into your surveys.
  2. Now that you are down to the basics: Spend appropriate money / effort on your survey. For example ~10% of the population may be functional analphabets, writing will not catch those. The more effort you spend, the better the results. Well-planned direct engagement might boost response rates (e.g., asking during a PT ride).
  3. Highlight the social value (e.g., public planning) of this particular survey, this may boost response rates and engagement.

Of course, all this assumes that the survey is important and has significant social value, which may be the elephant in the room here.

Clive Robinson May 23, 2025 6:24 PM

@ Grima Squeakersen

With regards Bruces comment about Gen Z.

Gen Z are reputed to have a certain outlook and social reputation that defines them.

I think you might find that Bruce was indicating he might appear to some to be emulating to some extent that Gen Z outlook and demeanor.

Gen Z are also known as the iGeneration (from a song title) and were the first generation after the WWW got going. For some reason they hit adolescence and puberty considerably younger than previous generations, and various explanations were put forward such as increased protein in the diet, increased refined chemicals in foods, and the one that “got the alligators shrunk naughty bits” from birth control chemicals and similar flurocarbons in the drinking water[1].

That said Gen Z had a much lower rate of teen pregnancy, but they were the generation that apparently invented “sexting”.

More interesting from the perspective of surveys is that Gen Z was when young pre-graduate “strongly left wing in outlook” but… “With age and experience post lockdown have become more and more right wing”, some “rabidly so” or have given up on the political process altogether.

Some have said that Gen Z got caught by the continuing shift into the knowledge economy and Covid lockdown by interrupted education. With some two or more years older than average for their University year. Worse that some also trained up for more traditional employment of their parents –via apprenticeships– and so did not understand technology like two or three generations before, even though they use technology a lot (more like being a driver not a mechanic so stuck with lower socio-economic rung gig-work). Thus in many places as more traditional jobs disappeared they feel they have been left without the life opportunities of previous generations. Thus some have became fair game for the MAGA nonsense that has zero chance of giving them employment prospects of any worth…

[1] There is some evidence for the artificial hormones in the water supply. That go back to the start of Gen Z,

https://www.miaminewtimes.com/news/see-you-later-alligator-penis-6362052

As you might know often the water you drink is taken out of a river upstream of your town and the human waste such as urine with minor cleanup goes back in the river downstream of your town. But that is also upstream of the next town down so… It’s been said of some places that the water you drink has come through six or seven bladders before you get to drink it… It also has been suggested that it might be the reason bra sales apparently go up in cup sizes as you go down stream…

JTC June 15, 2025 9:45 AM

My guess would be political polls are the toughest. Those who design such polls have to not only do their best to make questions as neutral as possible, but they also need to allow more non-traditional positions. For example, I have seen too many where all they care about is whether you lean conservative and/or Republican or liberal/Democrat. Everyone doesn’t fit neatly into their mold, but clearly they don’t care about anyone else. I expect political poll data to continue to be the most challenging and the least reliable.

Leave a comment

Blog moderation policy

Login

Allowed HTML <a href="URL"> • <em> <cite> <i> • <strong> <b> • <sub> <sup> • <ul> <ol> <li> • <blockquote> <pre> Markdown Extra syntax via https://michelf.ca/projects/php-markdown/extra/

Sidebar photo of Bruce Schneier by Joe MacInnis.