The error rate for hand-counted ballots is about two percent.
All voting systems have nonzero error rates. This doesn’t surprise technologists, but does surprise the general public. There’s a myth out there that elections are perfectly accurate, down to the single vote. They’re not. If the vote is within a few percentage points, they’re likely a statistical tie. (The problem, of course, is that elections must produce a single winner.)
Gorgasal • February 7, 2012 6:26 AM
@Vince Mulhollon: there have been quite a few studies on the accuracy of system inventories in retail. They usually find that around 40% of system inventories are accurate… and the other 60% over- or underestimate the amount of product actually on the shelf.
And misscans are also rife. Less so with barcoded CPG, more so with fruits & vegetables where the customer or the cashier actually has to identify the item he is weighing.
Don’t be too sure about accuracy in retail.
James R • February 7, 2012 6:27 AM
I can believe it, at least in the UK. Many options, many votes, and they need to be done overnight.
Also, most results are more than 4% clear, so it doesn’t matter so much. It gets interesting when you get into various local legislations on how small a gap is needed to get a recount.
syberghost • February 7, 2012 6:27 AM
Yes, it was funded by somebody with an axe to grind; The National Science Foundation. Their axe is people disbelieving science if it doesn’t meet their preconceptions.
bkd69 • February 7, 2012 6:57 AM
That was something that always bugged in reports and discussion about the Florida element of the 2000 POTUS election…every report of every poll leading up to the election includes the magic incantation ‘margin of error,’ and yet, people can’t seem to understand that a) an electorial poll would also have a margin of error, and b) that there quite possibly be a result that falls under that margin of error.
Terry Cloth • February 7, 2012 7:08 AM
This supports my thesis that the Electoral College is a good thing. Remember Florida in 2000? Imagine what it would be like if we had to recount every vote across the nation. In a close election, we’d never come to a conclusion accepted by more than 50% of the electorate.
What does need changing is the winner take all'' system—the major cause ofminority presidents”. A state’s electoral votes should be distributed proportionally to the vote in the relevant district (with the odd two going by statewide count).
Tomasz Wegrzanowski • February 7, 2012 7:46 AM
There’s no way in hell this 2% number is even remotely close to accurate.
Not even within order of magnitude.
Paul Renault • February 7, 2012 7:48 AM
1) Did this study only examine US elections? (A feudal patchwork of poorly-trained and -controlled patchwork of electoral commities, from what I’ve read.)
2a) If the margin of victory is substantially greater than 2% or even 5%, then the error rate makes no difference to the outcome, since, in the US, elections are first past the post.
2b) What is the error rate for elections where the margin of victory was 4% or less?
aikimark • February 7, 2012 7:52 AM
you don’t have to recount everything. Several years ago, someone did an analysis and found that you only needed to recount up to the point where the error would change the election results.
If I can find that article, I’ll post its link.
========
Although this Penn State professor has some good ideas about moving away from the (current) electoral college, I would want some protection against political party influence in redistricting.
http://www.physorg.com/news123434587.html
Tom • February 7, 2012 7:54 AM
My question is, how do they actually know this?
Whatever method they are using to determine the ‘real’ results will have an error rate as well. How do they know what this error rate is? Do they know that this error rate is less than that of the system they are trying to check?
I’m not saying this is an impossible thing to do, but I’d be fascinated to read the details of the process.
StylePolice • February 7, 2012 7:57 AM
everybody: relax!
The article states ‘can result in error rates of up to 2 percent’. Ask your broadband provider for the meaning of up to…
Stefan • February 7, 2012 8:00 AM
It doesn’t really matter, as long as the errors are unsystematic: They will cancel each other out. In the aggregate you’ll have an estimate quite close to the real result.
I’ve myself counted on several elections (in Germany) and I don’t consider the results 100% percent accurate. But 2% error rate (in the German system) seems to me somewhat high-fetched. I would estimate it more to 0.5%.
tim • February 7, 2012 8:04 AM
Elections are one offs. Every single time.
Whereas in retail or the popular example “removing money from an ATM machine” that fans of computerize voting machines bring up – you have plenty of time and opportunity to identify issues and resolve over time. And even after optimizing that process there is still an acceptable degree of fraud or error that is accepted (and even expected).
In an election – you just can’t say – oops we screwed up! lets just do this again tomorrow night!
Edmund in Tokyo • February 7, 2012 8:04 AM
What does that 2% number mean? Is it that 2%of ballots are somehow misassigned, or is it that the final count is off by 2%?
I couldn’t tell from the press release, but presumably if the result was out by 2% that would imply that considerably more than 2% of ballots were mis-counted, as a lot of the mistakes will cancel each other out.
Since the 2% number sounds quite big already, I’m guessing it’s the number of ballots that went wrong somehow, and the result was out by less than that.
James Sutherland • February 7, 2012 8:04 AM
The real key is “keep it simple” and apply technology in the right way. Vote by filling in a box or circle in pen. One vote per piece of paper, no other markings.
Now sort into piles – that bit can be automated quite reliably, and checked manually. Weigh each batch to be sure it’s 1,000 votes, not 999 or 1,001, look through to be sure they are all votes for A not B.
One problem with the Florida recounts, as I recall, was that the repeated manual handling could dislodge the “chad” from a ballot, changing or invalidating a vote: each recount could only be less accurate and reliable than the last.
A “runoff” system probably helps too: weed out the no-hope options quickly, where it doesn’t even matter if the protest candidate got 1% or 2%, then a simpler 2 or 3 horse race for the finish.
Tom • February 7, 2012 8:06 AM
Stefan: “It doesn’t really matter, as long as the errors are unsystematic: They will cancel each other out. In the aggregate you’ll have an estimate quite close to the real result.”
This is why I am opposed to the so-called ‘Hawkeye’ system in tennis. Line judges have an error rate. Hawkeve has an error rate (although most likely lower than that of the line judges).
As long as the errors are, as you put it ‘unsystematic’, then everything will even out in the end, so what does it matter whether the error rate is 2% or 0.5%?
Everybody assumes that just because a computer is involved that its output is the voice of God, giving us the absolute gospel truth. Wrong!
I would rather have an extra percentage error in calls than have to stop play every ten minutes for a challenge that drags on for eternity.
Ok, maybe that was slightly off topic. Sorry.
CGomez • February 7, 2012 8:11 AM
For the doubters, also consider that error rates in counting of ballots has much to do with the problem of creating unambiguous voting systems. Of course, the perforated punchcards Florida is famous for were already known to have problems (as stated by the FEC years earlier), even systems where you fill in bubbles or make fully punched holes can have issues.
I think this is a different issue from whether there should be a paper record at all. I still believe there should be. I don’t mind the use of electronic voting machines, but I think they should produce something… anything… that the voter can refer to him or herself to understand how they voted. This is not inexpensive, but I still think it is necessary.
We may not be able to beat the error rate, but we still would like to give a hand recount the best shot we possibly can and then be satisfied with the result.
Stefan • February 7, 2012 8:22 AM
Tennis ist different from voting: One Error affects all following actions. Therefore a lower error rate can make a game much fairer. I personally don’t like to mix computers and sports, but that’s more for aesthetic reasons ;).
In voting, it doesn’t really matter if the error rate is 2% or 0.5%, because the possibility of every count being slanted in the same direction is very low.
On the other hand, computers are prone to systematic errors. Therefore i much prefer human counting to machine counting.
llewelly • February 7, 2012 9:09 AM
“Ask someone from retail management if 2% failure rate in the cash
drawer or a 2% failure rate in invoicing or a 2% failure rate in
inventory is acceptable, even amongst overworked, underpaid,
unmotivated, undersupervised low level employees. ”
Cashiers count their till every day. Volunteer poll workers count
votes once per year. If they volunteer every year. Many only volunteer
every 4 years, or less often.
mikeash • February 7, 2012 9:11 AM
@Terry Cloth
I think you’ve failed to take into account the fact that, while the electoral college reduces the number of votes that have to be recounted in the event of a close result, it vastly increases the probability that a close result will occur. Instead of a single national stage, you now have 51 (including DC) local stages which can individually be close.
If we had had a direct popular vote for president in 2000, there would not have been a recount at all, and Gore would have been declared the winner by a close but undisputed margin.
Tony • February 7, 2012 9:36 AM
Sounds plausible – it is well known that humans are not good at repetitive tasks. Comments that these numbers sound crazy high are perhaps thinking about the process with just a few dozen ballots – taking a few minutes to complete. Instead think of thousands of ballots taking many hours.
Wayne • February 7, 2012 9:47 AM
What I would like to see, for Presidential elections that is, the winner of a Congressional district gets the electoral votes for that district. No winner take all from a given state.
Doesn’t deal with the issue of error, never going to fix that.
OnTheWaterfront • February 7, 2012 10:16 AM
It would be interesting to study the error rates of electronic voting systems.
Auntie • February 7, 2012 10:35 AM
Voterdome. Two candidates enter, one candidate leaves. Also, we’d get to watch them go at each other with chainsaws and pikes and the like. Oh, wait, that’s what we have now…..
NobodySpecial • February 7, 2012 10:35 AM
Canada has an automatic recount if the difference is 0.1% (1/1000). In Canada and the UK you can request a recount if the vote is ‘close’ (not defined)
Interestingly if there is an absolute tie the voting office has to settle the matter there and then – by some ‘fair’ means.
eg drawing straws ! http://www.bbc.co.uk/news/uk-england-manchester-13305738
NobodySpecial • February 7, 2012 10:38 AM
This should be easy research to reproduce. Anywhere that there was a recount gives you at least an estimate of the variation in counts – even if a standard deviation from a population of two is stretching things a little !
Keith • February 7, 2012 10:45 AM
2% at first attempt. But if a result is close, the votes will be recounted in most systems, which will likely reduce that error rate significantly. In many systems, they’ll keep recounting until both candidates are satisfied that the result is correct (even if the numbers aren’t exactly right).
SM • February 7, 2012 10:45 AM
I wonder how the complicated ballots in the US affect things. In Canada, ballots are normally a slip of paper with four or five circle/name/party tripples on it. There are provisions for mandatory recounts when the margin is below a certain percentage, and candidates can call for an optional recount if the margin is small.
Richard Harper • February 7, 2012 11:02 AM
In certified audits you adjust the level of scrutiny/accuracy to fit the level of significance (material v. de minimis) of the dollar amounts involved. In companies with huge inventories relative to the rest of the balance sheet, accuracy is paramount. Applying to elections — close elections require greater accuracy. Also, the run-off elections makes sense.
john Henry • February 7, 2012 12:11 PM
In Puerto Rico we vote manually. A penciled X on a paper ballot. 3 ballots, each a different color for state, district and municipal posts. We get 80% turnout, about 3mm votes cats typically.
Polls close at 2PM and we typically have fairly final results by 8PM or so. Hand counted. “Fairly final” in that the results almost never change but there is a checking/certification process that can take a week or so and some of the counts may change slightly.
I’ve been here for every election since 1972 and, at the governor level, results are typically very close. 2-3% is considered a landslide.
Except for one election (1980) I don’t recall ever hearing any serious questions about accuracy of vote counts or any other shenanigans.
I agree that they will never be dead nuts but 2% seems awful high to me in a well designed and run system.
We also have a voter ID card that, by law, can’t be used for anything else. Lots of counterfeit protections built into it. No card, you don’t vote.
John Henry
NobodySpecial • February 7, 2012 12:54 PM
The report quotes 0.5-1% for ‘read and count’ and 1-2% for sort and bundle but it doesn’t say if these are overall errors or spot-checks.
In sort+bundle (the UK system) the ballots are sorted into each candidate then bundled – typically into 100s – the bundles are then counted. The overall number of ballots in each box is checked against the number handed out at the poll (to prevent ballot stuffing) so it’s unlikely that the total count is out 2%.
I can imagine that spot checks of 100 ballot bundles are out by 1-2 papers, but assuming no deliberate fraud you would expect there to be an equal number of bundles over and under and the same error to apply to all candidates.
Brandioch Conner • February 7, 2012 1:03 PM
I think everyone needs to read the article.
“Based on the processing of the ballots, the researchers found a one-half to 1 percent error rate for the “read and mark” method, and up to a 2 percent error rate for the “sort and stack” method.”
So, one method resulted in a 0.5% – 1.0% error rate.
The other method resulting in a 2.0% error rate.
So there is at least one method that is BETTER than a different method.
“It is probably impossible to completely eliminate errors in hand counting of ballots,” Byrne said. “However, there are new auditing methods that capitalize on advanced statistical procedures that can help ensure that final election results better match what is actually on the ballots. It is important that we become aware of the limitations of current methods and develop alternative ways to improve the accuracy of election results.”
And they are trying to find better ways of reducing the error rate even further.
And that’s just with a single cycle of counting votes.
Anthony F • February 7, 2012 1:32 PM
Not statistically significant data to add to the discussion. I run the local polling place. I’ve run about 20 elections. We have about 1200 registered voters, get 900+ for presidential elections, 500 or so for off year federal, 300 for locals, half of the above for any primary. One source of error is mismatch between the numbered list of voters (the sign in sheet) and the machine count. My crew is extremely diligent (psychotically diligent) but we still go plus or minus 1 every five or six elections. I’ve spoken with other judges and they’re happy if their plus/minus is under 5 (which would make my crews’ heads explode).
Joe Buck • February 7, 2012 1:41 PM
Bruce, your summary of the article is wrong: they report errors of “up to 2 percent”, and you say “about 2 percent”. You are replacing a maximum observed result with an average.
Even so, I think that the number is too high, because it doesn’t take into account the way hand counts (typically for recounts) are actually done in the US. In each voting precinct there is a representative of each of the major parties present at the counting, each of whom is highly motivated to assign a vote to his or her own column and to oppose the efforts of the opposition to do the same. That means that there are multiple, highly motivated eyes checking each individual ballot.
Petréa Mitchell • February 7, 2012 1:44 PM
SM:
“I wonder how the complicated ballots in the US affect things.”
At a minimum, they probably increase the rate of spoilage (ballots being marked in an invalid manner, like selecting 2 people in a “vote for 1” race). One of the arguments for electronic voting is that it can eliminate spoiled ballots.
Anton • February 7, 2012 2:07 PM
I counted votes in Switzerland. Every pile of 100 vote cards was checked by someone else, sometimes by two people. Then they were weighed by precision scales. There is no way even a 1% error can creep into this process.
Holes in the paper ensure every vote card in the pile is oriented the correct way and if an ‘A’ pile contains a ‘B’ answer, you can fan through the cards and see the error.
The systematic methodology used is very thorough
Mut • February 7, 2012 2:30 PM
It’s important not to conflate the error rate with the uncertainty on the result. You cannot go from “the error rate is up to 2%” to “if the vote is within a few percentage points, they’re likely a statistical tie” because the second statement depends on the sample size and vote proportions. In a close election — the time when this actually matters — the statistical error due to this effect goes like [error rate]/sqrt(N) and is almost always going to be negligible compared to systematic effects.
But even this is to miss the point. The purpose of the vote tallying procedures is not just to count the votes — it’s to do so in a way that everyone agrees is fair. (Not perfect, just fair.) If you have that, your system can tolerate a small error rate. If you don’t, you’re screwed before the first ballot is cast.
LinkTheValiant • February 7, 2012 3:31 PM
I wonder how the complicated ballots in the US affect things.
Not every locality in the US uses those butterflies made infamous by the 2000 race. I know personally that my precinct uses optical-scan ballots, with a box for each race containing candidate name, party affiliation, and a circle beside each candidate’s name to be filled in. It’s almost entirely unambiguous which circle goes with which candidate. There are unquestionably areas that use ambiguous ballots, but it’s not universal across the US.
No, the complication on US ballots isn’t so much the design so much as the high number of races and issues that can be on one ballot. Fortunately my state holds its state elections on odd years, so national elections don’t add to the ballot length so much, but even so the ballot tends to have 10-15 items to be voted on. Even if only 5-7 of them are actually races (the rest being funding issues) that’s still plenty of time for voter fatigue to set in.
The purpose of the vote tallying procedures is not just to count the votes — it’s to do so in a way that everyone agrees is fair. (Not perfect, just fair.)
Unfortunately, fairness includes not only being fair, but also being seen to be fair. The second is the far more insidious problem surrounding any debate about voting procedures.
Isaac Asimov wrote a short story, “Franchise”, about a black box that would interview one carefully-selected person, and from that interview handle the entire election. Clearly such a system would be too open to human manipulation, but it is a nice look at how ridiculous present voting systems can be.
Brandioch Conner • February 7, 2012 3:43 PM
@Anton
“Holes in the paper ensure every vote card in the pile is oriented the correct way and if an ‘A’ pile contains a ‘B’ answer, you can fan through the cards and see the error.”
Exactly. The system needs to be designed with a 2nd (or “confirmation”) counting included in the process. As it would be even more unlikely that the same error would happen the 2nd time through. Unless it is not an error. But that’s a different subject.
This is also where computerized voting can help.
Instead of the voter marking on a ballot, the computer PRINTS the ballot with only the name of the chosen candidate PRINTED on the ballot. That way the voter can easily verify that the ballot is correct.
And, because having more data is usually a good thing … the ballots have “orientation holes” to make aligning them easy … and each candidate has a unique secondary set of punches. So even if the machine runs out of ink, the people counting should be able to count the ballot based upon which candidate’s holes were punched in it.
In order for a ballot to be unreadable, the printing would have to fail and the punching would have to fail.
Richard Gadsden • February 7, 2012 4:00 PM
Has anyone got a link to the paper?
My immediate concerns are:
a) ballot paper design. If these are OMR ballots or punched cards, then I think that’s wholly reasonable. If they were paper X on a single sheet (separate sheets for separate elections) where you can verify each ballot at a glance when you start bundling and counting the stacks, then I’d expect the rate to come down a lot. Rare is the UK election where a recount shifts more than 100 votes (approx 0.5%).
b) Presence of partisan monitors. If each counter has two monitors (one from each party) who are each trying to spot one particular type of error, then that should reduce the error rate.
If their model of an election is unrepresentative, then their results don’t apply to the types of elections it represents.
— They should go to Sunderland and see how an efficient, well-conducted manual count can be run and then see what lessons can be learned in places other than counting technique.
Varjohaltia • February 7, 2012 4:16 PM
Having participated in ballot counting in Finland, as some of the others have pointed out, this seems excessively high. But, as the article doesn’t link to the original article, it’s hard to tell. Finnish ballots tend to be wonderfully simple, with the voter writing in one number. That’s it. Those are easy to collate and sort and recheck. The ballot design and complexity as well as procedures involved and motivation of counters, to me, would seem to make a huge difference.
Bill Smith • February 7, 2012 4:40 PM
My representative in the Texas Legislature won by less that 0.01%: roughly 10 votes out of 50,000. Choosing the winner required a drawn-out recount and an investigation in the legislature. In retrospect, given that the voters supported both candidates by a statistically equal amount, it would have been better to decide the winner by a coin toss. Those committee officials could have better spent their time on more valuable work.
Dale. • February 7, 2012 8:01 PM
That’s a fairly extraordinary claim. Unfortunately, I can’t seem to find the actual study, so we really can’t tell if the article writers were accurate with their summary of the study…
Doug Coulter • February 7, 2012 8:37 PM
Hey, 8 entire precincts completely lost all their votes in the Iowa caucuses. Wonder who they went for? Did all the workers die?
Fraud is much bigger than counting errors.
Recount Experience • February 7, 2012 10:43 PM
A real issue is that the government organizations that run the elections are really far more concerned with how the elections appear than how they actually are.
We participated in a real recount. And let me assure you that ignorance is bliss. Been very jaded ever since. So, if you prefer the happiness of ignorant bliss, read no further.
In our case, it was a close election, only by a few votes. Yet, one of the precincts had votes of ~140% of those registered to vote. Far more than the few votes. But that didn’t matter to the agency that ran the elections. They never even noticed and the election was declared valid.
Truth finally did prevail, but it was, in spite of, and not because of the system. And we had a government agency that actually argued that ordinary citizens shouldn’t have the right (legal term was “standing”) to object to an outcome more votes being cast than eligible voters who showed up on election day.
You need to understand, that when you might show there was an election problem, you become the enemy of those who want to pretend that everything is always as it should be.
And no, the FBI, the National Guard, the BATF, etc. never showed up to arrest any of them.
So, here are a few things you can ask:
Do you think your system is actually well run. How do you know? When was the last time your election office reported what their margin of error was? Do they even know what it is? Worse, do they even want to know? Do they care?
Imagine a factory that claimed to want to reduce it’s defect rate, but had no idea what it was.
For smaller local elections, when your election agency reports results, are they reported by precinct? Are the number of eligible voters for that issue also reported by precinct?
This is information that would be quite easy to put up on an official web site. And it would make it quite easy for any citizen to spot where an error might have occurred. And that is likely why your election agency will never, ever, willingly make it available.
How quickly are results available? Do results take a long time? Or do there always seem to be a few precincts that hold out till the end, and then “discover” just enough votes to make a difference?
How transparent is the process? Is there any special counting program that is confidential or a trade secret? Unlike patents and copyrights, trade secrets are only useful to protect things that aren’t obvious. Counting ballots might not be a first computer science project, but it isn’t rocket science. Hard to imagine what kind of non-obvious “features’ one might be hiding?
What kind of equipment do they use? They all have problems. But if I were the dictator of a two-bit banana republic, I’d absolutely, positively, use the new electronic voting machines. Because, I could make sure it would declare me the winner.
But the real rigging is done out in the open with ballot access laws and gerrymandered districts. In many states, right here in the USA, the number of signatures that your petition needs varies depending on which party you want to run under. And gerrymandering districts is about outcome. Functionally, it is election apartheid.
Nick N • February 7, 2012 11:04 PM
“”two percent” I find that extraordinarily hard to believe. Ask someone from retail management if 2% failure rate in the cash drawer or a 2% failure rate in invoicing or a 2% failure rate in inventory is acceptable”
Maybe if every purchase was made in $1 coins/notes then you would have a point, but we have $50 and $100 notes so people don’t make (as many) errors when counting lots of single instances.
Chris • February 8, 2012 12:41 AM
In Australia, we vote on a Saturday (up until 6 pm) and votes are counted on Saturday night to give an overall result. Each candidate may nominate a scrutineer for each polling place to watch the count.
All votes are counted again during the following week. If the result is ‘close’, any candidate may request a recount. Recounts are repeated until there are two counts with the same result, or until both leading candidates agree that the result is valid.
Jon • February 8, 2012 1:52 AM
I once, some years ago, went through a screening process for an inventory checking contractor. One of the things they loudly reiterated was that if you got caught estimating, instead of ACTUALLY COUNTING, you were immediately fired.
That said to me that inventory inaccuracy is a serious problem. 2% is believable.
J.
renoX • February 8, 2012 2:42 AM
The thing is: it’s hard to say anything about this article except that a hand-counted ballot with errors which can go up to 2% is seriously flawed.
But it doesn’t seem to be difficult to have a “perfect” hand-counted ballot system: make small piles of ballot, count them twice (not by the same person): if the results differs count it again (with small enough piles of ballot, this should be rare), there will still be errors, but not 2%!
Jim T • February 8, 2012 6:56 AM
I have participated in 2 recounts in Texas. One was just mark sense paper ballots counted by machines, the other had about 50% paper and 50% DRE (electronic voting machine). A couple of points that seem to have not been noticed. First, In Texas, like many states, if the win is below a certain percentage of votes, a recount is automatically done. This is in effect an acknowledgement that all counting systems can have errors.
During the recount the paper ballots for each precinct were counted by two separate teams that didn’t know which precinct they were counting. If all 3 results were the same, the count was assumed accurate. If there were differences then another 2 counts were done. Errors in marking such as an X so big in one box that the ends crossed into another box caused machine counting errors. If there was a single dot in the box, was that a vote or simply a smudge? How bid before it counts. Second guessing the voters intent in a situation like that will have errors.
The individual electronic machines had internal memory that was compared to the totals reported by the control stations, and there were no differences found there. I observed the pre and post election logic and accuracy tests which are used to verify the programming and functioning of the machines but a subtle bug or malicious program modification could slip by. Looking at the errors we catch each election on these machines does not fill me with confidence.
Each of the recounts I helped with had final results different from the initial result. Didn’t change the results enough to change the outcome but the changes were big enough they could have. It showed me that, at least with paper ballots, there are way’s to reduce the counting errors to a very low level. It also showed me that with the current generation DRE Electronic machines, if there is an error in the counting there is no good way to find it.
Chris • February 8, 2012 12:30 PM
I actually think this is rather meaningless without a great deal more information. In the 2000 Florida recounts (which I’m highlighting simply because we have far more information on them than other recounts, not because of the outcome or importance) there were issues with how to infer intent based on various things (hanging/dimpled chads, what to infer from double punches, etc). This sort of error goes away with most electronic voting systems, but then the problems of fraud come up. So which type of ballot are we talking about? Additionally during the FL recounts there was the problem that the hand counts may or may not have had lower error rates than the machine counts (I never saw good data) but it was clear that the hand counts were producing higher vote numbers than the machine counts (counting ballots that the machines rejected more often than rejecting ballots the machines accepted), which means that if you only hand recounted in some districts you would introduce a biased error into the unbiased errors produced by the counting itself (since invariably districts vote more one way than the other). And even if you hand count all the ballots it would seem to me that people are more likely to be interested in who gets elected than machines and thus more likely to bias error, even if there are others watching.
So, to Bruce’s assertion that there’s a myth out there that it’s possible to count votes without error (which I agree with wholeheartedly), I would add that there’s a myth that hand counting is inherently superior to machine counting, thus we have laws in several states that if the spread is too close there must be a hand recount.
Jim T • February 8, 2012 4:16 PM
Chris, I’ve been involved in the mechanics of several elections. One thing I’ve seen is that machines and humans make different types of errors. As part of the paper ballot scanning software used locally, questionable votes (those with too few or too many votes) are displayed on a screen. A human can tell that a voter had accidentally rested the pen in the wrong box before filling in the one he meant. A machine can’t make that decision. Same thing when the mark is too small, or the pen is running out of ink, or as in my previous comment an X gets so big part of it is in another race. Humans catch those types of errors and machines don’t. There have also been cases where a speck of dust at the wrong spot in an optical scanner has caused several miscounted votes. At the same time people make types of mistakes that machines don’t.
When doing a count/recount it’s a good idea to use both human and machine recounts. It’s also a very good idea to do random, quality control recounts just to verify the results. After being involved in several elections and some recounts, I think the best system will allow a combination of machine and human counting. Never eliminate all possible errors, but different counting methods are more likely to have different types of common errors, which should help with the reliability of the results and reduce the frequency of common errors.
There is a reason that in many situations, arriving at a result by two distinct, independent methods, is a good idea. (Anyone remember some early FPU bugs?) The same applies to counting votes. Machines and humans commonly make different types of errors. Helps improve accuracy, reliability and fairness of the results. Not perfect but better then either individually.
Brandioch Conner • February 8, 2012 4:48 PM
@Jim T
@Chris
I think there is some confusion creeping in.
Incorrectly (or ambiguously) completed ballots should be distinct from the error rate of the process of counting votes. Start with the presumption that the ballots are unambiguous. Can you get the error rate on unambiguous ballots down to 0.0%? How?
That leads to the question of how to cast unambiguous ballots. Can it be done?
Maybe it isn’t possible to create an unambiguous ballot. Which leads to the question of how to handle ambiguous ballots (ambiguous to a machine / ambiguous to a person).
And that does not address the issue of an unambiguous ballot NOT reflecting the intent of the voter. See “butterfly ballot”.
https://en.wikipedia.org/wiki/File:Butterfly_Voters_View.jpg
Jim T • February 8, 2012 5:43 PM
@Brandioch Conner
To be meaningful, the error rate has to be deal with ballots filled out by humans. Any counting system needs to be able to deal with voter errors and mistakes that still leave the intent of the voter clear. On a paper ballot what do you do when the voter circles the box instead of filling it in. Intent is clear but current machines can’t count it. Similar problems happen with DRE’s, especially those with touch screens that need to be cleaned and calibrated.
A study that only deals with “unambiguous” votes gives a false picture of the trustworthiness of a system that will likely fail spectacularly at the most inconvenient time in an election.
Brandioch Conner • February 8, 2012 6:18 PM
@Jim T
“A study that only deals with “unambiguous” votes gives a false picture of the trustworthiness of a system that will likely fail spectacularly at the most inconvenient time in an election.”
Really? How so? That seems to contradict what the article that Bruce posted stated.
You seem to be confusing human vs machine. Why not have humans verify the machine’s evaluation of each vote? This has been brought up before. With a machine you trust the programmer as the final arbitrator of the election. Or the maintenance guy. Or … So you don’t trust machines.
And you’re also confusing voter intent on systems with calibration problems.
Jim T • February 8, 2012 7:18 PM
@Brandioch Conner
Part of what I’m responding to is the article, part the comments. The research is good as far as it goes, but the article doesn’t specify if the ballots counted were hand or machine generated and if there were variations in the marking similar to what you would find in a real election.
With the mandatory recounts on close votes required by some states, you effectively have humans verifying the machine count already.
The calibration errors on touch screens ends up causing problems like the butterfly ballots mentioned earlier in the comments.
With the current generation of DRE’s there are few of the checks and error correction methods you would expect in a machine with such an important function. There are many sites that track the errors reported during elections. Lots of errors. Some machine, some human. Current generation of voting machines requires me to trust that there are no errors or malfunctions in large parts of the process. The available evidence shows that is not the case. We are still in the Altair stage of voting machine development. Haven’t made it to the Apple 2 level yet. Lots of work still to do.
Brandioch Conner • February 8, 2012 7:59 PM
@Jim T
“The research is good as far as it goes, but the article doesn’t specify if the ballots counted were hand or machine generated and if there were variations in the marking similar to what you would find in a real election.”
Either way, their research showed that one method had a lower “error rate” than the other.
Now, would it be a correct usage of “error rate” if the ballots were ambiguous? I’d say no. But until the original paper is released I cannot say for sure.
“With the mandatory recounts on close votes required by some states, you effectively have humans verifying the machine count already.”
And, again, one method of human counting has been shown in the article to have a higher error rate than the other. The article seems to imply that they only studied two different methods. And possibly for a single cycle.
Is it possible to get a human process to 0.0% error rate on unambiguous ballots? Even if it involves multiple cycles of counts and verifications?
Is it possible to generate unambiguous ballots? With some form of built-in check?
And so forth.
sabik • February 9, 2012 3:09 AM
Hmm, as a computer professional, I’m not surprised at this… 2% is a fairly reasonable estimate for the fraction of lines with errors in computer programs with minimal testing. Usually written “20 bugs/kLoC” or “20 bugs per thousand lines of code”. Some say 10, some say 30 (or even 50), but 20 is certainly not a surprise.
So others make mistakes at the same rate as us.
The question isn’t about the fact that mistakes are made, it’s more about how to cross-check and verify and otherwise eliminate them from the final product (in this case, the final count). Even this paper does that by comparing two systems.
Corey Mutter • February 13, 2012 11:57 AM
In my county we use optical-scan ballots and hand-count 1% of the precincts (randomly selected) each time to check up on the machines.
I heard that every time there was a discrepancy, upon re-examination it was the hand count that turned out to be wrong.
guest • February 13, 2012 3:49 PM
political parties know about how much vote is necessary for a chance to win in a recount and they will call a recount when it is closer to 0,1% of the vote.
I have seen the major counting error not in the counting of the vote but in the report of the vote from the local to the central office. A typo or can shift some hundreds of vote. This is why local observer report local count to their political party so they know right away if that type of error happened.
Dave Burton • August 15, 2016 8:02 AM
Bill Smith wrote, “My representative in the Texas Legislature won by less that 0.01%: roughly 10 votes out of 50,000”
Wrong. “less than 0.01%” would be less than 5 votes out of 50,000, not “roughly 10.”
Subscribe to comments on this entry
Sidebar photo of Bruce Schneier by Joe MacInnis.
Vince Mulhollon • February 7, 2012 6:20 AM
“two percent” I find that extraordinarily hard to believe. Ask someone from retail management if 2% failure rate in the cash drawer or a 2% failure rate in invoicing or a 2% failure rate in inventory is acceptable, even amongst overworked, underpaid, unmotivated, undersupervised low level employees. Something as simple as counting Ds and Rs by wide awake, motivated people is simply not going to fail at a 2% rate “accidentally”. There must be a flaw in the experiment. Maybe the experiment was funded or controlled by someone with an axe to grind?