When Voting Machine Audit Logs Don't Help

Wow:

Computer audit logs showing what occurred on a vote tabulation system that lost ballots in the November election are raising more questions not only about how the votes were lost, but also about the general reliability of voting system audit logs to record what occurs during an election and to ensure the integrity of results.

The logs, which Threat Level obtained through a public records request from Humboldt County, California, are produced by the Global Election Management System, the tabulation software, also known as GEMS, that counts the votes cast on all voting machines—touch-screen and optical-scan machines—made by Premier Election Solutions (formerly called Diebold Election Systems).

The article gets pretty technical, but is worth reading.

Tags: , ,

Posted on January 23, 2009 at 7:43 AM • 39 Comments

Comments

Clive Robinson • January 23, 2009 8:19 AM

And the upshot is software that is,

1, Poor software specification
2, Badley written
3, Not audited correctly
4, Has complex and mainly usless update security.
5, The company says that it’s all fixed in revision x.y.z.a.16, but due to 4 above most machiens are many revisions behind.

Oh and the hardware can be easily opened and fairly quickly tampered with…

Q • January 23, 2009 8:24 AM

The real scary part is that this is actually the safest, most-auditable system: actual paper ballots (optical scan), with the machines just counting them.

What kind of problems are there with systems that don’t have paper ballot trails?

Dan • January 23, 2009 8:28 AM

An extremely telling comment “He assumes Premier has documentation explaining how to interpret the logs, but says if it does, the company doesn’t share that information with election officials…”

wiredog • January 23, 2009 8:34 AM

One interesting thing that comes out when you read the comments to that article. It turns out that “a University of Iowa computer scientist” doesn’t recognize Unix timestamps. Says something about the CS Department at U of I, doesn’t it.

Auditor • January 23, 2009 8:37 AM

Had the logs been clear and provided evidence of the events that transpired, did the vendor implement the audit logging properly? Even in the 2002 Voting System Standard, audit data must be maintained in a permanent record that cannot be modified. Problem is that is hard to do, voting system vendors don’t do it and labs charged with testing and certifying these systems did a poor job.

Marc B. • January 23, 2009 8:57 AM

I don’t think it’s worth reading for a lay person and even for most interested in CS. The article is bloated by statements of misunderstanding and speculation. The authors prevailing trait is confusion.

Of course the logs are complicated, maybe faulty, but until the author of the article does not really understand anything about them, he should report that fact and abstain from rambling about stuff he does not understand and speculating.

Maybe Wired should invest time and effort to actually understand the logs and give a thorough report then. So far I consider article worthless.

Carlo Graziani • January 23, 2009 9:03 AM

While Premier/Diebold certainly deserves corporate annihilation for their obdurate sustained ineptitude, let’s not allow the California SoS’s responsibility to pass under silence here. Why exactly was this failure mode discovered on Election Day? Did they do no disaster-recovery testing beforehand?

If so, this is a scandal. By virtue of its installed base of election machines, the SoS is essentially the largest IS organization in any state government nowadays. It is not enough that they should know election law and procedure: they also need to understand and implement standard best practices of the IS industry.

It is completely unacceptable that they should have no idea that their logging is non-functional going into an election. The scope of this problem is such that it should have been evident with even minimal DR testing. Which implies that they didn’t do any. Someone should be fired for that.

NM • January 23, 2009 9:10 AM

@wiredog: and using localtime() to display the time in an intelligible format was beneath Premier’s programmers?

Gerry Rossi • January 23, 2009 9:20 AM

The best procedure for ensuring that the log is accurate is to read it, rebuild all relevant totals and compare to what the on-line system reports. When the rebuilt totals match the on-line results, you can at least have some assurance that the log tape is accurate. This approach has been standard since the sixties. In this case, verification is more difficult because elections are relatively rare and are never re-run, but this test of log tape integrity should be part of the basic acceptance test process.

Ideally the rebuild program would be created by a third party organization from documentation of the log. In this case, as the vendor refuses to provide such documentation, the rebuild verification outlined above is essential and should be provided by the vendor before the system is put into service. It’s clear from this very good — and disturbing — article that such a test has never been (and probably never will be) performed.

This is computer operations design and testing 101.

What’s most disturbing is that we’ve been hearing about issues such as this one in the election tabulation industry for years. Election supervisors have neither the knowledge (their “experts” don’t understand timestamps??) nor inclination to fix this and incumbents, who were elected within this environment, have no desire to upset any applecarts. It will take a successful lawsuit rendering an election invalid to force any changes. Little chance of that happening. Here in New Jersey, they’ve been postponing fixes for years.

Sam • January 23, 2009 9:27 AM

I realise this may be a little unfair, but Point 2 on the first post above nearly broke my Irony Meter…

wiredog • January 23, 2009 9:28 AM

@NM
Speaking as a developer, no. The programmers should’ve used localtime() instead of just writing out the timestamp.

But how does a CS profesor at a major state university not recognize a timestamp as being, at least, a timestamp? That’s my issue with the analysis.

David • January 23, 2009 9:30 AM

Also of interest to voting fans are some of the results in the ridiculously close Minnesota Senate race.

The most interesting thing is probably 133 missing ballots. Nobody seems to doubt that they existed, but they can’t be found now. Is it then correct to use the original machine counts (which kind of defeats the purpose of relying on physical ballots only) or throw them out (making post-election ballot theft pay off, although I doubt that happened here, and disenfranchising 133 people who did vote)?

Standard practice in case a ballot was rejected by the counting machine was to void it and replace it, but there are precincts where this may not have happened, and possibly both spoiled and valid ballots were recounted. The recount was by humans, who are likely to accept ballots (on the basis of voter intent) that the machines rejected (on the basis of comparative reflectivity or something). Do we go with the ballot count or the earlier machine count if we have reason to suspect this?

Or, for that matter, absentee ballots, where the Minnesota system has problems. An absentee ballot could be filled out strictly according to the State-provided instructions and be technically illegal. Which ballots should be counted? Just compliant with the law? Compliant with the instructions? Where intent was clear as well as desire to comply, and the voter made technical mistakes?

From a statistical point of view, the electorate has not expressed a preference, there being no reason to reject the null hypothesis that the state just didn’t care. Depending on point of view, justice will be served if either is seated, or will only be served if neither is (since nobody in the race got a statistically significant plurality).

Trichinosis USA • January 23, 2009 9:38 AM

@Wiredog: Unix timestamps are heinously easy to manipulate if you have root on the machine. I am aware of at least one corporation that played games with it’s systems to facilitate accounting fraud. For an entire month, the date on their machine was April 1st.

paul • January 23, 2009 9:50 AM

Maybe someone forgot to set the log level to “comprehensible” before delivering the software. It really does make you wonder what the election-machine companies do with their time.

Jeff Dege • January 23, 2009 10:04 AM

What it really comes down to is that the trace logs the programmers use to track down bugs are not designed to act as an audit trail, and whatever fool at the SOS office who managed to let himself be convinced that they would serve for that function should have known better.

MysticKnightoftheSea • January 23, 2009 10:13 AM

Just a question:
Some time ago I heard that there was some open source project for vote counting software, with the ability to peer-review, much like peer-reviewed crypto.

Am I mis-remembering this, or does this exist? If so, where?

j • January 23, 2009 10:49 AM

The scariest part, I think, is:

the Iowa computer scientist mentioned in the post said it might be a time stamp but couldn’t decipher what it could be. Nor could a second engineer who looked at the log.

Unix time stamps are tricky ciphers.

MikeA • January 23, 2009 11:09 AM

In an increasingly DOS-based world (until lately), dates before 31DEC1979 just don’t exist, so if those numbers were timestamps, their epoch is an imaginary number. 🙂

Roy • January 23, 2009 2:13 PM

That timestamp, 1225737079, is 2008 Nov 03, a Monday, at 184119 GMT, which is the calendar day BEFORE the election. Maybe that’s when all the real votes were counted, deciding the results before those pesky voters showed up.

This makes me worry where the machines are getting their time from. Over the air from WWVB? From GPS? Or does some guy manually set the date and time?

I once saw a receipt from a Sparkle Car Wash that was an hour and thirteen minutes off the correct time. At least they had the day right.

Clive Robinson • January 23, 2009 5:11 PM

@ Sam,

“I realise this may be a little unfair, but Point 2 on the first post above nearly broke my Irony Meter…”

Not unfair at all.

Post in haste repent etc etc…

Bad spelling is unfortunaly one of my less usefull traits. Apparantly it has to do with being both left handed and left brained…

A “trick-cyclist” I know socialy once commented after I asked why left handers where excluded from most brain function studies (especialy with fMRI),

“The trouble with you left handers is your brains aren’t wired up right.”

The scary thing is the number of scientists, engineers and architects etc who are left handed (accountants on the other hand…).

Jonadab the Unsightly One • January 23, 2009 8:07 PM

Deputy Secretary of State Lowell Finley has
referred to the logs as “‘Greek’ to anyone
other than a programmer.”

What, you thought something as inherently technical as a logfile would be comprehensible to a layperson? Maybe I should print out some dpkg logs, or mysql logs, pass them around at work (non-IT industry; I’m the only computer guy) and see if anyone understands them.

Maybe the logs don’t have the info that’s needed, but then again, if the people who are looking at them don’t understand them very well, maybe the information is there and they’re just not seeing it.

Brad Conte • January 24, 2009 12:14 AM

From the previous article in the series:

The company has apparently known about the problem since 2004 and provided some election officials with a workaround

Millions upon millions of dollars spent, and we still can’t get functionality available in many $.99 calculators at the drug store. And it doesn’t even seem to bother anyone at the company.

o.s. • January 24, 2009 5:11 AM

“The manual log shows that the 197 ballots were scanned by Elections Manager Kelly Sanders on November 1, three days before the election. The receipt from the scanner also shows the ballots in the system, although there’s no date on the receipt — another problem for conducting audits.

The ballots even showed up in the status report (.pdf) printed from GEMS on election night. (See the 197 ballots that show up as “deck 0” on the first page of this report.)

But some time after election night, the tabulation software deleted the ballots. (A second report created after the election canvas was completed (.pdf) shows the 197 ballots missing.)

Premier attributed the problem to a programming error in GEMS that causes the first “deck” or the first batch of ballots counted by the software to be randomly deleted if a subsequent deck is intentionally deleted.”

To describe this situation in a word: “pathetic”. Seriously how hard is it to have everything stored in a database file as a .mdb file or something and then just delete the record by id? A program that randomly deletes records that its not supposed to just reeks of college level amateurism. Diebold/Premiere is either fully incompetent or malicious; I’m not sure but neither is good.

Steve Davies • January 24, 2009 3:15 PM

The logs are crap, lost data, gross level bugs that got through ‘testing’. For Voting Software !! I bet the brochures look good though. Vendor should be hauled over the coals in a BIG way in my view. Lots more that should be done but the internet is public so won’t list them.
Diabolical, behind farce. What percentage of the population voted using those machines ? Use of past tense deliberate. thats where they belong.

RonK • January 25, 2009 5:19 AM

@ Clive Robinson
The scary thing is the number of scientists, engineers and architects etc who are left handed (accountants on the other hand…).

Nah, you’re comparing apples and oranges. What’s important for an accountant is that each hand doesn’t know what the other one does.

Anonymous • January 25, 2009 5:49 AM

Only in America…
Here in Germany (80 million inhabitants, and voter turnout usually much higher than in the US) we vote on paper. With a pen. (Some fuckheads try to change that, but most elections are still made the old fashioned way).
After 6p.m. on election day (usually a sunday so nearly everyone can participate) the votes are counted. By hand, not by machines. By 8p.m. or so the first preliminary results are in. At 11p.m., the numbers are pretty solid and won’t change much anymore. The next day, the official result is published. All largely done without voting machines. The system works. No vote tampering that we know of. Everyone can watch the ballot counting in person. Recounts are easily possible. Why make things so complicated? Sometimes less is more. Don’t fix it if it’s not broken. This has worked for the last 60 or so years here (even longer in other locales), is cheap, it’s transparent to everyone. It’s also FAST. Fast enough, there is no need to have the results 10 minutes after the voting booths close, waiting for a few hours is a small price to pay for an untampered election without Florida recount fiascos!

Peter • January 25, 2009 12:40 PM

Q at January 23, 2009 8:24 AM:”The real scary part is that this is actually the safest, most-auditable system: actual paper ballots (optical scan), with the machines just counting them.

What kind of problems are there with systems that don’t have paper ballot trails?”

Hehe… if only… in the last Scottish Parliament elections they did just that (paper forms, machine count) and it was a total farse. There were a lot of constituencies where the number of “spoilt” ballots were far more than the majority of the apparent win. Interestingly enough SNP won over Labour by a single seat… now, that is pretty darn co-incidental by anyone’s measure.

Peter • January 25, 2009 12:43 PM

@Roy at January 23, 2009 2:13 PM:”Maybe that’s when all the real votes were counted, deciding the results before those pesky voters showed up.”

Nice one 😉

Peter • January 25, 2009 12:48 PM

@Anonymous at January 25, 2009 5:49 AM:”Here in Germany (80 million inhabitants, and voter turnout usually much higher than in the US) we vote on paper. With a pen…… Why make things so complicated? Sometimes less is more. Don’t fix it if it’s not broken. This has worked for the last 60 or so years here (even longer in other locales), is cheap, it’s transparent to everyone. It’s also FAST”

While I’d normally agree with you, if you change from a “first past the post” election to something like single transferable vote, then the counting would take several days… which may be too long to wait for democracy (personally, I wouldn’t mind if that was declared in advance, but it seems to be a hinderance).

Clive Robinson • January 25, 2009 1:40 PM

@ Peter,

“While I’d normally agree with you, if you change from a “first past the post” election to something like single transferable vote, then the counting would take several days…”

I wish that Politicos would leave the voting process alone. Single transferable vote is little more than a method for the “old guard” to keep their cusshy jobs and not be voted out by the electorate who see them as usless.

Calum • January 26, 2009 3:45 AM

The Scottish election farce can’t really be blamed on the machine. That voting paper was absolutely insane; the old lady in the booth next to mine was nearly in tears trying to understand it. I ended up having to show her where to mark her paper, it was so bad. The machines would have been more or less OK had it not been for the state of the voting paper.

Good old Arrow; whoever we vote for, he’s guaranteed not to get in. Personally, I like approval voting – exactly the same, except you can vote for as many as you like.

bob • January 26, 2009 7:06 AM

Obviously the government {isnt going to solve this | doesnt know how to solve this | is incapable of solving this.}

So heres how we fix this ourselves. We start a public-domain, open-source, GPL vote system project that runs on a basic linux x86 box. It would need a user interface, supervisory update capability (to change who is running) audit trail, printed output, secure communications facility, a way to prove to the ordinary user that the version running on the box is the same source code as was tested and a master tally unit to aggregate county, state and national totals and serve the count publicly.

We spend the next 2 years designing, developing, testing and PUBLICLY VETTING it. Then we unite behind it and push for it to be adopted as the national standard.

TheDoctor • January 26, 2009 7:33 AM

@bob: Wrong approach.

With adopting the mere concept of a computer based voting system you’re on the slippery downhill track to hell.

Use paper and pencils instead.
…ok, that’s a futuristic approach, but it solves the problem…

David • January 26, 2009 8:54 AM

@German anonymous:

How many actual questions do you have on your ballots? Here in the US, on an even-year election, I’m typically voting for a President or Governor, likely a Senator, certainly for a Representative, two State-level legislators, maybe a dozen judges, half a dozen or so more minor officials, and a State Constitution amendment or two.

You can certainly make a case that we vote for too many things here (there’s positions I’d like to make appointed, simply because I don’t think anybody pays attention to the candidates), but that’s the way it is.

If we had separate ballots for each question, I suspect the sheer numbers would cause problems. If we used fewer sheets of paper, it would delay the counts considerably. (The most effective way to hand-count I know of is to sort, then count.)

Moreover, I’d like to see some sort of voting other than simple plurality (we’ve been having a lot of third-party candidacies here in Minnesota), and that would make it harder to hand-count.

TheDoctor • January 26, 2009 9:30 AM

@David:
Yes, we know about that fact.
We also know that there is one election day where all elections are held.
This is different in germany (Do I think thats better ? No I don’t think so).

Nevertheless, think what you prefer, convenience or untampered democracy ?

Paper count is easy to parallelize, so counting is not the problem and for the multitude of elections, maybe going to two or three election days ?

The problem for us aliens 🙂 with anything that’s going on in the USA: our polititians tend to copy the bad habits and ignore the good ones…

bob • January 26, 2009 1:54 PM

@TheDoctor: Personally I think the IBM Votematic system we used for the first 20+ years that I voted were the best solution (cardboard card mounted in an inexpensive portable jig, punched out with a stylus and counted by a centralized card reader) were superior to ANY computerized UI, however there is a “its better if it has a computer” mentality out there that I dont believe can be overcome, so I put forth the best way to minimize the impact of using a computer.

Besides after the official language of the United States becomes Spanish, it will be easier to have a “click here for French, Italian or English” option on a computer.

EbiDK • January 26, 2009 3:53 PM

As I also said in a comment on the article:

There should be made a free/open source electronic voting solution(both the hardware and software) so that everyone can see exactly what is going on and check that it is tamper proof.

Companies should be allowed to build the hardware according to the design and sell it so there would be someone to mass produce it.

Leave a comment

Login

Allowed HTML <a href="URL"> • <em> <cite> <i> • <strong> <b> • <sub> <sup> • <ul> <ol> <li> • <blockquote> <pre> Markdown Extra syntax via https://michelf.ca/projects/php-markdown/extra/

Sidebar photo of Bruce Schneier by Joe MacInnis.