Cheating on Quantum Computing Benchmarks

Peter Gutmann and Stephan Neuhaus have a new paper—I think it’s new, even though it has a March 2025 date—that makes the argument that we shouldn’t trust any of the quantum factorization benchmarks, because everyone has been cooking the books:

Similarly, quantum factorisation is performed using sleight-of-hand numbers that have been selected to make them very easy to factorise using a physics experiment and, by extension, a VIC-20, an abacus, and a dog. A standard technique is to ensure that the factors differ by only a few bits that can then be found using a simple search-based approach that has nothing to do with factorisation…. Note that such a value would never be encountered in the real world since the RSA key generation process typically requires that |p-q| > 100 or more bits [9]. As one analysis puts it, “Instead of waiting for the hardware to improve by yet further orders of magnitude, researchers began inventing better and better tricks for factoring numbers by exploiting their hidden structure” [10].

A second technique used in quantum factorisation is to use preprocessing on a computer to transform the value being factorised into an entirely different form or even a different problem to solve which is then amenable to being solved via a physics experiment…

Lots more in the paper, which is titled “Replication of Quantum Factorisation Records with an 8-bit Home Computer, an Abacus, and a Dog.” He points out the largest number that has been factored legitimately by a quantum computer is 35.

I hadn’t known these details, but I’m not surprised. I have long said that the engineering problems between now and a useful, working quantum computer are hard. And by “hard,” we don’t know if it’s “land a person on the surface of the moon” hard, or “land a person on the surface of the sun” hard. They’re both hard, but very different. And we’re going to hit those engineering problems one by one, as we continue to develop the technology. While I don’t think quantum computing is “surface of the sun” hard, I don’t expect them to be factoring RSA moduli anytime soon. And—even there—I expect lots of engineering challenges in making Shor’s Algorithm work on an actual quantum computer with large numbers.

Posted on July 31, 2025 at 7:00 AM22 Comments

Comments

Clive Robinson July 31, 2025 9:05 AM

@ Bruce, ALL,

With regards,

“They’re both hard, but very different. And we’re going to hit those engineering problems one by one, as we continue to develop the technology.”

We also know that some problems are not solvable in the frame of human understanding / existance. Two such are,

1, Getting human sized matter to or beyond the speed of light.
2, Getting human sized matter to go back in time.

Part of the first we can get close to and that could give “false expectations”. The second well the result of that if it could be done would be rather more than existential for the universe by our understanding of the laws of nature and the universe.

Thus when we look at “quantum computing” we have to ask two things,

1, Do we yet know enough to understand the barriers?
2, Are the barriers actually possible to solve?

The answer to the first is clearly “NO”. Which makes the answer to the second an “Open Question” at best currently. However with a gradient that increases daily, there is an attached increasing probability of the answer being either “NO” or such that it “Might as well be NO”.

One thing we see is that as the number of QBits goes up so does the noise that makes the results exponentially difficult to obtain. To get past this we think needs at a minimum exponentially more Qbits.

Thus two further questions arise,

1, Just how many QBits can their be in a bounded universe?
2, What tiny tiny fraction of those can actually be used in a meaningful way?

But before we get there, there is another question that is more practical in nature.

Whilst helium is plentiful in the universe it is actually incredibly scarce on earth. Also each time we “dump helium” it gets into the upper atmosphere, where it gets knocked into space as solar radiation strips it off.

Thus,

“Can we afford the helium to make general quantum computing viable?”

wiredog July 31, 2025 10:34 AM

@clive
Well, if we solve the controlled hydrogen fusion problem we can manufacture helium.

Might be easier to capture the helium emitted by decaying reactor fuel though.

Anonymous Coward III July 31, 2025 11:39 AM

Wow, thanks for this funny paper. It’s quite good timing as I’m preparing to re-enter uni and consider all kinds of software testing projects anew.

Whilst I’ve just downloaded the paper and not read it yet beyond the funny intro, I don’t think you can immediately be so dismissive about the power of an abacus (I’m less sure about why you’d use a dog for arithmetic). I mean, some of the techniques I use everyday take the mathematician’s trick of using an abacus and go one stage further they allow you to use them even if you don’t actually have a live abacus before you.

Of course, we should exercise caution regarding the so-called post-quantum encryption systems, since, the current orthodoxy that says factorisation is hard on a classical computer (ie: one not deliberately exploiting quantum effects for the computation, let’s say) — that current orthodoxy isn’t necessarily true, it’s just accepted wisdom until a new 12 year old comes along and proves that a classical computer could do that all along.

Nevertheless, thanks for the super funny and interesting reading. A new edition to my software testing strategy, and perhaps, future focused, a way to write a science paper that people will read!

Winter July 31, 2025 1:27 PM

land a person on the surface of the sun

The landing part is not hard, beyond the fact that we still have to decide what counts as the surface and what state of matter the person should have when landing.

If the idea is to get the person back alive, then we still have to find out the boundaries within which the laws of nature even allow it to be done.

Gogognome July 31, 2025 2:20 PM

The article contains a lot of humor and a couple of its references are published on April first. I know very little about this subject, so it feels like this whole article is an Aprils Fools’ joke.

Nevertheless, the arguments seem to be sound and the “Proposed Quantum Factorisation Evaluation Criteria” make sense to me.

So, how serious is this article? The way Bruce Schneier quotes it, gives me the impression that he thinks this is a serious article.

Perhaps I should think of this article as a comedian explaining a serious subject in a funny way?

Peter Gutmann July 31, 2025 7:14 PM

Actually Scribble is a friend of mine’s dog (I’m more a cat person), he’s a loveable guy who will try and beat you to death with his wagging tail when he sees you. Unfortunately we said goodbye to him yesterday so he won’t be able to match any more quantum factorisation records in the future :-(.

The paper definitely isn’t an April Fool’s joke, it was just very hard to write a serious-sounding paper on a subject like this.

There’s also a companion paper, or talk actually, on the nontechnical background to this, “Why Quantum Cryptanalysis is Bollocks A Lesson from History”. That’s what motivated the paper, my co-author Stefan Neuhaus from the Zurich University of Applied Sciences suggested turning the barking-dog comment in the talk into a proper paper.

Now I just need to find somewhere that’ll publish something like this. Claimed quantum breakthroughs are traditionally announced in Nature or similar but I suspect they might have some very specific ideas about the suitability of this one for publication.

Clive Robinson August 1, 2025 12:57 AM

@ Peter Gutmann,

“Unfortunately we said goodbye to him yesterday so he won’t be able to match any more quantum factorisation records in the future :-(.”

I’m sorry to hear that. Pets become members of the family, and the families social circle.

I had a friend some years ago who had a large ball of fluff with an equally active tail. Like many humans the ball of fluff was known not by his “official name” but by the nickname of “Cannonball” due to his habit of launching himself from the hallway stairs at the chest of callers at the front door and putting on a credible emotional attack with an over active tongue. All of which might have been fine if he was a yappy ankle muncher, not a very fit and active border collie. He also had a fondness for being hugged by small children and playing “football” (soccer) with older children in the park.

His owner wryly noted that the size of their social circle had been very much enlarged by Cannonball. Sadly as happens with time they are both nolonger with us but fond memories follow them both.

Anonymous Coward III August 1, 2025 4:51 AM

@Peter Gutmann

Perhaps you could do a couple of versions of that paper, one for a maths/physics journal and the other for, say, a journal on software testing. It might be that the suggestions ofr journals contained in this message turn out to be unsuitable, but it might help by a micro-amount.

Consider:

https://onlinelibrary.wiley.com/journal/10991689

As regards software testing.

And consider:

https://www.lms.ac.uk/publications/nonlinearity

As regards maths/physics.

Happy paper rewriting and thanks for the wonderful paper so far.

Anonymous Coward III August 1, 2025 5:10 AM

@Peter Gutmann

Alternatively: one math paper, one physics paper and one software testing one.

Where the physics one deliberately targets or presents the research from information theory-side, since, the physicists these days claim that that’s really the realm of physics (if you’re a philosopher you could argue the point).

But it’s also relevant cos exploding it from 1 to at least 3 papers allows for more coverage and perhaps increases the likelihood of answers being found to address some of the concerns (though that might require a lot more changing of outfits or days to pass, whichever happens first (this statement is not aimed at theatre on-stage actors who make day changes quicker than most mortals!)

iAPX August 1, 2025 7:03 AM

@Peter Gutmann

Cool paper, I love the tone and the humour, while it describe something I didn’t expected: the true state of Quantum factorization.

Cat owner here so barking is not possible and they are really random meow generators, except when hungry. This is Inversion of Control, compared to dogs.
But also have my old slide rules, so I should run for quantum supremacy too.
Maybe factoring 42, to push the limits of Science further?!?

Thanks a lot!

M Welinder August 1, 2025 8:05 AM

The criteria for successful quantum factoring in that paper are a bit too strict.

Surely it’s enough that the system consistently produces one factor. (Section 7, item 4.) Then the Vic-20 can be deployed to find the other.

Larry Seltzer August 1, 2025 2:28 PM

I feel sorry for the dog too, but at least cats aren’t taking all the quantum abuse anymore.

Peter Gutmann August 2, 2025 2:15 AM

We really wrote it for fun, trying to massage it into multiple different papers would turn it into work and would probably end up pretty dry reading. It’s also not clear whether the result would be publishable – it’s not announcing any (serious) new result so would probably have to end up as a short letters-to-the-editor sort of submission.

Celos August 2, 2025 5:35 AM

It is a real pity something like this is hard to publish. Because what it does is take a step back and ask the question “Are we really making the progress that gets claimed?” and then provides evidence that this is not actually the case. As such it is a negative meta-result and these would be really important if scientists were more concerned with actual progress than with looking good (and keeping their jobs).

IMO far too much effort is invested in appearances and “spectacular” questions, especially in CS research, and too little in real questions that need answers if we ever are to get reliable and secure computing with good usability.

Clive Robinson August 2, 2025 9:48 AM

@ Celos,

With regards,

“Because what it does is take a step back and ask the question “Are we really making the progress that gets claimed?” and then provides evidence that this is not actually the case”

Perhaps the issue is something that is it’s self a question. That is,

“What is progress?”

Much of it is not proving something is true or false, but gathering knowledge both old and new and using it to support humanity.

As an engineer I know we productively use a lot of things we do not actually fundamentally understand such as “inertia”.

We measure them and extract reliable characteristics, which we them call “intrinsic properties” or “natural tendencies” and then call the equations we derive to match the graphs of our measurements “principles”.

Thus we have both progress in that we can apply the principle usefully but also ignorance because we know nothing beyond the observations of the measurements.

Thus both practical principles and theoretical ignorance marching hand in hand to where? we have no idea, but it is progress.

But which if either is actually progress?

But there is an issue we don’t talk about untill it’s threatened as it is currently in the US.

Untill the mid Victorian era, progress was mostly artisanal and frankly dangerous. With knowledge of how to solve the dangers hard won by destruction and death kept secret as “trade / guild secrets”. To not just gain wealth but power over others with.

It was around this time that independently wealthy people became “gentlemen experimenters and philanthropists” and added to the domain of Natural Philosophers that were breaking away from religious stricture. The term Scientist[1] was coined in 1833 and along with a movement for philanthropy that paid those who were technically able but not wealthy humanity started to move forward very rapidly.

That forward progress in “information / knowledge” fairly free of constraint at that time gave rise to considerable economic growth thus both money and power. Both of which in effect got reinvested in science and thus created not just rapid development in knowledge but society as well. A little less than two centuries has seen a lot of progress in society through industry whilst what some would regard as progress has been a lot slower in fundamental knowledge.

Two things have become clear,

1, Knowledge / information is worth more than profit and personal / organisational power.
2, Knowledge / information only comes about due to profit being reinvested.

It’s a cycle that only a fool would think can be broken…

Because without progress in science the economy will collapse and thus profit and power will be likewise effected.

Part of the current hype behind AI is to replace humans as those who are scientists so the accumulation of knowledge can be automated. But it just does not work that way, because information alone is not knowledge.

Unfortunately not understanding that just demonstrates how foolish those who seek just profit and power actually are.

As more are realising the purpose of humans has become,

“Curiosity about and above all things”.

From which the gaining of information and turning it into knowledge to perpetuate the process is the reason for “the progress” that is the continued human existence.

Cutting back on the gaining of knowledge or science is the real existential threat to humanity and much else besides. Because humans have taken this world beyond a tipping point.

[1] The term “scientist” some claim was actually coined as a sexist insult or term of prejudice. However the word science was around at least a half century before and was based on the latin word for knowledge.

So people claiming to be working in “Knowledge Science” or “Information Science” are exercising a term in a redundant way.

piglet42 August 6, 2025 12:12 PM

“He points out the largest number that has been factored legitimately by a quantum computer is 35.”

Actuially no: “So far as we have been able to determine, no quantum factorisation has ever factorised a value that wasn’t either a carefully-constructed sleight-of-hand number or for which most of the work wasn’t done beforehand with a computer in order to transform the problem into a different one that could then be readily solved by a physics experiment”

Leave a comment

Blog moderation policy

Login

Allowed HTML <a href="URL"> • <em> <cite> <i> • <strong> <b> • <sub> <sup> • <ul> <ol> <li> • <blockquote> <pre> Markdown Extra syntax via https://michelf.ca/projects/php-markdown/extra/

Sidebar photo of Bruce Schneier by Joe MacInnis.