Basketball Referees and Single Points of Failure

Sports referees are supposed to be fair and impartial. They’re not supposed to favor one team over another. And they’re most certainly not supposed to have a financial interest in the outcome of a game.

Tim Donaghy, referee for the National Basketball Association, has been accused of both betting on basketball games and fixing games for the mob. He has confessed to far less—gambling in general, and selling inside information on players, referees and coaches to a big-time professional gambler named James “Sheep” Battista. But the investigation continues, and the whole scandal is an enormous black eye for the sport. Fans like to think that the game is fair and that the winning team really is the winning team.

The details of the story are fascinating and well worth reading. But what interests me more are its general lessons about risk and audit.

What sorts of systems—IT, financial, NBA games or whatever—are most at risk of being manipulated? The ones where the smallest change can have the greatest impact, and the ones where trusted insiders can make that change.

Of all major sports, basketball is the most vulnerable to manipulation. There are only five players on the court per team, fewer than in other professional team sports; thus, a single player can have a much greater effect on a basketball game than he can in the other sports. Star players like Michael Jordan, Kobe Bryant and LeBron James can carry an entire team on their shoulders. Even baseball great Alex Rodriguez can’t do that.

Because individual players matter so much, a single referee can affect a basketball game more than he can in any other sport. Referees call fouls. Contact occurs on nearly every play, any of which could be called as a foul. They’re called “touch fouls,” and they are mostly, but not always, ignored. The refs get to decide which ones to call.

Even more drastically, a ref can put a star player in foul trouble immediately—and cause the coach to bench him longer throughout the game—if he wants the other side to win. He can set the pace of the game, low-scoring or high-scoring, based on how he calls fouls. He can decide to invalidate a basket by calling an offensive foul on the play, or give a team the potential for some extra points by calling a defensive foul. There’s no formal instant replay. There’s no second opinion. A ref’s word is law—there are only three of them—and a crooked ref has enormous power to control the game.

It’s not just that basketball referees are single points of failure, it’s that they’re both trusted insiders and single points of catastrophic failure.

These sorts of vulnerabilities exist in many systems. Consider what a terrorist-sympathizing Transportation Security Administration screener could do to airport security. Or what a criminal CFO could embezzle. Or what a dishonest computer-repair technician could do to your computer or network. The same goes for a corrupt judge, police officer, customs inspector, border-control officer, food-safety inspector and so on.

The best way to catch corrupt trusted insiders is through audit. The particular components of a system that have the greatest influence on the performance of that system need to be monitored and audited, even if the probability of compromise is low. It’s after the fact, but if the likelihood of detection is high and the penalties (fines, jail time, public disgrace) are severe, it’s a pretty strong deterrent. Of course, the counterattack is to target the auditing system. Hackers routinely try to erase audit logs that contain evidence of their intrusions.

Even so, audit is the reason we want open-source code reviews and verifiable paper trails in voting machines; otherwise, a single crooked programmer could single-handedly change an election. It’s also why the Securities and Exchange Commission closely monitors trades by brokers: They are in an ideal position to get away with insider trading. The NBA claims it monitors referees for patterns that might indicate abuse; there’s still no answer to why it didn’t detect Donaghy.

Most companies focus the bulk of their IT-security monitoring on external threats, but they should be paying more attention to internal threats. While a company may inherently trust its employees, those trusted employees have far greater power to affect corporate systems and are often single points of failure. And trusted employees can also be compromised by external elements, as Tom Donaghy was by Battista and possibly the Mafia.

All systems have trusted insiders. All systems have catastrophic points of failure. The key is recognizing them, and building monitoring and audit systems to secure them.

This is my 50th essay for Wired.com.

Tags: , , , , , ,

Posted on September 6, 2007 at 4:38 AM • 23 Comments

Comments

Pseudonym • September 6, 2007 6:04 AM

Having worked on precisely this problem, I can report that the sport that is most vulnerable to manipulation is boxing.

The thing that makes boxing the most vulnerable is the way it’s scored. The score for a round is not known to until the round is complete, making it impossible to audit what points were awarded and why.

Chalmer • September 6, 2007 6:39 AM

This is certainly an issue in many arenas…not just games, IT, brokerages, etc.

For example, I came across a facility where they relied heavily on remaining up and running 24/7. Unfortunately, the system was very complex and extremely high risk. Some components of the system were very dangerous and safeguards were in place to protect the workers. One such safeguard was a shutdown trigger attached to a door. If you opened the door, the components inside shutdown so that you wouldn’t be killed. If the components inside the room shutdown, the entire system shutdown. Once the facility went down, it took considerable expense and time to restore the system back to full operability.

The crazy thing is that the door did not have any sort of controls (besides big warning signs) to keep people from opening the doors and even more importantly…you didn’t even have to open the door…if you bumped it hard enough, the door sensors triggered and bammo! down she went like a house of cards.

I always thought that if a competitor or bad guy wanted to take this facility down, all they had to do was get a janitor on staff who could bump the door while walking by.

The way I understand it, the type of situation, were a seemingly insignificant event could trigger catastrophic results was not uncommon in the industry.

Gerard • September 6, 2007 6:49 AM

Interesting. The NHL now ices 2 referees for each game because 1 wasn’t enough to catch all the holding infractions that were seen to be devaluing the quality of the game. I wonder if the addition of a 2nd referee (they rotate the pairs) acts as an informal immediate audit and a deterrent to this type of monkey business.

Chris • September 6, 2007 6:59 AM

Stragely “Professional Sport” is better off with the possibility of biased referees (gambling of course needs the appeaance of fairness).
If the best team won every match then no one would bother watching. A ‘bad’ ref
enhances the chances of the underdog winning (that is where the money is to be made betting).
As a final point any passionate supporter will tell you there is nothing better than watching your team get battered but hold on and sneak a win at the death with a goal (point touchdown etc) that everyone except the referee could see should never have been given.
I can’t think of any security system that would benefit in this way but the aim of Pro sport is to make money not to produce a fair result.

DBH • September 6, 2007 7:47 AM

You mention two problems, trusted insider and catastrophic failure. Auditing catches trusted insiders, but doesn’t really resolve catastrophic failure. Perhaps some systems (like basketball refs) can’t design around CFs but the idea that a single TSA agent would compromise that systems is bad. In fact, agents regularly cycle through, people are assigned to search lines automatically, and (controversially) flyers records are searched for patterns that might indicate problems. If we include behavioral profiling, and did inspections getting on planes (with random rotation of agents) instead of getting into sterile areas, we would make the system more robust by making a single agent very unlikely to cause a CF…

For basketball, the situation is tougher. A ref must not only call fouls, but control the game. He needs absolute authority for his role; if players could continually challenge refs, it would make play untenable. Perhaps a player can call for an instant replay, but if he looses the appeal, he gets a technical? Unfortunately, so many calls are judgment, not absolute, that such review would only be sensible in the context of the calls made during the whole game. Statistical auditing of each refs calls, by team and player importance, would probably be the best way to detect a problem. But only after the fact.

CGomez • September 6, 2007 8:06 AM

I understand there was no need to get into the minutiae of the popular North American sports, but I have felt for a long time basketball is the most vulnerable to bad or corrupt officiating.

With the sole exception of the NFL’s pass interference rule, no other sport gives the referee more authority to practically put points on the scoreboard (a solid free throw shooter should make at least 70% of his chances).

However, even in the case of the NFL, that play is non-reviewable. It’s considered a “judgment call” and the lead referee has no role in reviewing it. Maybe someday that will change, but it doesn’t change much. It just means both the rear official making the call and the lead official “watching the replay” would have to be corrupted. It would mean the play would be more scrutinized (as it is shown forty million times for viewers at home while they review it), so that is something.

I actually don’t believe that a sole referee would be able to completely turn an NBA game upside down. I think instead what was more likely is that Donaghy simply had so much experience with the players and his fellow referees that he was the ultimate insider. He knew what players’ styles of play were trouble with what referees, and probably knew so much about the game he was a ringer for guessing who would win. Then, he could make a few close calls and make sure of the outcome.

The question is how much the other referees knew. I think at some point they would have to ask, “Hey, what’s the deal, Donaghy?” Apparently, the sports media is beginning to dig out that it should have been obvious, but I disagree. If it were so obvious, the same sports media would have caught it beforehand and started writing something was fishy.

John Moore • September 6, 2007 8:09 AM

I don’t have a problem with monitoring employees per se, but the problem I am seeing is that monitoring is arbitrary. For instance, we monitor for information leakage by employees of customers’ personal information, yet it is okay for an administrative assistant to email an Excel spreadsheet with 5 people’s credit card information to a hotel employee for a conference. Our company has a secure web site for retail customers, yet business customers can email a form with their credit card information to upgrade their account.

NotabigTennisFan • September 6, 2007 8:41 AM

“There is at least one sport with team size greater than 1 and less than 5. Doubles tennis springs immediately to mind.”

Oh yeah there’s lots of action on doubles tennis matches! 🙂

Kevin McGrath • September 6, 2007 8:56 AM

Dealing with trusted insiders means you have to do a very good job of screening new employees and monitoring your existing ones. If you hire and retain people you can trust most of the risks can be minimized but you can still have the occasional person who decides to go off the reservation. In hindsight it seems that a lot of this “loose cannon” behavior was detected but ignored for various reasons.

One other point is that as a manager don’t create and/or support policies that cause disgruntled employees, or at least do whatever you can to keep your trusted insiders as happy as possible.

This is also why you need to have good emergency response plans.

shoobe01 • September 6, 2007 9:57 AM

Or what a criminal CFO could embezzle.

This is a terrific article on how trusted insiders can screw a company (or government) out of lots of money:
http://www.baselinemag.com/article2/0,1540,1972062,00.asp

Lots of good lessons about security, like procedures that are blithely ignored cause the boss-man says its okay.

(And its hard to think of anything else when I see posters extolling individual workers to be ethical, or turn off the light on your cube to save money.)

Parallel • September 6, 2007 10:09 AM

Baseball has a similar problem; the umpire calls balls and strikes and a bias there can make a huge difference. Baseball has figured out how to audit the umpires using QuesTec’s Umpire Information System; see http://en.wikipedia.org/wiki/QuesTec for a brief introduction.

Note that the umpire still makes the calls, and those calls are completely binding on the game at hand. The audits are supplied to the umpire after the game, not during the game itself. That accomplishes the goal of long-term audit while preserving the power of the umpire to make calls quickly. I’m guessing that lots of potential suspicious behavior patterns on the part of an umpire can be tested statistically over the long term.

Carlo Graziani • September 6, 2007 10:13 AM

The trouble with catching NBA refs at this sort of thing is that the signal is buried in a lot of noise.

This problem can be overcome for systemic patterns of refereeing behavior spanning many games over many seasons, since the data can be cumulated until the signal stands out. This is how Wolfers and Price detected signatures of racial bias in NBA refereeing (see http://bpp.wharton.upenn.edu/jwolfers/Press/NBARace/NBARace(NYT).pdf).

Unfortunately, corruption doesn’t normally operate this systematically. In this case, there isn’t a single team or group of players that is being systematically assisted or hobbled over many games. So you can’t prove — even statistically — that a ref had an ulterior purpose in mind in calling some early fouls. That’s too common a circumstance to be suspect in and of itself.

I suppose correlating ref behavior with Vegas odds tables might have some promise. It would be a hell of a data mining project, but at least the Donaghy case might supply a known-positive testing/training case.

SportsFan • September 6, 2007 10:20 AM

“The NBA claims it monitors referees for patterns that might indicate abuse; there’s still no answer to why it didn’t detect Donaghy.”
…
“All systems have trusted insiders. All systems have catastrophic points of failure. The key is recognizing them, and building monitoring and audit systems to secure them.”

Donaghy was eventually outed, though not through the expected means and in the timeliest fashion. Just adding to your point: Closing the time gap between infraction and recognition is important to minimize the risks of on-going abuses. Of course, it does nothing for single-point catastrophes where the perpetrator cares not about any consequences.

Congratulations on your 50th essay!

FP • September 6, 2007 10:40 AM

A soccer referee was jailed in Germany last year for manipulating second and third league games. He was tempted by organized crime figures with sports cars and 5-figure lumps of money.

Manipulation in soccer can work very well because a referee’s decision is final — there is no way for either team to appeal. A single bad decision (such as a penalty kick, calling offside, or pulling the red card (immediate player disqualification) instead of a yellow card (warning) for a foul) can affect the outcome of a game. And it is accepted that every judge will make a bad call once in a while, with the assumption that mistakes cancel each other out over the course of a game.

Certainly, if a referee keeps making bad, partisan decisions, he or she will be investigated, but there is much less scrutiny outside of the first league, and it is near impossible to prove unless there is the “audit log” in the form of a television broadcast.

UNTER • September 6, 2007 10:42 AM

“The NBA claims it monitors referees for patterns that might indicate abuse; there’s still no answer to why it didn’t detect Donaghy.”

Why? Because the NBA’s interest is not in avoiding game-fixing, but in avoiding the appearance of game-fixing. Their goal is to maximize betting, thereby maximizing viewership. The way to maximize betting is to create the appearance that the game results are random, relative to the odds.

Since decent cheaters don’t care about who actually wins, just beating the odds, it’s difficult for single players to identify that. It can only be done by a full audit – if games under a certain ref have a greater deviation from the betting odds, you can be sure that there’s fixing. But to actually investigate and discipline would signal to bettors that games are fixed; by not “finding” cheating, the small-time bettors who fund the system never discover game-fixing.

I see no reason to believe that all major sports are not riddled with fixing. To guarantee that, we would have to see fairly regular disciplining and prosecution of refs – since it’s safe to assume that at least some would be tempted by cheating. How much money changes hand every Sunday in NFL betting? And you’d want me to believe that no one ever pays off a ref or player to affect the spread, or the over-under?

W Ricker • September 6, 2007 11:14 AM

Fascinating discussion of Audit and SPOF; with all the sports analogies in business, using sport as a risk allegory is quite attractive.

On the tangential sporting issues …

Both boxing (mentioned) and figure skating (not mentioned prior!?) have been repeatedly “reformed” to elminate obvious fixing of judges’ scoring. Or at least to hide the appearance of bias. But are they “sports”?

As alluded to up-thread, points-shaving to allow the favorite to win but the underdog punters to win “on the spread” should not affect season standings and may benefit management by keeping interest up – good for advertising/TV-rights, seat sales, concessions. Points-shaving costs the player only individual stats satisfaction, unlike throwing games which hurts the player and teammates and fans.

Team size

But boxing and (singles) skating aren’t “team” sports, even though skaters often skate for national team glory. [whence comes motivation for fixing. But this latter extension would make all “singles” events at Olympic and other multisport Games “team”, homeopathetically diluting the meaning of “team” to nonsense.]

If Doubles Tennis is the smallest “team” sport, so are pro beach volleyball and couples skating and 2-crew bobsled/sleigh. (I decline to add tag-team “wrestling”.) Athletics Relay (with or without hurdles) and swim relay have teamsize 4, also below basketball 5 (on court). Since basketball allows free (re)substitution, the 5 on the court is less relevant than the number in rotation, much as no NFL player in the starting 11 or NHL player in starting 6 plays all 60 minutes. In Baseball and Soccer/Futbol, most of the starting 9 or 11 will finish the match.Effective team size for Basketball, Soccer, Baseball are roughly even.

Among sports with 5 active players, RollerDerby has only one position allowed to score, the only position with assigned offensive responsibility. If bet on points-spread, this is nirvana.

If concerned with points-shaving, professional teams should tie more of compensation to incentives that include scores and assists, not just records and trophies.

[Massive Incentives would solve the aging “Free Agent” wants to finish career with “his team” problem — give ’em the 5 year contract they want, but with guaranteed compensation dropping year on year from the offered 1 year value, and the incentives making up the difference to their desired level or more. If they are as fit to play at peak level for another 5 years as agent claims, they’ll get it all or more.]

Pseudonym • September 6, 2007 8:09 PM

W Ricker: If it’s played at the Olympics, I don’t think it’s fair to quibble about if it’s a “sport” or not. It’s close enough.

Stefan Wagner • September 6, 2007 10:27 PM

@UNTER: The safer you are, your paid referee will fix a game, the higher you may bet, thus influencing the odds, so observarting refs by deviation of results with bets will fail in the long run. 🙂

Michael • September 20, 2007 7:30 PM

@W Ricker: no … NHL player in starting 6 plays all 60 minutes

Actually the goaltender in hockey normally plays the entire game. The other 5 players rotate throughout the game.

Michael

Leg • October 24, 2007 12:17 AM

I am surprised that there are
no mentions of the biggest counter-example I can think of: Ultimate
Frisbee.

Even at the game’s highest levels, at the
UPA National Championships, the sport is completely self-officiated.
There is a code of conduct referred to as “Spirit of the Game” which
dictates that every player is responsible for having a complete
understanding of the rules, that players respect the rules and each
other when making calls, and that no player may engage in “win at all
costs” conduct under any circumstances. There are no referees; in some
hotly contested games in upper-level tournaments, there may be passive
observers available upon request — but their role is strictly to
supplement the integrity of player calls by offering informed, disinterested perspective.

The distributed nature of this self-officiated sport goes right to the
heart of your point regarding security and distributed responsibility.
People who make bad calls in Ultimate are ostracized, and sometimes
banned from the game. I have only seen one or two examples of this in
all my years of play, and in those cases it took particularly egregious
examples of misconduct in order to provoke a punitive response. It
breeds a sense of honor and maturity into the game through its players,
and has a self-reinforcing effect on game integrity (which is the
metaphor for security in this case). From the player’s perspective, the
incentive is wildly in favor of everyone on the field making sure the
integrity of the game stays high.

There was an experiment in the 90’s with introducing refs into
ultimate via a small new league in North Carolina. The idea was that referees would reduce the
number of arguments on the field, keep the game moving more quickly, and
make the sport more tv-friendly. The change in attitude and integrity
with the introduction of referees was instantaneous: Players
immediately shedded any sense of responsibility for game integrity, and
instead started testing the boundaries of what they could get away with
from the refs. They were no longer responsible for upholding the
integrity of the game, so there was no longer any incentive to care
about it. The whole experiment was cast aside shortly thereafter.

Leave a comment

Login

Allowed HTML <a href="URL"> • <em> <cite> <i> • <strong> <b> • <sub> <sup> • <ul> <ol> <li> • <blockquote> <pre> Markdown Extra syntax via https://michelf.ca/projects/php-markdown/extra/

Sidebar photo of Bruce Schneier by Joe MacInnis.