By now, I’m sure you’ve heard of the “Red Button or Blue Button” problem.

Red Button or Blue Button: Everyone in the world has to take a private vote by pressing a red or blue button. If more than 50% of people press the blue button, everyone survives. If less than or equal to 50% of people press the blue button, only people who pressed the red button survive. Which button would you press?

This problem uncovers some surprisingly deep issues about decision theory, collective action, and your ethical obligations to others.

I think you should push blue. Here is a modest case for why.

Let’s start easy. Suppose you only care about your own life. You can visualize all the possible outcomes using this payoff matrix:

If you push the red button, you’re guaranteed to live, and if you push the blue button, you might die. So, if you only care about your own life, you should push the red button, because doing so weakly dominates pushing the blue button.

Takeaway: If you only care about your own life, push the red button.

But do you only care about your own life?

Think about all the other people’s lives that are at stake. Your friends, your family, and all your other loved ones are also playing this game. Random strangers are too. And although you probably don’t care as much about random strangers as you do about your loved ones, you probably care at least a little bit about random strangers.

Further, here’s a fact that you can be almost certain of: of the 8 billion people on Earth, at least one person will press the blue button. I’m not claiming lots of people will, just that, almost certainly, at least one person will. Maybe they’re scared. Maybe they’re confused about the situation. Maybe they think they’re taking the moral high ground. Maybe they’re creating a problem when one doesn’t need to exist. Whatever their reasons, you can be almost certain that at least one person will press the blue button.

And further, however bad their reasons, they don’t deserve to die for it. (Let’s set aside the issue of the risk to yourself for a moment.) I’ve seen comments that say, “The only reason to press blue is to save the other idiots that also pressed blue.” But surely, if someone does something stupid or negligent, I’m willing to bet you wouldn’t suddenly stop caring about their life.

Consider this case:

Beach: You’re relaxing on a beach. Nearby, someone who can’t swim is playing on a pier that extends into the water. They know they can’t swim. They climb onto the rail, fall in, and start drowning. You could easily save them. There is a life jacket next to you, and all you have to do is throw it to them.

This person is certainly an idiot for playing on top of a pier, knowing that they can’t swim. They created a problem when one didn’t need to exist. But, I think, you would still throw them the lifejacket.

Takeaway: You (probably) care about other people’s lives, including the lives of people who make stupid or negligent choices. (And if you’re not convinced, think about your loved ones.)

But Blue Button or Red Button is not like tossing someone a life jacket! There is a risk to your own life if you push the red button. You might care about other people’s lives in general, but you wouldn’t be willing to pay a real cost, like risking your life, to save them.

Notice that we have changed the problem. The problem now is: what should you do in cases in which you must contribute to a public good (saving as many lives as possible, including your own) by incurring a cost to yourself (risking your life)? In other words, the problem is starting to look like a collective action problem.

Here is an example of a collective action problem.

Littering: There is a beautiful public park that you and lots of other people love. There are no garbage cans in the park. You are carrying litter with you. It would be very easy and convenient for you to throw the litter as you pass through the park. Most people would barely even notice it. On the other hand, you could carry your litter with you and throw it away at home, but this would be somewhat annoying and inconvenient.

What should you do? Here is one way to think about this problem. You might reason that it would be better to litter since your tiny piece of trash would make almost no difference to the beauty of the park. But remember, everyone is in the same situation as you are! And if everyone litters, the park would be ruined.

You can again visualize all the possible outcomes using a payoff matrix:

The worst outcome is when you don’t litter, but everyone else does (bottom left). You have the inconvenience of carrying your litter home, but still end up with a dirty park. A slightly better outcome, but still bad, is when you litter and everyone else does (bottom right). There’s no more inconvenience, but your beloved park is still ruined. A good outcome is when you don’t litter, and neither does anyone else (top left). You have a beautiful park, at a small cost to yourself. The best outcome is when you litter, and no one else does (top right). You still have a clean park and the convenience of not carrying your trash home.

The problem is that no matter what everyone else does, it’s better for you to litter. If everyone else doesn’t litter (that is, if we’re in the top row), then you should litter, since you get a clean park either way and a bit of convenience if you litter. If everyone else does litter (that is, if we’re in the bottom row), then you should also litter, since you get a dirty park either way, and a bit of convenience if you litter. Littering strictly dominates not littering.

But since everyone else is in the same situation you are, they will also litter! And you will end up in the bottom right cell, with a dirty park and only a bit of convenience, instead of a much better outcome, with a clean park and only a small amount of inconvenience. Yet intuitively, the best outcome for everyone is the top left cell, where everyone carries their litter home and enjoys a beautiful park.

This is a collective action problem. Each individual must pay a small cost to preserve a public good, the good would be preserved even if any individual refused to pay, but if everyone refused to pay, the public good would be destroyed. A lot of real-life cases are like this. Cases of companies polluting, citizens voting, and even concert attendees putting up their phones to record the show can be thought of as collective action problems.

Given that you care about other people’s lives, Blue Button or Red Button is a lot like Littering. You might reason that it would be better to push red since the probability of your vote making a difference is vanishingly small.1 But remember, everyone else is in the same situation as you are! And if a majority pushes red, then those unlucky people who pushed blue, whom you (probably) care about, will die.

Here again, the worst outcome is when you push blue but the majority doesn’t (bottom left). You end up risking your life and dying, and lots of people end up dying as well. Since we are assuming you care about other people’s lives too (especially your loved ones’ lives), this is the worst outcome. A better outcome, but still bad, is when you push red, but the majority pushes blue (bottom right). You do survive, but lots of people end up dying. Again, since we are assuming you care about other people’s lives too, this is still a bad outcome. A good outcome is when you push blue and the majority does too (top left). Everyone survives, but you had to risk your life. The very best outcome is when you push red but the majority pushes blue (top right). You didn’t have to risk your life, and everyone still survives.

The problem is that, just as before, no matter what everyone else does, it’s almost always better for you to push red (since the probability of your vote being decisive is vanishingly small). If you are confident the majority will push blue, there’s no need to risk your life and push blue - just push red! If you’re confident the majority won’t push blue, then you should push red to save yourself - there’s no need to throw your life away for nothing! Just as in Literring (if we ignore the vanishingly unlikely situation in which your vote is the deciding vote), pushing red strictly dominates pushing blue.

But lots of people are in the same situation as you are. So, we will all end up in the bottom right cell, with a majority pushing red, and a few poor, confused people pushing blue and dying. Intuitively, the best outcome is again the top left cell, where most people push blue, and no one dies.

Takeaway: If you care about other people’s lives, Blue Button or Red Button has a similar structure to a collective action problem, like the problem of littering, voting, or polluting.

What should you do if you are in a collective action problem? Based on the reasoning I gave above, it seems that you should refuse to pay the cost, and so end up in a less-than-ideal situation.

But a hidden assumption crept into the discussion. An assumption about the right theory you should use to make decisions. To see what that assumption is, consider this last case.

Twin Littering: Just like Littering above, except with a twist. This time, you and all the other people who in the park are psychological twins. Because of your genetic makeup, personality, and background, there is a 99% probability that you will all make the same decisions when placed in the same situation.2

What should you do now?

Suppose you think, “I should litter, since no matter what anyone else does, I’m better off if I litter.” But this time, remember that everyone else is your psychological twin, so you can be confident they are thinking the exact same thing. So, if you choose to litter, then you have a 99% confidence that everyone else will make that same decision to litter. On the other hand, if you choose not to litter, you can be very confident that everyone else will also not litter. So, you might reason, it would be better not to litter, since your very decision not to litter will make you confident that no one else will litter either.

If you agree with this reasoning, then you might like Evidential Decision Theory (EDT). Roughly, EDTists think that you should choose the action that you expect will maximize value, conditional on your choosing that action. More formally, EDTists calculate an action’s value by:

where

refers to the probability that you’ll end up in some state S, conditional on taking some action A, and

refers to your utility function U, which spits out a number that represents how much you value a certain action A and a state S.

Essentially, you should take the fact that you chose to litter as evidence for the fact that other people will litter, even though your choosing to litter doesn’t cause them to do so. EDTists will therefore choose not to litter in Twin Littering, because they know that their psychological twins will act just as they do, with a high probability.

However, you might not like this kind of reasoning. You might reason, your choice to litter won’t cause anyone to litter! It just so happens that your actions and everyone else’s are correlated, because you and everyone else in Twin Littering are psychological twins. You should still litter, you might think, because again, no matter what anyone else does, you are better off littering. This is what we learned, you might insist, from looking at those payoff matrices, and finding out that littering strictly dominates not littering.

If you like this kind of reasoning, you might like Causal Decision Theory (CDT). Roughly, CDTists think that you should choose the action that you expect will cause the state with the highest value. More formally, CDTists calculate an action’s value by:

where

refers to the probability that a certain action A will cause a certain state Si.

Essentially, since your choice to litter is not causally connected to anyone else’s choice, CDTists think you should litter in Twin Littering. By payoff matrix-style reasoning, no matter what anyone else does, you are better off littering.

Takeaway: An EDTist will not litter in Twin Littering, and a CDTist will.

So, EDTists will choose not to litter in Twin Littering, while CDTists will choose to litter. What does this have to do with real-life collective action problems, like Littering?

The answer is that although in real-life cases, you do not have a bunch of psychological twins running around, you can still reasonably assume you are very similar to many other people out there. In real life, there are lots of people who have similar personalities, preferences, and backgrounds to you, who reason in the same way that you do, and therefore, who make similar decisions to you when placed in the same situations.

Take Littering. You cannot be 99% confident that everyone will make the same choice that you do, but you can be reasonably confident that lots of people, with similar personalities, preferences, and backgrounds to you, will make the same decision as you do. In other words, if you choose to litter, you can reasonably expect all the other people who are similar enough to you to litter as well. Maybe it’s not 99%. Maybe it’s closer to 50%. But whatever it is, you can be reasonably confident that lots of people will act like you.

So, if you are an EDTist, you will reason that, since many other people will make the same choice as you do, you should choose not to litter, since your choice not to litter will be evidence that lots of other people will not litter as well. (On the other hand, a CDTist will still litter in Littering.)

Takeaway: If you are an EDTist, and there are enough people out there who think as you do, you will (probably) not litter in Littering.

That, finally, brings us back to what you should do in Red Button or Blue Button. First, note that although you don’t have a bunch of psychological twins out there, you can be reasonably confident that there are lots of people with similar personalities, preferences, and backgrounds to you. When given the choice between pushing red and pushing blue, they will deliberate in the same way that you do, reach the same conclusion that you do, and then make the same choice that you do.

So, if you like EDT, you should (probably) push the blue button, because you will take your choice to push blue as evidence for what a chunk of similar people will choose to do. If there are enough people who will deliberate like you and make the same choice as you, then this will result in a blue victory, and no one will die.

However, if you like CDT, then you should just push the red button. You will reason that pushing the blue button won’t cause anyone else to push the blue button too (at best, it will just be correlated) and pushing the red button, as we figured out above, strictly dominates pushing the blue button.

But how many people in the world will deliberate as you do? If, say, only 10% of the population deliberates as you do, this may not be enough evidential lift for it to be worth pushing blue. In other words, it may turn out that pushing blue increases your expectation of a blue victory, but not by much. So, maybe even if you are an EDTist, you should be wary of pushing blue, since you still have a high expectation of all the blue-pushers dying, including yourself.

Takeaway: If you are an EDTist, and there are enough people out there who think as you do, you will (probably) push blue.

So far, we have been thinking about what the right decision is, assuming that you are only trying to maximize what you value. (We assumed that you wanted to save the lives of other people.)

But you might think that this leaves something out. You might think that in a collective action problem like Littering, you shouldn’t just do what is best for yourself or what you value. You might think that decision theory can tell you what to do if you want to get the most value for yourself, but this ignores the ethics of your decision. You might think you have an ethical obligation to not litter, to vote, to not pollute, and in general, to cooperate in collective action problems. If you are sympathetic to this idea, you might think we need to go beyond decision theory… to ethics.

There are many ethical theories that can make sense of collective action problems. One is Kantian Ethics, according to which there are certain ethical constraints on action that go beyond maximizing what you value. Roughly, Kantian Ethics says, before you take an action, think about what rule (maxim) you are acting on. Then, imagine a world in which this is a rule that everyone follows (universal law). Finally, ask yourself, in this world, are you still able to pursue the goals you are committed to pursuing (will)?

For example, in Littering, if you choose to litter, the rule you would be acting on is something like, “It is OK for people to litter to avoid the inconvenience of carrying litter home.” Now, you imagine that everyone else could act on this rule. In such a world, the park would be ruined. And finally, a goal you are committed to pursuing is having a clean park, because you love the park. So, you shouldn’t litter.

So, similarly, in Red Button or Blue Button, if we assume that some poor, confused individuals will press blue, Kantian Ethics would recommend pushing blue as well. If you push red, then the rule you would be acting on is something like, “It is OK not to take on small risks to your life, despite the fact (because of our assumption) that this will result in some people dying.” Now, you imagine everyone else could act on this rule. In such a world, you might find yourself in a situation in which other people need to take on small risks to protect your life, but don’t. And obviously, a goal you are committed to pursuing is not dying. So, you shouldn’t push red.

Takeaway: If you think that you have an ethical obligation to cooperate in collective action problems, you will also think you have an ethical obligation to press blue.

That’s my modest case for blue. Here is the final takeaway.

Final takeaway: If you can be confident at least one other person will press blue, and if you care about other people’s lives, including the life of this one person who might press blue, not just your own, and if you are an EDTist, and if there are enough people in the world like you, who will deliberate just as you do, then you should push blue. If you aren’t an EDTist or if you think there aren’t enough people in the world like you, then you might still think you have an ethical obligation to cooperate in collective action problems, and so you will think you have an ethical obligation to press blue.

In Shawshank Redemption (1994), one of the film’s main characters, Red, played by Morgan Freeman, repeatedly appears before a parole board. Each time, the parole board asks him the same question:

“Do you feel you’ve been rehabilitated?”

He gives careful, rehearsed answers to try to convince them that he has been rehabilitated, but each time, they deny him parole. Finally, at the end of the film, he bluntly responds that he doesn’t think he understands what the word “rehabilitated” means, and that he thinks that the concept is something the board has made up. After that hearing, the board grants him parole. The audience is left with the feeling that the parole board has no clear, rigorous criteria for making parole decisions; they depend on the whims and fancies of individuals who use their intuition and personal judgment to make decisions about whether or not to grant an individual their freedom.

The film is set between the 1940s and the 1960s, before more sophisticated techniques were put in place to make parole decisions. Nowadays, prediction tools use sophisticated algorithms, taking into account a wide variety of factors, including an individual’s criminal history, age at first arrest, substance abuse, employment history, family background, and even attitudes toward crime.

One such tool is COMPAS, developed by a company called “Northpointe” (now Equivant). It uses statistical models trained on historical data to make predictions about things like an individual’s likelihood of violating parole or failing to show up for a sentencing hearing. In theory, it helps parole boards make scientific, data-based decisions, unlike what was depicted in Shawshank Redemption.

COMPAS has been used to evaluate more than 1 million defendants across the country. But in 2016, ProPublica published an article written by Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner, called “Machine Bias”. They looked at over 10,000 defendants in Broward County, Florida. They compared what COMPAS predicted about their likelihood to recidivate or commit another crime to what actually happened. And they made a startling accusation.

They charged that COMPAS was biased against Black defendants. It was much more likely to incorrectly flag Black defendants as “high-risk” and incorrectly flag White defendants as “low-risk”.

In their words,

In forecasting who would re-offend, the algorithm made mistakes with Black and White defendants at roughly the same rate but in very different ways. The formula was particularly likely to falsely flag Black defendants as future criminals, wrongly labeling them this way at almost twice the rate as White defendants. White defendants were mislabeled as low risk more often than Black defendants.

The creators of COMPAS at Northpointe published a response, written by William Dieterich, Christina Mendoza, and Tim Brennan, to the ProPublica article. The research report rejected ProPublica’s claims. They argued that for any given risk score that COMPAS assigned, Black and White defendants with that risk score re-offended at the same rates. In their words,

When the correct classification statistics are used, the data do not substantiate the ProPublica claim of racial bias towards blacks. The proper interpretation of the results... demonstrates that [the risk scores] are equally accurate for blacks and whites.

Here’s the twist – they were both right.

How can that be?

A Toy Model

The problem, it turned out, didn’t have anything to do with parole, sentencing, or racial bias. It was a problem that went deeper, to the mathematics of prediction itself.

To illustrate the problem, let’s use a toy model that abstracts away from the messy details in the data that Julia Angwin and her colleagues used.

It’s a model of grapes. Let’s take 2000 grapes, 1000 green and 1000 purple, and mix them all together. And suppose that some of them are poisoned.

There are many reasons why this model isn’t like the real-life case, besides the fact that the real-life case is about actual human beings and the model is about grapes. Here are a few worth highlighting. First, COMPAS gave individuals risk scores between 1 and 10, while our tool is more coarse-grained. It only sorts individuals into “high-risk” and “low-risk” categories. Second, in our model, grapes are either poisoned or not poisoned, and this is fixed before our tool even looks at them. In the real-life case, individuals in prison who were given higher risk scores by COMPAS were often denied parole and detained for longer. This might affect the outcomes - maybe someone who was given a higher risk score and detained will, because of that, become more or less likely to re-offend in the future. So, we might worry that in the real-life case, predictions had (at least some) effect on the outcomes. Third, since we made up data about the grapes, we know exactly how many grapes are poisoned or safe. In the real-life case, however, the data had to be collected. But, naturally, the outcomes we are able to collect data on are not about whether individuals re-offend, but about whether individuals are re-arrested. If, for example, Black neighborhoods are more likely to be policed, Black individuals are more likely to be stopped, and Black individuals are more likely to be searched, then we might worry that the data COMPAS uses doesn’t completely accurately represent the underlying rate of offending, but rather reflects patterns of policing and arrest.1

Our model avoids some of these tricky issues, and for our purposes, these differences won’t matter, since they won’t affect the points in this article.

Base Rates

Back to the model. Here is a crucial fact about the grapes in our model. The green grapes and purple grapes are not poisoned at the same rates. In other words, they have what’s called different “base rates”.

If we count up all the poisoned grapes in the original pile, we will find that 50 percent of purple grapes are poisoned (500 out of 1000) whereas 25 percent of green grapes are poisoned (250 out of 1000).

A Prediction Tool Sorts Grapes into Risk Categories

Suppose we use an algorithmic tool, just like COMPAS, that makes predictions about whether or not any given grape is poisoned. It doesn’t know what we know about these grapes; it doesn’t know which ones are poisoned and which ones are safe. It sorts grapes into two categories, “high-risk” and “low-risk”, based on some historical data.

Suppose that it isn’t perfect: it puts some grapes that are actually poisoned in the “low-risk” category and some that are actually safe in the “high-risk” category. But it is fairly accurate.

Say that it splits the purple grapes, 750 and 250, into the “high-risk” and “low-risk” categories, and it splits the green grapes, 125 and 875, into the “high-risk and “low-risk” categories, respectively. (I picked these specific numbers somewhat arbitrarily, but with certain constraints that will be clear in a moment.)

There seems to be a problem. There are way more purple grapes in the “high-risk” category, and way more green grapes in the “low-risk” category. In fact, if we try to visualize this data, we can see this more clearly.

What we can see clearly is that the number of purple grapes is much larger than the number of green grapes in the “high-risk” category, and the number of green grapes is much larger than the number of purple grapes in the “low-risk” category.

Percentages of Risk Labels Across Groups

But this, on its own, doesn’t necessarily mean that the tool is unfairly biased. Since the groups have different base rates - the groups are poisoned at different rates - what we need to look at are the percentages. For example, what percentage of purple grapes in the “high-risk” category are poisoned? How does this compare to the percentage of green grapes in the “high-risk” category? (We would then ask the same questions about the “low-risk” category.)

It turns out that the tool we used to sort grapes into different categories ends up with exactly the same percentages across groups. More specifically, exactly 60 percent of both purple and green grapes are poisoned in the “high-risk” category, and exactly 20 percent of both purple and green grapes are poisoned in the “low-risk” category. (This is no accident - I picked specific numbers about how the tool makes predictions to ensure this. In reality, the creators of COMPAS tweaked the algorithm it used to ensure this result.)

In machine learning contexts, this is called “calibration”. Say a tool makes predictions about certain outcomes, like whether an individual re-offends or whether a grape is poisoned. A risk score, like the risk scores our hypothetical tools assigns to grapes, is calibrated within groups just in case: among the individuals who have the same risk score, the actual probability of the outcome is the same, regardless of which group they belong to.2

For us, if the same percentage of purple grapes and green grapes are poisoned in the “high-risk” category (and similarly for the “low-risk” category), then our risk scoring is calibrated. Essentially, a purple grape and a green grape with the same risk score (high-risk or low-risk) must have the same probability of being poisoned.

If a prediction tool is calibrated, it seems as if the tool is fair. In contrast, if it is not calibrated - if say, a green and purple grape in the “high-risk” category have a different probability of being poisoned - it seems as if it is unfair.

This is exactly what Northpointe claimed. COMPAS was calibrated across groups; a Black individual and a White individual with the same risk score had the same probability of re-offending.

But calibration doesn’t tell us the whole story, and this is exactly what ProPublica claimed. ProPublica conceded that COMPAS was calibrated but still claimed it was unfair.

False Positive Rates Differ

To see why that might be, let’s turn back to our grape model, and this time, look at a different graphic.

We are still looking at high-risk and low-risk grapes, but this time, they are grouped by grape color. Now, notice that there are grapes, of both colors, that were put in the “high-risk” category but are completely safe. These are false positives, grapes that are actually safe but incorrectly labeled “high-risk”. For an imperfect prediction tool, it’s inevitable that there will be some false positives; we could only completely avoid false positives if the tool were 100 percent accurate at predicting which grapes were poisoned and which ones were safe.

The important thing to notice now is that our prediction tool resulted in false positives at different rates, depending on the grape color. Look at the “Purple Grapes” section (on the left) and notice how many purple grapes were put in the “high-risk” category, but are completely safe. (For purple grapes, there are 300 safe grapes in the “high-risk” category, and only 200 safe grapes put in the “low-risk” category.) Now, look at the “Green Grapes” section (on the right) and notice that there are very few safe grapes in the “high-risk” category. (For green grapes, there are just 50 safe grapes in the “high-risk” category, and as many as 700 in the “low-risk” category.)

What we can see is that the false positive rate is different across different color groups. The false positive rate for a given group is the percentage of grapes that were incorrectly marked “high-risk”.3 We can see this clearly below. (The false negative rate, the rate at which poisoned grapes were put into the “low-risk” category, is also higher for green grapes, but let’s focus on the false positive rate here.)

There are 300 purple grapes marked “high-risk” but that are actually safe. This is out of a total of 500 safe purple grapes. So, the false positive rate for purple grapes is 60 percent. On the other hand, there are 50 green grapes that are marked “high-risk” but are actually safe. This is out of a total of 750 green grapes. So, the false positive rate for green grapes is only 6.7 percent.

In short, the false positive rate for purple grapes is 60 percent, but only 6.7 percent for green grapes. Think about it like this. Suppose we threw out every grape marked “high-risk”. Now, if you’re a safe, purple grape, you have a 60 percent chance of wrongly being thrown out. There you are, minding your own business with no ability to poison anyone, but 60 percent of the time, you’ll be thrown out anyway. On the other hand, if you’re a safe, green grape, you only have a 6.7 percent chance of wrongly being thrown out. This seems unfair.

This is precisely what ProPublica argued. Even though COMPAS was calibrated – Black and White individuals with the same risk score had the same probability of re-offending – it had different error rates. For example, the false positive rate was much higher for Black individuals than for White individuals. As a result, Black individuals who did not re-offend were much more likely to be predicted to be high-risk than their White counterparts. These individuals were denied parole, forced to pay higher bail, denied release, and even given longer sentences. The burdens of the errors that the tool made were disproportionately falling on Black individuals. (The false negative rate was also higher for White individuals, but again, let’s just focus on the false positive rate.)

Impossibility

Given this issue with COMPAS, it seems natural to think that we should adjust COMPAS so that it no longer has different false positive rates for Black and White individuals. Ideally, our tool would be calibrated (same risk scores, same probability of re-offense) and have equal error rates.

But it turns out, in any realistic situation, this is mathematically impossible. In a result achieved by Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan, and then in another result achieved independently by Alexandra Chouldechova, it was proven that in any realistic situation, you cannot simultaneously have calibration and equal error rates. They proved that in any realistic situation, if we have a calibrated tool, then it must have different error rates, and if it has equal error rates, it must violate calibration.

What do I mean here by “realistic situation”? There are two conditions that need to be met for the impossibility result to hold. First, the tool isn’t perfectly accurate. A calibrated tool that is perfectly accurate will, by definition, make no errors, so the error rate will be 0. So, unless some tool is able to predict with 100 percent accuracy whether an individual will re-offend or not, this condition is met.

Second, and more importantly, there are different base rates between groups. In our grape model, this meant that green grapes and purple grapes were not poisoned at exactly the same rates. In the real-life situation, this means that Black individuals and White individuals don’t re-offend at exactly the same rate.

Now, there may be many reasons for this. Note that, as I mentioned above, the difference in “base rates” isn’t a difference in re-offense rates, but a difference in re-arrest rates. Perhaps, as I mentioned above, this difference is partly due to an excess of policing and surveillance of Black neighborhoods. But even setting this aside, there are many factors that correlate with recidivism that are unevenly distributed between Black and White people. Poverty and income instability, unemployment, housing instability, and less access to education and training can increase the probability of re-offense. If there are disparities in these factors between Black and White individuals, this might also partly explain differences in base rates. Further, returning to a neighborhood with higher crime rates and weaker social services may cause a higher likelihood of re-offending. If there already exist higher crime rates with weaker social services in Black neighborhoods, these differences can be exacerbated.4

Whatever the possible explanations, the important point here is that the base rates differed. If they differ, a calibrated tool that is not 100 percent accurate will make errors at different rates, by mathematical necessity.

Understanding Impossibility: Calibration but Unequal Error Rates

The proofs I mentioned above were for any possible (imperfect) model, with any given population that had differing base rates. It was a general proof. But it might be worth seeing the main idea at work in an example that gives a sense of how (in a realistic situation) calibration and equal error rates are mutually incompatible. Let’s turn back to our grape model one last time.

First, why does satisfying calibration imply that there will be different error rates?

Let’s look at our calibrated prediction tool. For both purple and green grapes, a grape in the “high-risk” category has a 60 percent chance of being poisoned, and a grape in the “low-risk” category has a 20 percent chance of being poisoned. (This is the same graphic as above.)

Now - and this is the key - purple and green grapes have different base rates. Overall, 50 percent of purple grapes are poisoned, and only 25 percent of green grapes are poisoned.

Because of that, two things happen. First, to get calibration, it has to be that in the “high-risk” category, the percentage of poisoned purple grapes equals the percent of poisoned green grapes. So, the prediction tool puts a higher percent of the purple grapes into the “high-risk” bin. (If it didn’t - if say, it put exactly 50 percent of the purple grapes and exactly 50 percent of the green grapes, then because of the different base rates, the probability of a high-risk grape being poisoned would differ across groups, violating calibration.) Now, because the tool isn’t perfect - high-risk grapes are still 40 percent safe - this will inevitably mean putting a higher percentage of safe purple grapes into the “high-risk” bin than green grapes.

Second, because the base rates differ, a higher percent of green grapes are safe compared to purple grapes.

Now, in our model, the false positive rate is

We have seen that (1) the percent of safe purple grapes in the “high-risk” bin is pushed higher, and at the same time, (2) the percent of safe green grapes is higher than the percent of safe purple grapes. In other words, the numerator is pushed up for purple grapes and the denominator is pushed up for green grapes. So, the false positive rate goes up for purple grapes and down for green grapes. That’s why calibration forces the false positive rate to differ when base rates differ.

Understanding Impossibility: Equalized Error Rates but No Calibration

Now, let’s try to understand the other direction - why does equalizing the error rates violate calibration?

Let’s look at our grape model again. Suppose we noticed that the error rates differed based on the grape color: for purple grapes, the false positive rate was 60 percent, but for green grapes, the false positive rate was only 6.7 percent.

Suppose we tried to tinker with the algorithm to fix this, say, by adjusting the predictions so that the false positive rate equalized. We would have to explicitly tell the algorithm, “If you see a purple grape, use the old procedure, but if you see a green grape, use this new procedure so that it is more likely to end up in the ‘high-risk’ category.’

Let’s say we adjust the algorithm so that it puts 500 more green grapes in the “high-risk” category instead of the “low-risk” one. (I picked “500” in order to equalize the base rates, as will be clear in a moment.) Now, there are 1375 grapes in the “high-risk” category, and many more of them are green grapes.

By moving green grapes into the “high-risk” category, we have made it so that more green grapes are incorrectly marked as “high-risk”. This, based on the numbers of grapes we have, equalizes the false positive rates across groups.

But to do this, remember, we had to change the algorithm in our prediction tool! We had to adjust it, so that a green grape with a low predicted probability of being poisoned would still end up in the “high-risk” category. We had to do this while still using the same algorithm for purple grapes. Because of this, we end up with different percentages of poisoned grapes across risk categories. (This is unlike before, in which the percentages of poisoned grapes were identical across risk categories.) It’s no longer the case that exactly the same percent of poisoned grapes end up in “high-risk” and “low-risk” categories, respectively.

The tool, in other words, is no longer calibrated.

Conclusion

What is the takeaway of all this? We can put the problem like this. There are two views we might have on fairness in this machine learning context. On one view of fairness, the calibration view, a fair prediction algorithm is one that is calibrated. Any two individuals from two different groups with the same risk score should have the same probability of re-offending. Defenders of this view would argue that violating this would mean using a different procedure for individuals of different groups. On another view of fairness, the error rate view, a fair prediction algorithm is one that has equalized error rates across groups. Defenders of this view would argue that violating this would mean that one group disproportionately bears the burden of errors made by the tool.

And the takeaway is that in any realistic situation, it’s mathematically impossible for us to design a tool that satisfies both.