THE SIGNAL AND THE NOISE by Nate Silver

• Human judgment is intrinsically fallible.
• The original revolution in information technology came not with the microchip, but with the printing press.
• Hedgehogs are type A personalities who believe in Big Ideas—in governing principles about the world that behave as though they were physical laws and undergird virtually every interaction in society.
• Foxes, on the other hand, are scrappy creatures who believe in a plethora of little ideas and in taking a multitude of approaches toward a problem.
• Foxes, Tetlock found, are considerably better at forecasting than hedgehogs.
• Foxes sometimes have more trouble fitting into type A cultures like television, business, and politics. Their belief that many problems are hard to forecast—and that we should be explicit about accounting for these uncertainties—may be mistaken for a lack of self-confidence.
• But foxes happen to make much better predictions. They are quicker to recognize how noisy the data can be, and they are less inclined to chase false signals. They know more about what they don’t know.
• Too much information can be a bad thing in the hands of a hedgehog.
• Hedgehogs who have lots of information construct stories—stories that are neater and tidier than the real world, with protagonists and villains, winners and losers, climaxes and dénouements—and, usually, a happy ending for the home team.
• You can get lost in the narrative. Politics may be especially susceptible to poor predictions precisely because of its human elements: a good election engages our dramatic sensibilities.
• The FiveThirtyEight forecasting model started out pretty simple—basically, it took an average of polls but weighted them according to their past accuracy—then gradually became more intricate. But it abided by three broad principles, all of which are very fox-like.
• Principle 1: Think Probabilistically
• Our brains, wired to detect patterns, are always looking for a signal, when instead we should appreciate how noisy the data is.
• We have trouble distinguishing a 90 percent chance that the plane will land safely from a 99 percent chance or a 99.9999 percent chance, even though these imply vastly different things about whether we ought to book our ticket.
• With practice, our estimates can get better.
• What distinguished Tetlock’s hedgehogs is that they were too stubborn to learn from their mistakes. Acknowledging the real-world uncertainty in their forecasts would require them to acknowledge to the imperfections in their theories about how the world was supposed to behave—the last thing that an ideologue wants to do.
• Principle 2: Today’s Forecast Is the First Forecast of the Rest of Your Life
• Another misconception is that a good prediction shouldn’t change.
• Ultimately, the right attitude is that you should make the best forecast possible today—regardless of what you said last week, last month, or last year. Making a new forecast does not mean that the old forecast just disappears.
• Making the most of that limited information requires a willingness to update one’s forecast as newer and better information becomes available.
• It is the alternative—failing to change our forecast because we risk embarrassment by doing so—that reveals a lack of courage.
• Principle 3: Look for Consensus
• Every hedgehog fantasizes that they will make a daring, audacious, outside-the-box prediction—one that differs radically from the consensus view on a subject.
• Quite a lot of evidence suggests that aggregate or group forecasts are more accurate than individual ones, often somewhere between 15 and 20 percent more accurate depending on the discipline. That doesn’t necessarily mean the group forecasts are good. (We’ll explore this subject in more depth later in the book.) But it does mean that you can benefit from applying multiple perspectives toward a problem.
• The word objective is sometimes taken to be synonymous with quantitative, but it isn’t. Instead it means seeing beyond our personal biases and prejudices and toward the truth of a problem.
• Pure objectivity is desirable but unattainable in this world.
• Wherever there is human judgment there is the potential for bias.
• The way to become more objective is to recognize the influence that our assumptions play in our forecasts and to question ourselves about them.
• The goal, as in formulating any prediction, is to weed out the root cause:
• Olympic gymnasts peak in their teens; poets in their twenties; chess players in their thirties11; applied economists in their forties,12 and the average age of a Fortune 500 CEO is 55.
• To be sure, whenever human judgment is involved, it also introduces the potential for bias.
• most of us are still in a state of mental adolescence until about the age of twenty-four.
• Sanders has no formal definition of what a player’s mental toolbox should include, but over the course of our conversation, I identified five different intellectual and psychological abilities that he believes help to predict success at the major-league level.
• Preparedness and Work Ethic
• Concentration and Focus
• Competitiveness and Self-Confidence
• Stress Management and Humility
• Adaptiveness and Learning Ability
• These same habits, of course, are important in many human endeavors.
• The key to making a good forecast, as we observed in chapter 2, is not in limiting yourself to quantitative information. Rather, it’s having a good process for weighing the information appropriately. This is the essence of Beane’s philosophy: collect as much information as possible, but then be as rigorous and disciplined as possible when analyzing it.
• The litmus test for whether you are a competent forecaster is if more information makes your predictions better.
• Good innovators typically think very big and they think very small. New ideas are sometimes found in the most granular details of a problem where few others bother to look. And they are sometimes found when you are doing your most abstract and philosophical thinking, considering why the world is the way that it is and whether there might be an alternative to the dominant paradigm. Rarely can they be found in the temperate latitudes between these two spaces, where we spend 99 percent of our lives. The categorizations and approximations we make in the normal course of our lives are usually good enough to get by, but sometimes we let information that might give us a competitive advantage slip through the cracks.
• The key is to develop tools and habits so that you are more often looking for ideas and information in the right places—and in honing the skills required to harness them into W’s and L’s once you’ve found them.
• Given perfect knowledge of present conditions (“all positions of all items of which nature is composed”), and perfect knowledge of the laws that govern the universe (“all forces that set nature in motion”), we ought to be able to make perfect predictions (“the future just like the past would be present”). The movement of every particle in the universe should be as predictable as that of the balls on a billiard table. Human beings might not be up to the task, Laplace conceded. But if we were smart enough (and if we had fast enough computers) we could predict the weather and everything else—and we would find that nature itself is perfect.
• Probabilism was, at first, mostly an epistemological paradigm: it avowed that there were limits on man’s ability to come to grips with the universe. More recently, with the discovery of quantum mechanics, scientists and philosophers have asked whether the universe itself behaves probabilistically.
• Perfect predictions are impossible if the universe itself is random.
• The most reliable way to improve the accuracy of a weather forecast—getting one step closer to solving for the behavior of each molecule—is to reduce the size of the grid that you use to represent the atmosphere.
• Chaos theory applies to systems in which each of two properties hold: The systems are dynamic, meaning that the behavior of the system at one point in time influences its behavior in the future; And they are nonlinear, meaning they abide by exponential rather than additive relationships.
• The most basic tenet of chaos theory is that a small change in initial conditions—a butterfly flapping its wings in Brazil—can produce a large and unexpected divergence in outcomes—a tornado in Texas. This does not mean that the behavior of the system is random, as the term “chaos” might seem to imply. Nor is chaos theory some modern recitation of Murphy’s Law (“whatever can go wrong will go wrong”). It just means that certain types of systems are very hard to predict.
• Exponential operations, however, extract a lot more punishment when there are inaccuracies in our data.
• Chaos theory therefore most definitely applies to weather forecasting, making the forecasts highly vulnerable to inaccuracies in our data.
• Sometimes these inaccuracies arise as the result of human error. The more fundamental issue is that we can only observe our surroundings with a certain degree of precision. No thermometer is perfect, and if it’s off in even the third or the fourth decimal place, this can have a profound impact on the forecast.
• Humans can make the computer forecasts better or they can make them worse.
• They are too literal-minded, unable to recognize the pattern once its subjected to even the slightest degree of manipulation. Humans by contrast, out of pure evolutionary necessity, have very powerful visual cortexes.
• The statistical reality of accuracy isn’t necessarily the governing paradigm when it comes to commercial weather forecasting. It’s more the perception of accuracy that adds value in the eyes of the consumer.
• Calibration is difficult to achieve in many fields. It requires you to think probabilistically, something that most of us (including most “expert” forecasters) are not very good at. It really tends to punish overconfidence—a trait that most forecasters have in spades. It also requires a lot of data to evaluate fully—cases where forecasters have issued hundreds of predictions.
• If you compare the frequencies of earthquakes with their magnitudes, you’ll find that the number drops off exponentially as the magnitude increases. While there are very few catastrophic earthquakes, there are literally millions of smaller ones—about 1.3 million earthquakes measuring between magnitude 2.0 and magnitude 2.9 around the world every year.
• This pattern is characteristic of what is known as a power-law distribution, and it is the relationship that Richter and Gutenberg uncovered. Something that obeys this distribution has a highly useful property: you can forecast the number of large-scale events from the number of small-scale ones, or vice versa.
• What happens in systems with noisy data and underdeveloped theory—like earthquake prediction and parts of economics and political science—is a two-step process. First, people start to mistake the noise for a signal. Second, this noise pollutes journals, blogs, and news accounts with false alarms, undermining good science and setting back our ability to understand how the system really works.
• In statistics, the name given to the act of mistaking noise for a signal is overfitting.
• You’ve given me an overly specific solution to a general problem. This is overfitting, and it leads to worse predictions. The name overfitting comes from the way that statistical models are “fit” to match past observations.
• In almost all real-world applications, however, we have to work by induction, inferring the structure from the available evidence. You are most likely to overfit a model when the data is limited and noisy and when your understanding of the fundamental relationships is poor; both circumstances apply in earthquake forecasting.
• Overfitting represents a double whammy: it makes our model look better on paper but perform worse in the real world. Because of the latter trait, an overfit model eventually will get its comeuppance if and when it is used to make real predictions. Because of the former, it may look superficially more impressive until then, claiming to make very accurate and newsworthy predictions and to represent an advance over previously applied techniques.
• We may, without even realizing it, work backward to generate persuasive-sounding theories that rationalize them, and these will often fool our friends and colleagues as well as ourselves.
• The theory of complexity that the late physicist Per Bak and others developed is different from chaos theory, although the two are often lumped together. Instead, the theory suggests that very simple things can behave in strange and mysterious ways when they interact with one another.
• Complex systems seem to have this property, with large periods of apparent stasis marked by sudden and catastrophic failures. These processes may not literally be random, but they are so irreducibly complex (right down to the last grain of sand) that it just won’t be possible to predict them beyond a certain level.
• Indeed, economists have for a long time been much too confident in their ability to predict the direction of the economy.
• There is almost no chance19 that the economists have simply been unlucky; they fundamentally overstate the reliability of their predictions.
• Results like these are the rule; experts either aren’t very good at providing an honest description of the uncertainty in their forecasts, or they aren’t very interested in doing so.
• Getting feedback about how well our predictions have done is one way—perhaps the essential way—to improve them.
• As Hatzius sees it, economic forecasters face three fundamental challenges. First, it is very hard to determine cause and effect from economic statistics alone. Second, the economy is always changing, so explanations of economic behavior that hold in one business cycle may not apply to future ones. And third, as bad as their forecasts have been, the data that economists have to work with isn’t much good either.
• Most of you will have heard the maxim “correlation does not imply causation.” Just because two variables have a statistical relationship with each other does not mean that one is responsible for the other.
• Most statistical models are built on the notion that there are independent variables and dependent variables, inputs and outputs, and they can be kept pretty much separate from one another.39 When it comes to the economy, they are all lumped together in one hot mess.
• A forecaster should almost never ignore data, especially when she is studying rare events like recessions or presidential elections, about which there isn’t very much data to begin with. Ignoring data is often a tip-off that the forecaster is overconfident, or is overfitting her model—that she is interested in showing off rather than trying to be accurate.
• The other rationale you’ll sometimes hear for throwing out data is that there has been some sort of fundamental shift in the problem you are trying to solve.
• The problem with this is that you never know when the next paradigm shift will occur,
• An economic model conditioned on the notion that nothing major will change is a useless one.
• The third major challenge for economic forecasters is that their raw data isn’t much good.
• Most economic data series are subject to revision, a process that can go on for months and even years after the statistics are first published. The revisions are sometimes enormous.
• Any illusion that economic forecasts were getting better ought to have been shattered by the terrible mistakes economists made in advance of the recent financial crisis.
• Statistical inferences are much stronger when backed up by theory or at least some deeper thinking about their root causes.
• If you’re looking for an economic forecast, the best place to turn is the average or aggregate prediction rather than that of any one economist.
• The aggregate forecast is made up of individual forecasts; if those improve, so will the group’s performance.
• When you have your name attached to a prediction, your incentives may change.
• The less reputation you have, the less you have to lose by taking a big risk when you make a prediction.
• Things like Google search traffic patterns, for instance, can serve as leading indicators for economic data series like unemployment.
• Danger lurks, in the economy and elsewhere, when we discourage forecasters from making a full and explicit account of the risks inherent in the world around us.
• Extrapolation is a very basic method of prediction—usually, much too basic. It simply involves the assumption that the current trend will continue indefinitely, into the future. Some of the best-known failures of prediction have resulted from applying this assumption too liberally.
• Extrapolation tends to cause its greatest problems in fields—including population growth and disease—where the quantity that you want to study is growing exponentially.
• Perhaps the bigger problem from a statistical standpoint, however, is that precise predictions aren’t really possible to begin with when you are extrapolating on an exponential scale.
• One of the most useful quantities for predicting disease spread is a variable called the basic reproduction number. Usually designated as R0, it measures the number of uninfected people that can expect to catch a disease from a single infected individual.
• In theory, any disease with an R0 greater than 1 will eventually spread to the entire population in the absence of vaccines or quarantines.
• In many cases involving predictions about human activity, the very act of prediction can alter the way that people behave.
• A case where a prediction can bring itself about is called a self-fulfilling prediction or a self-fulfilling prophecy.
• A self-canceling prediction is just the opposite: a case where a prediction tends to undermine itself.
• Needlessly complicated models may fit the noise in a problem rather than the signal, doing a poor job of replicating its underlying structure and causing predictions to be worse.
• Still, while simplicity can be a virtue for a model, a model should at least be sophisticatedly simple.
• If you can’t make a good prediction, it is very often harmful to pretend that you can.
• The philosophy of this book is that prediction is as much a means as an end. Prediction serves a very central role in hypothesis testing, for instance, and therefore in all of science.
• The key is in remembering that a model is a tool to help us understand the complexities of the universe, and never a substitute for the universe itself.
• Finding patterns is easy in any kind of data-rich environment; that’s what mediocre gamblers do. The key is in determining whether the patterns represent noise or signal.
• Bayes’s theorem is concerned with conditional probability. That is, it tells us the probability that a theory or hypothesis is true if some event has happened.
• Usually, however, we focus on the newest or most immediately available information, and the bigger picture gets lost.
• The idea behind Bayes’s theorem, however, is not that we update our probability estimates just once. Instead, we do so continuously as new evidence presents itself to us.
• Many scientific findings that are commonly accepted today would have been dismissed as hooey at one point.
• Absolutely nothing useful is realized when one person who holds that there is a 0 percent probability of something argues against another person who holds that the probability is 100 percent.
• Making predictions based on our beliefs is the best (and perhaps even the only) way to test ourselves.
• One property of Bayes’s theorem, in fact, is that our beliefs should converge toward one another—and toward the truth—as we are presented with more evidence over time.
• Technology is beneficial as a labor-saving device, but we should not expect machines to do our thinking for us.
• In accordance with Bayes’s theorem, prediction is fundamentally a type of information-processing activity—a matter of using new data to test our hypotheses about the objective world, with the goal of coming to truer and more accurate conceptions about it.
• In chess, we have both complete knowledge of the governing rules and perfect information—there are a finite number of chess pieces, and they’re right there in plain sight. But the game is still very difficult for us. Chess speaks to the constraints on our information-processing capabilities—and it might tell us something about the best strategies for making decisions despite them. The need for prediction arises not necessarily because the world itself is uncertain, but because understanding it fully is beyond our capacity.
• A heuristic approach to problem solving consists of employing rules of thumb when a deterministic solution to a problem is beyond our practical capacities.
• Heuristics are very useful things, but they necessarily produce biases and blind spots.
• Chess players learn through memory and experience where to concentrate their thinking.
• Elite chess players tend to be good at metacognition—thinking about the way they think—and correcting themselves if they don’t seem to be striking the right balance.
• The blind spots in our thinking are usually of our own making and they can grow worse as we age.
• Computers are very, very fast at making calculations. Moreover, they can be counted on to calculate faithfully—without getting tired or emotional or changing their mode of analysis in midstream. But this does not mean that computers produce perfect forecasts, or even necessarily good ones. The acronym GIGO (“garbage in, garbage out”) sums up this problem. If you give a computer bad data, or devise a foolish set of instructions for it to analyze, it won’t spin straw into gold. Meanwhile, computers are not very good at tasks that require creativity and imagination, like devising strategies or developing theories about the way the world works.
• The search results that Google returns, and the order in which they appear on the page, represent their prediction about which results you will find most useful.
• Google’s best-known statistical measurement of a Web site is PageRank,45 a score based on how many other Web pages link to the one you might be seeking out. But PageRank is just one of two hundred signals that Google uses46 to approximate the human evaluators’ judgment.
• Nevertheless, a commitment to testing ourselves—actually seeing how well our predictions work in the real world rather than in the comfort of a statistical model—is probably the best way to accelerate the learning process.
• In many ways, we are our greatest technological constraint. The slow and steady march of human evolution has fallen out of step with technological progress: evolution occurs on millennial time scales, whereas processing power doubles roughly every other year.
• We have to view technology as what it always has been—a tool for the betterment of the human condition. We should neither worship at the altar of technology nor be frightened by it. Nobody has yet designed, and perhaps no one ever will, a computer that thinks like a human being.49 But computers are themselves a reflection of human progress and human ingenuity: it is not really “artificial” intelligence if a human designed the artifice.
• The key thing about a learning curve is that it really is a curve: the progress we make at performing the task is not linear.
• Luck and skill are often portrayed as polar opposites. But the relationship is a little more complicated than that.
• More broadly, overconfidence is a huge problem in any field in which prediction is involved.
• In the United States, we live in a very results-oriented society. If someone is rich or famous or beautiful, we tend to think they deserve to be those things.
• As an empirical matter, however, success is determined by some combination of hard work, natural talent, and a person’s opportunities and environment—in other words, some combination of noise and signal.
• Across a number of disciplines, from macroeconomic forecasting to political polling, simply taking an average of everyone’s forecast rather than relying on just one has been found to reduce forecast error,14 often by about 15 or 20 percent.
• Very often, we fail to appreciate the limitations imposed by small sample sizes and mistake luck for skill when we look at how well someone’s predictions have done.
• Herding can also result from deeper psychological reasons. Most of the time when we are making a major life decision, we’re going to want some input from our family, neighbors, colleagues, and friends—and even from our competitors if they are willing to give it.
• There are asymmetries in the market: bubbles are easier to detect than to burst.
• Noisy data can obscure the signal, even when there is essentially no doubt that the signal exists.
• Scientists require a high burden of proof before they are willing to conclude that a hypothesis is incontrovertible.
• In formal usage, consensus is not synonymous with unanimity—nor with having achieved a simple majority. Instead, consensus connotes broad agreement after a process of deliberation, during which time most members of a group coalesce around a particular idea or alternative.
• The goal of any predictive model is to capture as much signal as possible and as little noise as possible. Striking the right balance is not always so easy, and our ability to do so will be dictated by the strength of the theory and the quality and quantity of the data.
• Although climatologists might think carefully about uncertainty, there is uncertainty about how much uncertainty there is. Problems like these are challenging for forecasters in any discipline.
• Uncertainty is an essential and nonnegotiable part of a forecast.
• The fundamental dilemma faced by climatologists is that global warming is a long-term problem that might require a near-term solution.
• Republicans have moved especially far away from the center,112 although Democrats have to some extent too.
• The dysfunctional state of the American political system is the best reason to be pessimistic about our country’s future. Our scientific and technological prowess is the best reason to be optimistic. We are an inventive people.
• In the field of intelligence analysis, the absence of signals can signify something important (the absence of radio transmissions from Japan’s carrier fleet signaled their move toward Hawaii) and the presence of too many signals can make it exceptionally challenging to discern meaning.
• When a possibility is unfamiliar to us, we do not even think about it. Instead we develop a sort of mind-blindness to it.
• [T]here are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns—there are things we do not know we don’t know.—Donald
• If we ask ourselves a question and can come up with an exact answer, that is a known known. If we ask ourselves a question and can’t come up with a very precise answer, that is a known unknown. An unknown unknown is when we haven’t really thought to ask the question in the first place.
• An unknown unknown is a contingency that we have not even considered.
• Good intelligence is still our first line of defense against terror attacks.
• Where our enemies will strike us is predictable: it’s where we least expect them to.
• Whatever range of abilities we have acquired, there will always be tasks sitting right at the edge of them. If we judge ourselves by what is hardest for us, we may take for granted those things that we do easily and routinely.
• Nature’s laws do not change very much. So long as the store of human knowledge continues to expand, as it has since Gutenberg’s printing press, we will slowly come to a better understanding of nature’s signals, if never all its secrets.
• There is no reason to conclude that the affairs of men are becoming more predictable. The opposite may well be true.
• Our brains process information by means of approximation.8 This is less an existential fact than a biological necessity: we perceive far more inputs than we can consciously consider, and we handle this problem by breaking them down into regularities and patterns.
• Our brains simplify and approximate just as much in everyday life. With experience, the simplifications and approximations will be a useful guide and will constitute our working knowledge.11 But they are not perfect, and we often do not realize how rough they are.
• There is nothing wrong with an approximation here and there.
• The problem comes when we mistake the approximation for the reality.
• Bayes’s theorem requires us to state—explicitly—how likely we believe an event is to occur before we begin to weigh the evidence. It calls this estimate a prior belief.
• the vast majority of the time, collective judgment will be better than ours alone.
• Information becomes knowledge only when it’s placed in context. Without it, we have no way to differentiate the signal from the noise, and our search for the truth might be swamped by false positives.
• To state your beliefs up front—to say “Here’s where I’m coming from”12—is a way to operate in good faith and to recognize that you perceive reality through a subjective filter.
• Bayes’s theorem says we should update our forecasts any time we are presented with new information.
• If our ideas are worthwhile, we ought to be willing to test them by establishing falsifiable hypotheses and subjecting them to a prediction.
• Most of the time, we do not appreciate how noisy the data is, and so our bias is to place too much weight on the newest data point.
• It’s more often with small, incremental, and sometimes even accidental steps that we make progress.
• Prediction is difficult for us for the same reason that it is so important: it is where objective and subjective reality intersect. Distinguishing the signal from the noise requires both scientific knowledge and self-knowledge: the serenity to accept the things we cannot predict, the courage to predict the things we can, and the wisdom to know the difference.
• But our bias is to think we are better at prediction than we really are.