The Book of Why: The New Science of Cause and Effect by Judea Pearl (PDF)

Description

A Turing Award-winning computer scientist and statistician shows how understanding causality has revolutionized science and will revolutionize artificial intelligence “Correlation is not causation.” This mantra, chanted by scientists for more than a century, has led to a virtual prohibition on causal talk. Today, that taboo is dead. The causal revolution, instigated by Judea Pearl and his colleagues, has cut through a century of confusion and established causality — the study of cause and effect — on a firm scientific basis. His work explains how we can know easy things, like whether it was rain or a sprinkler that made a sidewalk wet; and how to answer hard questions, like whether a drug cured an illness. Pearl’s work enables us to know not just whether one thing causes another: it lets us explore the world that is and the worlds that could have been. It shows us the essence of human thought and key to artificial intelligence. Anyone who wants to understand either needs The Book of Why.

User’s Reviews

Reviews from Amazon users which were colected at the time this book was published on the website:

⭐It is doubtful that Professor Pearl is at all surprised by the polarity in the reviews of this book. I imagine, in fact, he has a slight smile on his face. This is a man that clearly does not cower from a debate.To me this is not so much a book about science but a book about statistics, which are used almost universally. Many of his examples involve science – hard or social is irrelevant – because that is the world he knows. My world is business, and I can tell you from experience that everything he says about the disregard for causality and the limitations of linear statistics using data alone is spot on.The book covers many fronts but the overarching theme is causality. Why? When we investigate cause and effect how do we know that we have reached the right conclusion without challenging that conclusion, both intuitively and using the tools of mathematics?One of the great myths of science today is that we have conquered the causality problem. We haven’t. Most scientific discoveries are ultimately proven wrong, or at least incomplete. Major drug studies cannot be replicated. And peer review alone – the gold standard of proper science – is not, by itself, any guarantee of truth. In a recent study of scientific papers published on COVID-19, all of which were peer reviewed before being published in prestigious journals, the researchers found that a surprising number ultimately had to be retracted.In my world, the world of business, the results are both staggering and a bit horrifying. A large percentage of students graduating from university today with an interest in business have degrees in something to do with data: data mining, data analysis, Big Data. Data is the new marketing.If you want to launch a program or make an investment you must first make the “business case.” That means you must create a statistical case, almost always based on data. Unfortunately, these cases are often wrong and businesses continue to make bad investments.The preoccupation with data is based on the belief that “data are facts.” But that’s only partially true. Data are facts only in a specific context. And there can be an infinite number of contexts in the real world, a world that is constantly changing.With data in hand, people are no longer asking why. They are no longer even bothering to access their intuition to ask what they might be missing. Intuition, in fact, has become a dirty word, something akin to voodoo or folklore.When it comes to AI, Professor Pearl notes that we are not as far along as many people assume. We are decades away from AI that is even remotely humanlike. Because, as Pearl notes, machines cannot imagine what isn’t. They cannot ask why at even the simplest level.Yet humans, even young children, do it all the time. At least we used to. Which is why I don’t believe we will ever create AI that is humanlike. We don’t yet understand how or why humans think intuitively and what prompts us, or allows us, to imagine alternate realities. How can we teach machines to do it?We can only use algorithms, piled one on top of the other, to calculate a probable answer. And while machine learning can make these algorithmic machines incrementally more accurate, I do not wish to defer to an incrementally more accurate answer when it comes to the big issues of life and society. Or my health.It has been widely reported, for example, that the engineers of Google are no longer entirely sure how their search engine works. It’s too complex. Which is why modifications are not just calculated and applied. They are tested first, on a large test database, to see what results they get. Those results are then reviewed intuitively to see both if they make sense and are what the engineers expected.And that is how we should treat all statistics. Why? Why? Why? Professor Pearl has given us some tools to help in the process. But he has not given us a final solution, as even he admits. Nonetheless, he has moved us down the path. His methods still require assumptions and work largely in the world of probabilities.This book will be a tough read if you are uncomfortable with mathematics. And there are a lot of models and formulas that will be impossible to decipher if you don’t speak the language of mathematics. In every case where he offers a formula, however, he explains what it says, so that while he admits a personal fondness for formulas you can really just ignore them and still get a lot from this book.He is a little harsh, however, regarding other people in the scientific world, past and present, some of whom have obviously offended him in the past. I found that a little off-putting, which is the only reason I didn’t rate the book a 5. Nonetheless, this is an insightful book by a passionate man and I believe I invested my time wisely in reading it.

⭐Given a valid causal framework, this book shows how to use collected data to answer previously unanswerable questions. I’m convinced the process is good. It doesn’t dive into causal discovery or the process of validating a causal framework—that is left to the scientist/user. This is the missing link, and it’s a huge gap because without this link his process is worthless. It is up to you still to make his process work.The largely undetected/unacknowledged limitation on AI/ML is their inability to validate the generalizations they make during training and use during testing/fielding because they invoke simple enumeration to find associative relationships or correlations and not causal relationships. The author mentions there is no science without generalizing (without induction), but does not cover how to validate generalizations, which is desperately needed if AI is to act on valid generalizations. The scientific method properly understood is a method of induction to find causal relationships via method of difference and similarity. The author’s understanding of the scientific method falls short and adopts the common understanding found in most textbooks, which is wrong and hinders causal discovery.His said he wouldn’t define casualty, and provides these reasons in chapter 1: “Any attempt to ‘define’ causation in terms of seemingly simpler, first-rung concepts must fail. That is why I have not attempted to define causation anywhere in this book: definitions demand reduction, and reduction demands going to a lower rung.”But he defined it only a chapter before in the introduction and says it’s simple: “the definition of “causation” is simple, if a little metaphorical: a variable X is a cause of Y if Y “listens” to X and determines its value in response to what it hears.”This is a huge editorial oversight and is likely to confuse the reader. The definition he gives is good enough to understand what he’s talking about, and good enough to generally reveal the value of his method, but it’s inadequate for going the next step of causal discovery.The way Aristotle considered causality is the application of the law of identity applied to action—this is the proper conceptualization of causality in my view. Given on object with X properties, doing Y to the object will cause the object to do Z every time because of its X properties. This allows us to generalize because everything with X properties will necessarily have to act the same way, Z, when doing Y to it. Y causes object X to do Z, because it is X; logically translates to all X will do Z when Y acts on it—the generalization. The scientific method when properly understood and applied is a method to discover “Y causes object X to do Z, because it is X” thus allowing the validated generalization “all X will do Z when Y acts on it”. This in short is the missing link we all need if the generalizations we use and act on are to be valid.

⭐We have all heard the old saying “correlation is not causation”. This is a problem for statistics, since all it can measure is correlation. Pearl here argues that this is because statisticians are restricting themselves too much, and that it is possible to do more. There is no magic; to get this more, you have to add something into the system, but that something is very reasonable: a causal model.He organises his argument using the three-runged “ladder of causation”. On the bottom rung is pure statistics, reasoning about observations: what is the probability of recovery, found from observing these people who have taken a drug. The second rung allows reasoning about interventions: what is the probability of recovery, if I were to give these other people the drug. And the top rung includes reasoning about counterfactuals: what would have happened if that person had not received the drug?Intervention (rung 2) is different from observation alone (rung 1) because the observations may be (almost certainly are) of a biassed group: observing only those who took the drug for whatever reason, maybe because they were already sick in a particular hospital, or because they were rich enough to afford it, or some other confounding variable. The intervention, however, is a different case: people are specifically given the drug. The purely statistical way of moving up to rung 2 is to run a randomised control trial (RCT), to remove the effect of confounding variables, and thereby to make the observed results the same as the results from intervention. The RCT is often known as the “gold standard” for experimental research for this reason.But here’s the thing: what is a confounding variable, and what is not? In order to know what to control for, and what to ignore, the experimenter has to have some kind of implicit causal model in their head. It has to be implicit, because statisticians are not allowed to talk about causality! Yet it must exist to some degree, otherwise how do we even know which variables to measure, let alone control for? Pearl argues to make this causal model explicit, and use it in the experimental design. Then, with respect to this now explicit causal model, it is possible to reason about results more powerfully. (He does not address how to discover this model: that is a different part of the scientific process, of modelling the world. However, observations can be used to test the model to some degree: some models are simply too causally strong to support the observed situation.)Pearl uses this framework to show how and why the RCT works. More importantly, he also shows that it is possible to reason about interventions sometimes from observations alone (hence data mining pure observations becomes more powerful), or sometimes with fewer controlled variables, without the need for a full RCT. This is extremely useful, since there are many cases where RCTs are unethical, impractical, or too expensive. RCTs are not the “gold standard” after all; they are basically a dumb sledgehammer approach. He also shows how to use the causal model to calculate which variables do need to be controlled for, and how controlling for certain variables is precisely the wrong thing to do.Using such causal models also allows us to ascend to the third rung: reasoning about counterfactuals, where experiments are in principle impossible. This gives us power to reason about different worlds: What’s the probability that Fred would have died from lung cancer if he hadn’t smoked? What’s the probability that heat wave would have happened with less CO2 in the atmosphere?[p51] “probabilities encode our beliefs about a static world, causality tells us whether and how probabilities change when the world changes, be it by intervention or by act of imagination.”This is a very nicely written book, with many real world examples. The historical detail included shows how and why statisticians neglected causality. It is not always an easy read – the concepts are quite intricate in places – but it is a crucially important read. We should never again bow down to “correlation is not causation”: we now know how to discover when it is.

⭐I am always cautious when a book proclaims to be about a new science. I am reminded of Albert-Lazlo Barabasi’s Linked and Stephen Wolfram’s a New Kind of Science. They make big promises but they often fail to deliver, or what they are delivering is something which is more a rediscovery rather than something new.Pearl’s book is similar. His views and methods on causality are important but they are not the only possible way forward even if he is convinced that they are. What he proposes is a new graphical way of looking at scientific problems that allows you to understand causality. His bête noire is statistics which he sees as having obstructed the development of causal theories for the last century. I have to declare here than in some ways I am a statistician and I find his constant going on about how bad statistics is while then using the same language and equations as statistics somewhat annoying. He is right that the founders like Fisher and Pearson were bullies thugs and dictators who straight-jacketed their science for many years. But the Bayesians have largely undone there mistakes. What statisticians are is pedantic, but so are philosophers. Popper tells us we can only disprove and never prove anything but I am pretty sure that the Earth goes around the sun. Pearl is squally pedantic in describing what he will and will not allow to be called causal inference and he creates his own do calculus to represent this. But this has to be reduced to conditional probability (statistics) in order to be able to use data to solve.His diagrams are very useful but again I am unconvinced by the proofs of completeness offered and by the claims that it is a completely objective system. It depends on what terms researchers put in the diagram. Pearl is right that the statisticians were too pedantic and so excluded causal arguments but in trying to establish his method as completely objective I think he falls into the same trap. We have to accept that science is never completely objective. We are always restricted by our language, metaphor and the current state of our imagination. This is not to say his method is not a step forward. It is just to say he claims too much.This book was written to make Pearl’s views more accessible and it is written with a co-author whose presence only shows itself as an example in a later chapter. Most of the time it is written in the first person which is odd for a book with two authors. It is part biography, part history and part textbook. For the most part it succeeds in its aim but the chapters on counter-factuals and mediation are definitely not an easy read and need much better explanations. So while the ideas are important it just doesn’t quite deliver them in an accessible way.

⭐This is not a book on cause and effect in physics. Instead it tells the story og how classical statistics was separated from cause and effect by its development as a mathematical transformation (a so called “reduction”) of observed data, independent of how and why these data were measured. It was argued the the statistical results should be objective without any intervention in the observational process. The resulting correlations cannot, however, tell us anything about cause and effect. R. A. Fisher invented (in 1924) the randomized controlled trial in order to avoid a subjective intervention. This is the old science of cause and effect.The definition og causality is so important, because it determines the time direction of the future and the past. We can only remember the past, not the future. Any intelligence (artificial og natural) must involve causality. This book is about how a new science of cause and effect can be joined to statistics, so a robot with real humanlike intelligence can be created (eventually).This implies that Google’s DeepLearning and TensorFlow cannot possibly be real intelligence. They are data driven like classical statistics and do not allow causality.

⭐This is a book about a fascinating and important subject which is almost incredibly difficult to read. Not because the concepts are excessively challenging, not because the organisation of the book is poor – it is logical, well-structured and uses good examples. Not even because the subject matter is without excitement – the wars of the frequentists and the Bayesians are presented in a suitably dynamic manner.But because it lack formal presentation. Pearl introduces technical terms like do-calculus, confounder, counterfactual, backdoor paths and frontdoor adjustment, and then continues to use them never having given a concise and accessible definition.It’s fine to introduce a new concept by means of examples, derivations and formulae, but at some point it is absolutely necessary to provide a concise and formal definition – maybe even two or three definitions, one in strict formal terms, one in ordinary language and one in terms of the application and use of it “An x is an X if and only if it is a y and painted blue by a left handed kangaroo” so the test is “if you think something might by an X, check if it’s a y and look for kangaroos, blue paint pots and used brushes”.This allows the reader to work through the material which leads to the formal statement, repeatedly if necessary, until he feels happy that he has understood the concept and understands the formal statement. He can then progress to later material, knowing that if the earlier term recurs, and he can’t quite remember what it means (“Was it a left or right handed kangaroo?”) he can go back to the single, concise formal definition, refresh his understanding (or revert to the material which culminates in the definition if it no longer rings a bell) and get back to the new stuff.All in all, an important and fascinating work with a really annoying flaw.I finally resorted to making my own notes about what the concepts meant, but I’d still rather have the words of the author.If a writer doesn’t do this, he makes his work far harder to understand.Pearl writes well

⭐First 2 chapters ok but found it hard going then and have temporarily given up. Did A level maths and a science degree but still found the logic quite hard. Think that is probably necessary given the subject. relates to the difference between the reasoning and understanding of computers and of people. And why the current gap is still there and difficult to cross. (Maybe thankfully!)

Keywords

Free Download The Book of Why: The New Science of Cause and Effect in PDF format
The Book of Why: The New Science of Cause and Effect PDF Free Download
Download The Book of Why: The New Science of Cause and Effect 2018 PDF Free
The Book of Why: The New Science of Cause and Effect 2018 PDF Free Download
Download The Book of Why: The New Science of Cause and Effect PDF
Free Download Ebook The Book of Why: The New Science of Cause and Effect