A Flaw in the Scientific Method

The Raven Paradox

How could we begin a blog called “The Paradox Ravens” without first discussing the Raven Paradox? This particular paradox is one that will be close to the hearts of anyone who considers themselves a scientist or an analyst in any capacity, as it challenges the way that we become confident in a hypothesis based on our observations.

In the 1920s, a group of philosophers in Vienna created a movement called logical empiricism. As the name suggests, they were a group of people who summarised the scientific method as being a combination of logic (correct reasoning) and empiricism (all concepts originate from experience). Essentially, they believed that knowledge comes from observing the world and then generalising the findings using logic. In the 1940s, a philosopher named Carl Hempel came up with “The Raven Paradox” in order to challenge this line of thinking.

The Raven Paradox

Hempel thought of a statement using the ideals of logical empiricism. Imagine you have the hypothesis: all ravens are black. To prove this statement you could go out and obtain empirical evidence (according to The Cornell Lab of Ornithology the best way to attract ravens to your backyard is by leaving out seed, grain, pet food or just leaving the lid off your rubbish bin. They do admit that it might cause more damage than they’re worth...). However you decide to source ravens, you would then observe their colour. Let us assume that you do spot a lot of ravens and they are indeed all black. As a logical empiricist, you are pleased - every black raven you spot should make you more confident in your hypothesis that all ravens are black. If you were to formalise this inductive logic into a general statement, you could say: “by observing increasing numbers of A with property that they are B, this supports the hypothesis that all A are B”. You can test this statement by replacing A with ravens and B with black. Nothing controversial so far.

Carl Hempel came up with the analogy in the 1940s.

You can use something called logical equivalence to change the statement we made above. As a very loose definition, logical equivalence is the concept that more than one statement can have the same meaning. Logical equivalence is quite a broad and formal field in logic and mathematics but if you apply some of the ideas to language, some of the implications are fairly comical. One of the best examples of this that I found was the comment “I thought you were younger than that” compared to “you’re older than I thought”. Both expressions logically equivalent, one far more likely to upset someone than the other (I think humans just value context far more than logic but that’s probably a story for another time). Ok, back to the topic. There is a type of equivalence called contraposition where you switch the hypothesis and conclusion of a conditional statement and then you negate both. As an example, the contrapositive of the saying “If it rained last night, then the plants are wet” is the statement “if the plants are not wet, then it did not rain last night”. The key feature here is that both statements are true. Via contraposition, the statement “all ravens are black” is equivalent to “all non-black things, are not ravens”. This seems weird, but actually if you read it a few times it is not controversial.

Here is where the paradox comes in. We go back to our original statement with As and Bs. We can replace A and B, this time with “non-black things” and “not ravens” respectively, which gives “by observing increasing numbers of non-black things with the property that they are not ravens, this supports the hypothesis that all non-black things are not ravens”. However, the expression “all non-black things are not ravens” is equivalent via contraposition to “all ravens are black” as we saw earlier. This changes the statement to “by observing increasing numbers of non-black things with the property that they are not ravens, this supports the hypothesis that all ravens are black”. This is a weird statement. The paradox is occasionally called “the paradox of indoor ornithology” because this statement implies that you can sit in your living room in the evening and observe your green slippers, your red coffee mug and your pink pyjamas and each one of these observations will increase your confidence that all ravens are black.

Solutions

Some of the more well-read ornithologists amongst you are thinking “wait, there are non-black ravens!” and you would be right. Leucistic ravens have been sighted and they are white - here is an article about Jasper for example (warning: not the cutest looking bird in the sky...). So when we come to think about the resolution of this paradox, we are addressing this line of thinking rather than whether specifically ravens are black. There are many proposed resolutions to the paradox. I will give you three possible, well-known suggestions (including my favourite).

Hempel’s Resolution

Hempel, who came up with the paradox in the 1940s, provided his resolution in the 1960s (I think he was busy doing other stuff…). He basically just said that by observing something that is not black and not a raven (like green slippers for example) you do support the hypothesis that all ravens are black, even if by a tiny amount. Although this might not satisfy our intuition, it does mean that the scientific method remains unchanged and there is essentially no paradox to worry about.

Popper’s Resolution

Karl Popper was a big believer in falsifiability, that is, that science is about disproving theories rather than proving them. In other words, finding evidence that disproves a theory is prioritised over evidence that proves a theory. This implies that observing a black raven only disproves the hypothesis that no ravens are black, just as observing a non-black raven disproves the hypothesis that all ravens are black. Although this is an interesting concept it does imply that seeing a large amount of black ravens does nothing to support your hypothesis that all ravens are black. The hypothesis has simply not been disproven yet.

(Standard) Bayesian Approach

Full disclosure, this is my favourite resolution and I’m pretty biased as I use Bayesian statistics in my research. There are a great many Bayesian solutions to this paradox, however, Irving Good’s presentation in 1960s is potentially the best way to describe the underlying theory (and it was later coined the Standard Bayesian solution). I want to avoid going into huge mathematical detail so I will try and summarise without technicalities but I will mention names of theorems for people who are interested.

Bayesian statistics was named after Presbyterian minister Thomas Bayes and is essentially the application of Bayes’ Theorem, which is a fundamental and surprisingly simple result of probability theory. Bayes’ Theorem is a formula that updates probabilities given new data.

Good calculated the weight of evidence that was provided by observing a black raven or green slippers with respect to the hypothesis that all the ravens in a collection of objects are black. The weight of the evidence is given by the logarithm of the Bayes factor (essentially a rearrangement of Bayes’ Theorem) which updates the odds of the hypothesis. We assume that the number of ravens is very small in comparison to the number of non-black objects (which is probably true). You can then derive that the evidence gathered by observing green slippers is basically zero in comparison to spotting a black raven. If you are interested in the maths, a full derivation can be found here.

A schematic illustrating relative changes in confidence for different observations according to the Bayesian method. From top left round clockwise: non-black non-raven, black non-raven, non-black raven, black raven. Red dashed line and blue solid line depicts current and updated confidence level respectively.

Where does this leave the Scientific Method?

This is by no means a fully answered problem and there are still academic papers being written today on the topic. The paradox caused some philosophers to completely disregard the scientific method. Paul Feyerabend, for example, came up with methodological anarchism, that is in science, there are no universal methodologies. There are also a few variations on the paradox that cause problems. An example is the statement “The king of France is bald”. By the Bayesian method, every non-bald non-king provides confidence that the king of France is bald, however, there is no king of France and so this method can give you confidence in a hypothesis that is false (there is actually a solution to this using an extension of the Bayesian method).

For me, I think that this paradox is veridical (i.e. it is true but sounds weird). I think that the paradox just highlights the need for context in scientific experimentation. For example imagine if there were 21 objects in the universe, 11 ravens and 10 non-ravens with 8 of the non-ravens also being non-black. Would it not be more efficient to check if the 8 non-black objects were non-ravens rather than checking that the 11 ravens are all black?