Completeness

Completeness requires that all relevant data is graphed. No cherry-picking is allowed. Any exclusions or censored values must be clearly identified as part the graph, preferably in a note to the graph.

Completeness also concerns with providing all relevant information for decoding a graph, through external and internal identification, including axes scales, false origins, units of measurement, reference lines and functional forms.

Completeness in identification should not be sacrificed for maximising the data-ink ratio but must be achieved with reduced visual prominence. Readability is an important property of completeness that eliminates uncertainty about representation. If encoding is implicit, then the internal graph identification must be comprehensive and eliminate ambiguity.

For example, the graph above on US abortions and cancer screenings discussed in the quality of decoding accuracy, is not only inaccurate but also incomplete, because it does not show the year-by-year data so we have no way of knowing how good are these linear projections. This graph is also incomplete in its lack of accountability, as it does not cite any data source (nor the article). In fact, the data provided by the Centers for Disease Control and Prevention suggests the data used in that graph has been fabricated.

Perhaps the most infamous example of incompleteness is the graph presented to the NASA’s executives on the relation between O-ring strength in the fuel rockets and temperature, on the eve of the Challenger’s launch. This lack of completeness in this graph led to catastrophic decision making.

The original report by Norton Thiokol was the following:

This graph is complete in the sense that it reports all data on all tests done on the pairs of rockets in different temperatures. However, this graph suffers from low relevance as it plagued with chart-junk that makes decoding nearly impossible. Someone from NASA took this graph and turned it into this scatter graph:

This graph is incomplete because it cherry-picked only the data from the tests that recorded O-ring failures. It ignored all data from the tests that were successful (no failures). As a result, the message from this graph is inconclusive. It is hard to say that there is a pattern between temperature and O-ring failure.

A more complete graph is to use the entire data, and perhaps even fit a predictive model of some sort, given that on the day of the launch the temperature was predicted to be between 26-29 degrees Fahrenheit (-3 degrees Celcius):

This graph is complete and suggests certain failure (right-click to open the image in a new tab).

Always show the data

A statistical form of analysis may be use the entire data but a data graphic is never complete unless you actually show the data itself. Otherwise, we cannot conclude with confidence that the data supports the analysis.

This point is best elucidated by the famous Anscombe (1973) quartet problem. Consider the following dataset that describes relations between pairs of variables (y1,x1), (y2,x2), (y3,x3) and (y4,x4):

There is not much one can say by just looking at a table of data. Perhaps I can help you by telling you that the mean and the standard deviation of every x variable is the same up to the sixth decimal point, and the mean and the standard deviation of every y variable is the same up to the second decimal point. What if I also told you that the correlation between all pairs is the same at 0.816, and I to convince you I will also show you the linear regression fit:

Does it help to know this information? Here we have a predictive model that is fitted on the data, but we have no idea of knowing how good is this model because we cannot see the data! Hence, this graph is incomplete because we do not know if the data supports the model, which it obvious does not:

Completeness as a second consideration

Remember that the foremost quality of data graphs is decoding accuracy. Completeness only comes second. That is to say, we can sacrifice completeness in order to satisfy accuracy. But if we do violate completeness, then we must say so.

For example, the graphs below show the relation of abalone height with their diameter from a sample of 4,177 abalone. Naturally, as the height of this marine mollusc increases the diameter should increase as well. The left-hand side graph suggests a very steep relation, probably in the order of 4-5 times increase of diameter for every unit increase in height. Notice that the steepness of the scatter plot relation suggests about 75-77 degrees angle, and this is equivalent to about a slope coefficient of about 4 to 5.

The left-hand-side graph has therefore low decoding accuracy, and this is because of the effect from one extreme value with diameter 1.53. There are two ways to fix this inaccuracy: either extend the scale of both y-axis and x-axis to a common range from 0 to 1.53 given the same unit of measurement, or remove the extreme value. In this case, I prefer to do the latter because this extreme value appears to be an isolate case that is truly extraordinary. As a result, the right-hand side graph now suggests a slope of about 2.3 which equivalent to about 65 degrees angle.

This is a now a more accurate data graph but it is also incomplete. To compensate, I make sure to report in a note to the graph of the exclusion of this one observation.


Back to Decoding accuracy ⟵ ⟶ Continue to Encoding relevance

Demetris Christodoulou