Les politiques d'austérité : à cause d'une erreur Excel ?

Hey guys! Today we're going to talk about something that's a little different to what we usually discuss here; we're going to talk about economics. Though, we're not exactly going to talk about its governing principles - we've got our fair share of people already doing just that, better than me :). Instead, we're going to look at a little annecdote. An annecdote that shows us some rather intersting things on how science is used when making political discussions. It might even be useful to us tod

ay. [Intro theme] Our story begins in 2010, in the very midst of an economic and financial crisis, (one which began a couple of years earlier). The governments of all nations are trying to figure out how they can logically get themselves out of this crisis in the quickest and most efficient way possible. It is at this time, that two economists, Kenneth Rogoff and Carmen Reinhart, publish a scientific article that is about to become extremely influential. Now, these two aren't exactly anybody; th

ey're two big names from the university of Harvard, in the United States. Moreover, they published their article in a specialist review: the 'American Economic Review' which is one of the most prised reviews for economic research. So even at first glance, we're dealing with something rather important. The article in question is titled: 'Growth in a Time of Debt', [French translation of aritcle name]. Its goal is to ascertain as to whether there is a link that exists between the economic growth o

f a country and its level of debt - more precisely, its public debt accumulated by the state. To do this, the two researchers based their data on the economic statistics - going all the way back to 1946 - of 20 western countries: the United States, France, Portugal, Germany, et cetera. They tried to see if there is, on average, a link between (economic) growth and debt. To clarify, the debt we're talking about here is not a budgetary deficit for a single year, but instead total debt accumlated o

ver time, year after year. As (evidently) this amount is contingent upon the economic size of a country, we would commonly express this as a percentage of GDP. Let us look at an example: the GDP of Germany in 2018 was 3344 billion Euros, whilst its cummulative debt was 2070 billion. Therefore, the cummulative debt of Germany is (valued at) approximately 62% of its GDP. Do note, that since this is an accumulation of all of the debt of a state, i.e. something built up year upon year, an amount cou

ld well exceed 100% of GDP; there is no logical contradiction. At the time of this video, Italy's cummulated debt is at about 135% of its GDP. Alright, so, our two economists from Harvard are collecting statistics on the economic growth and debt of each of these 20 counties, between 1946 et 2009. Let us imagine that they keep a large spreadsheet with all these figures, and that, by looking to see if there is link between the two, they produce the following analysis... Foremost, they group the co

untries into four types of cases: for each country and for each year, its debt is either between 0 and 30%, or between 30 and 60% , between 60 and 90%, or, (well) over 90% They then look at the average economic growth in each of these four cases. Here is what they find: in the first case, when debt is below 30% the average rate of econmic growth is 4.1%; in the second case, 2.8%: a little less; in the third case, between 60 and 90%, it's still at 2.8%; but once we get above 90%, bam! No more ave

rage economic growth at all; in fact, there is a very slight recession: – 0,1%. So this is very interesting. On one hand, it would seem to show that there is link between debt and economic growth, which, from an economic standpoint, seems certainly plausible. Sure, we can consider that at high levles of debt is associated with weaker economic growth. Why not? But, above all, the study shows us that there exists a perilous threshold: below 90% debt, things are all fine and dandy, yet, above 90%,

growth plummets drastically; there's a fatal threshold. The existence of this threshold of 90% is the main message of the article. And, due to the reputation of authors, and that of the journal in which it was published, it played, or would certainly seem to have played, the role of a rather important influence on the economic policies that were put into place at the time. (I take a moment to remind you that this happens in 2010.) Behind the conclusions of these two economists, there is this cle

ar, driving, idea that, despite the crisis, it is crucial to prevent public from debt exceding the fatal threshold of 90%. If we follow the reasoning that going over this threshold risks lastingly crippling econmic growth, all must therefore be done to avoid it - i.e. we must implement policies of financial austerity, keep a tightness of budget, save on public services, et cetera. This conclusion of Reinhard and Rogoff's article on the 90% threshold, is found multiple times in the political and

economic debates that took place at the time. In April 2011, for example, 40 US senators interviewed the two economists. And, as shown in these excerpts, the debates revolved around this famous 90% threshold. Closer to home, the European Comissioner for Economic Affairs, Olli Rehn, would write to the respective Finance Ministers of the Union "It is widely understood, on the basis of authoritative research, that when the level of public debt reaches approximately 90%, it tends towards a negative

impact on the dynamism of the economy, which is in turn translated into weak economic growth for several years." The UK Chancellor of the Exchequer (Finance Minister) made a similar declaration at the time: "Latest research suggests that once debt surpases approximately 90%, the risks of a significant negative impact on long term economic growth become highly detrimental." (translated from the French on screen) In the Anglophone world, Reinhart and Rogoff's conclusion became the subject of artic

les in 'The Economist', the 'New York Times' the 'Wall Street journal' the 'Washington Post', and so on. In universities and academia, the article had an enormous influence: according to Google Scholar, it has to this day been cited more than 3300 times in scientific works. However, the slight issue with this story, as I'm sure you've already guessed from the title of this video, is that the conclusions of this article that have been so influential, are based on a calculation made via Excel; a c

alulation that had a rather problematic typo in one of its formulae. The story behind the discovery of this error is rather interesting. It took place a while later - 3 years later, to be precise - in 2013, when an economics student by the name of Thomas Herndon was set an assignment. He was a student at the university of Massachusetts, in Amherst, not too far from Harvard. One of his professors had set him a fairly ordinary-looking task: to take an economics article of his choice, and attempt t

o replicate its results. Thomas Herndon decided to choose the so influential article of Reinhart et Rogoff, and thus set out to find the data, and try to replicate their analysis. Yet, it was impossible to reproduce their results. He spoke to his professors who retorted that such a task should not be so difficult, seeing as the data is readily available. After all, there is but the simple feat of taking averages; it could not be that hard. But no, regardless, they could not produce the same figu

res as Reinhart and Rogoff. With support of his professors, who are also began to have their doubts, our student wrote to the two Harvard economists to ask them for detials on their calculations. Reinhart and Rogoff send an Excel spreadsheet containing the data they used and the analysis they did. And it is there where our studend discovers three problems: The first is that the two Harvard economists calculated their averages using an unconventional method - not that it is mathematically incorre

ct, but it was rather odd, as the way it allocates different weights to each country is seemingly random. For example, with their method of calculating averages, some years in New Zealand ended up having an enormous impact on the result - and even more significantly than this, those of the United Kingdom. Secondly, the Haravard economists excluded certain variables from their analysis that were apparently available. Notably, those involving Canada, New Zealand and Australia before 1950. As it ha

ppens, this was a period of strong economic growth for these countries, despite having a level of debt above 100%. It is difficult to discern from the article whether these data were negelcted due to having been forgotten, whether its exclusion was justified, or whether it was omitted to intentionally skew the result; Reinhart and Rogoff did not mention these omissions. And finally, the third issue, the most spectacular of the three: there was a large gaff in one of the formulae of their Excel f

ile. To achieve their final result, i.e. the figures they had published, (those I had mentioned,) Reinhart et Rogoff took an average that completely excluded Australia, Austria, Belgium, Canada, and Denmark, the first five countries listed alphabetically. We can see the reason in this screenshot from the Excel file obtained by journalist Mike Konczal. It shows clearly the error: taken average taken covers only the first 15 of the 20 total measured countries. Perhaps a formula was not updated, or

the sufficient amount of cells was not selected. Overall, what was demonstrated by our student, and subsequently published in an article he co-wrote with his professors, was that, when we account for these three errors, the result changes drastically. Here is a 'before' and 'after' comparison: we have here the average rate of economic growth in all four cases (less than 30% debt, 30-60, 60-90, and above 90%). For the first three groups, following the adjustment we find figures analogous to what

Reinhart and Rogoff had found: But, for the last group, where debt excedes 90%, we find 2.2% as the rate of growth, and not the -0.1% initially affimred by the two authors from Harvard. Quite the significant difference! Athough, granted, we still have the general conclusion that higher levels of debt are correlated to somewhat weaker rates of economic growth, the central message, that of the fatal 90% threshold, disappears altogether. So whilst (to recap) this seemingly magical threshold has l

argely allowed for the justification of austerity measures in the United States, and in Europe, due to an apparently foolish Excel error, which needed 3 years before it was debunked, (predictably) the two Harvard economists, when this happened, barely admitted their error lip service, saying that it does not change their conclusions! This story, in all its incredulity, led me to want to investigate the data myself, to figure out more precisely what was going on. Alright, so I had already read th

e original article; it's rather short and is a fairly straightforward read. Yet, without being an expert in econometrics, I still found it surprisingly lacking when it came to its statistical analysis: it looks only at averages; there's no mention of a standard deviation, and don't even get me started on whether there is a test for statistical significance. When we bare in mind the publication standards of academia, including economics, I was a little surprised. I asked myself how this article c

ould have been published in what is pretty much THE reference review for economics, and I eventually found my answer. This article did not follow the usual process of peer review. I've already spoken about this concept in my video on "how to read a study". When researchers, from any discipline, write a scientific article to report their discovery, said article is normally submitted through a process of rigorous re-reading by other researchers in the field, whose job it is to verify that the arti

cle meets the sufficient standards of quality in order to be published. But, manifestly, for this issue of the American Economic Review in which the article was published, there was no formal re-reading process by peers. By and large, we trusted the authors based on their pedigree. To better underand what was 'hiding' behind these average levels of economic growth calculated by the individuals in question, well, I wanted to see the data myself, with my own eyes. I'll show you a few simple analys

es we can do. So, to be clear, these data are readibly available; I'll include the links to them in the description. And I have here a table, where each line has a country, a year, and two values: one corresponding to debt, and the other to economic growth, both as a percentage of GDP. There are roughly 1200 lines, and we have 60 years' worth of data for each of the 20 countries. To begin with, we can start by simply making a graph that shows economic growth as a function of debt for every line

of data. Here's what we get: We can zoom in a little on the section that is of interest to us, but overall we can see that... well, it's a mess! There are points of data everywhere; we're quite far from a mechanical, automatic, or even absolute link between debt and economic growth. And that is nothing surprising. It's quite normal: the economic growth of country depends off a whole load of factors, not only level of debt. But to present (off the cuff) only averages by category, without accounti

ng for spread of data, well, it gives the impression that the link is far stronger than it actually is. Though, despite everything else, when we properly look at the graph, we can see the slight effect: a little less growth for higher levels of debt. If we want to quantify this more precisely, we can, for example, do a linear regression analysis (something rather simple) and we can find the following result: that there is indeed a link, and that it is statistically signficiant: that it is not ju

st due to chance. The coefficient shows that a 10% increase in debt is associated with on avergage, 0.18% less economic growth. So, the general effect would seem to exist; there is a correlation. We do not know, a priori, if there is a causal effect one way or the other, or if it is more complicated than that - but one important point is that this effect holds explanatory power for only a very small part of economic growth. To quantify this, we talk about the 'explained variance', this is to say

: the amount of variation in the data that is explained by our factor, in our case: debt. This is the famous R² that people sometimes use (sometimes incorrectly) sometimes in Excel. Here R² is 0.04, which is not a lot. This means that the variable of debt accounts for only 4% of the variability of economic growth. We could try to go a step further and eventually look for non-linear relations, yet we can see that there doesn't exist a fatal 90% threshold at all, neither here, nor anywhere else. I

f there perhaps is a little something going on, it is more likely at around the 30-40% mark, where we have a slight change in gradient. We can even work out that if we are to limit our data, to measure debt between 30 and 150%, the link between economic growth and debt would seem to disappear completely. In brief, we could have quite a bit of fun doing multiple analyses with these data. I'll stop myself here, as it's not my field of expertise, and there are others who have evidently done this mo

re diligently. However, we'll still do a final calculation: if a split my data up into four categories of debt, as done in the article, and I calculate the averages, I find the same corrected figures as those of the researchers at the university of Massachussetts - what a relief! And with all this in mind, I want to illustrate something crucial, which does not only concern economic research, and that is the importance of researchers to publish all their data, and the code used for their analyses

. Lately, in many scientific disciplines, we have been witness to what we would call a "crisis of reproducibility". This is to say that, certain discoveries published by researchers are not able to be replicated, and/or repeated by other researchers in the same field. So clearly, this casts doubt upon the intial conclusions. We understand that in certain cases it can be rather difficult to properly reproduce an experiment that happens to be long, costly, delicate... but when it comes to results

obtained by a numerical calculation, an analysis of data... well, there should be no excuse. Any researcher, from any field of research, should be able to access the code, the data, and be able to verify themselves that everything is correct. And we should not take three years to realise that one of the most influential articles in a field is incorrect due to a mistyped Excel formula, something which, admittedly, can happen to anybody, and hence demonstrates the importance of all of these system

s of verification. To somewhat moderate this conclusion that, perhaps, financial austerity could have been caused by an Excel formula, we could, at the very least, identify two things: the first is that the Excel error is not that which is the most biased in the conclusions of the original article. When we look at the details, we see that it is especially in some part, their weird way of doing averages, and otherwise their seemingly unjustifiable exclusion of certain data, data which evince them

selves entirely pertinent. Second, we can nonetheless ask ourselves about the actual influence the original article had on the adoption of austerity policies in the face of the crisis. As much as it may have been mentioned multiple times as a justification by influential political agents, one can imagine that if this article did not exist, well, these agents would have found other arguments to justify their stances. Confirmation bias, it's a classic. But, alas, if this story would allow us to ad

vance towards a more transparent and reproducible type of scientific research, that already wouldn't be too bad, especially when scientific conclusions are used to guide political action. And yes, all reference to our current situation is wholly intended. There, that's all for this slightly unusual episode. For those who are interested, I am, of course, sharing the data and the code used for the analysis in the description; that's the least I can do. Subscribe, click the bell, and all that, so y

ou don't miss out on anything. Latest news of the channel can be found on our Facebook and Twitter pages, and we shall meet again soon for another of our videos. See you! [Music] – Subtitling: Armel François -