The model, credited with forcing the Government to make a U-turn and introduce a nationwide lockdown, is a “buggy mess that looks more like a bowl of angel hair pasta than a finely tuned piece of programming”, says David Richards, co-founder of British data technology company WANdisco.
“In our commercial reality, we would fire anyone for developing code like this and any business that relied on it to produce software for sale would likely go bust.”…
No one at all has nice words to say about the code and yes, no commercial organisation would use, release or base a decision upon it.
But the real problem at the heart here is the assumption made. That people don’t change their behaviour in the face of changed incentives – they only change their behaviour if government tells them to. Yes, yes, economists use models to make astrologers look good but at least they don;t make that mistake – all the interesting bits in any economic forecast are about how will people change their behaviour?
The quality of the code seems to me to be a red herring. If modeling the epidemic was a doable thing then surely every advanced nation would have the capability? The fact that nobody produced a markedly better prediction than the UK suggests it was not doable with the limited information available and you might as well has asked David Icke rather than prof. “worst case” Ferguson.
Surely other governments went into lock down, never having heard of Pantsdown?
I think that thing of micro-experts being very good people, but needing checking, applies here. You need a few generalists to make things work (Page and Brin were smart guys, but Schmidt knew how to run a software business).
As I understand it Pantsdown has had a far wider influence that just helping Bottler Johnson on to his and our doom.
Lots of nations seem to know of his electronic shit-smearing–remember Bill Gates part-finances the Imperial gang. And–like the climate eco-liars– “scientific” needle-sharers quickly poison each other. Or the increasingly leftist eco-freak science establishment reacts just so despite plenty of skeptics.
BoM4
I thought that was Boris’s job in this case, or the cabinet’s and all their clever SPUDs? Being strategists, asking the right questions, bigger picture perspective and all that….
More lions (™ Steve)
The Government managed to scare people into making the lockdown far more “effective” than anticipated. People are complying not because the Government told them to do so but because probability means nothing to them, and they are terrified of the virus.
What the criticisms of the code are focusing on is how badly the code is written when compared to modern, accepted coding standards – and I’ve little doubt the criticisms are justified (I haven’t read the code). These are simplistic criticisms however. They don’t tell you anything about the quality of the model.
Nor is programming to modern standards any guarantee of “correctness”. Programming & functional mistakes appear in modern code all the time, as any computer user can attest.
The only way to establish the quality of a model is to compare it to reality. This is “easy” with engineering & aerodynamic type models where the inputs & environment are relatively well known and you can continually refine your model against the results of real world experiments.
How do you rationally test a pandemic model. Pandemics, are by their very nature singular events. If that model is being used to manage the pandemic, it becomes one of its own inputs and any errors will typically be magnified.
IIRC the Ferguson model predicted a top bound of 500,000 deaths if nothing was done. To validate the model the UK would have had to continue as if nothing was happening – not just no lockdown but full attendance at work, concerts, sports fixtures etc. How many deaths from CV19 that would have occurred without the lockdown can only be a matter of conjecture.
https://en.wikipedia.org/wiki/All_models_are_wrong
That’s fair, up to a point, but the criticism has gone beyond that already. Ferguson’s model assumed that there would be no change in behaviour given the existence of the pandemic. There were two sets of behaviour and two only. As normal and whatever government insisted as a result of the lockdown.
Did peoples’ behaviour change as a result of the existence of the pandemic and without the lockdown? Yep, sure did and obviously so. Which is where the real error lies.
I’m afraid that spaghetti code is the result of a long evolution. No doubt human DNA has the same defect. The point is does running the code get you anywhere near the right answer, whatever that is.
A major defect of spaghetti code with a variety of dead ends and paths that are no longer followed is that viruses can get in and take it over. Human DNA being no different in effect notwithstanding the differences in structure. Another defect is that it is difficult to follow.
Also, modern programming standards are sometimes just as shite or more so than the older stuff. In my view, often worse.
Flint
“What the criticisms of the code are focusing on is how badly the code is written when compared to modern, accepted coding standards – and I’ve little doubt the criticisms are justified (I haven’t read the code). These are simplistic criticisms however. They don’t tell you anything about the quality of the model.”
But that’s not true. You say yourself that the model only had two options – on or off. Well, that specific point was one of the major criticisms of the model – no possible tweaks for subsequent changes in behaviour – not just the buggy code. Any half competent reviewer (or end user) should have understood that very basic limitation and highlighted it in spades.
And be prepared to stand up and tell the country: “and hence, neither the model nor Piers Morgan were entirely useful for this purpose”, because of such basic limitations, and that the same underlying model had as a result got F&M, etc all badly wrong in the past. If the limitations were understood beforehand, that makes it even worse.
Models have to be responsive to reality (i.e. incorporate feedback from the real world) to be useful as a real-time tool. I’m not sure Ferguson’s model was designed that way. Certainly it’s structure as revealed makes it more or less impossible to be responsive like that in any meaningful sense.
In musing on it I did wonder if a large cooperative ensemble of software entities like Weta used to do the CGI in Lord of the Rings might be more useful in its emergent behaviour. Certainly the software structure would be more amenable to inquiry and optimisation.
Isn’t it that Ferguson has been so consistently wrong about all his predictions: Foot & Mouth, Mad Cow, SARS, H1N1 and on and on? His trousers should have been removed years ago to keep him out of trouble.
AnyF–Project Fear worked this time because of a–temporarily– credible fear of death.
No one not an EU owned clown believed anyone was going to die from Brexit.
But for the germ Hollywood has done 60 years of groundwork.
The Satan Bug/The Andromeda Strain/The Omega Man (3 versions of the Richard Matheson novel)/28 days Later/28 Weeks Later/ Colony/ The Crazies/World War Z / Last Train to Busan/The Walking Dead TV series—and those are just the ones with a pseudo-science infection “explanation” for catastrophe. There are countless zombie films with a supernatural explanation for the mayhem–a vast number.
All that lot have put the shits up folk . All it needed was pundits and political hacks to pertform a publicised pants-shite and the world was off to the races.
Contagion
That’s a good film.
The Omega Man (3 versions of the Richard Matheson novel)
To be pendantic, the Richard Matheson novel is “I am Legend” and one of the film versions was called ‘The Omega Man’.
There’s an analysis over chez Hector Drummond on why an apparently similar model (in the way it sought to work, not the code) produced such abominably wrong results for Sweden.
https://hectordrummond.com/2020/05/14/simon-anthony-why-were-the-predictions-of-the-swedish-model-so-wrong/
The analysis, as perhaps you’d suspect is that if you get the Very Big Effect wrong then chuntering on about details is a waste of everybody’s time. Here, the Very Big Effect is basically getting R (or its surrogate, case doubling time) horribly wrong and getting length of stay in hospital wrong.
Bizarrely, doubling time is directly observable in the population. But they got it wrong.
Of course, before the disease has progressed, you get horribly wrong estimates, and it’s wise to be cautious. But you should be prepared to backtrack when actual data arrives. And so you, as a government, have a responsibility to provide nuanced warnings and carefully-described actions which one hopes to backtrack on – it’s better for most to be pessimistic, and increase sunniness as data arrives than to be overly optimistic and then start upping the fear and doom and lockdown.
Alternatively you can point out Ferguson is a card carrying leftist and that they make no secret of the fact that they see damaging the economy as in their interest. Communists worked out a century ago that communism doesn’t take in the industrial west because people are too rich and too free to find it appealing. Thus they must be made less rich and less free . It is purely coincidental that Ferguson consistently produces false data that will harm the economy yet retains the confidence of those in power.
Andrew C–That is right;
The Last Man on Earth –1964 Italian film with Vincent Price . Closest to the novel and on You Tube in both orig black and white and colourized versions.
The Omega Man–1971 with Charlton Heston–the best IMO
I am Legend –with Will Smith. A poor version but with fantastic CGI vision of post humanity New York reverting slowly to nature.
@Tim
I have no idea what the assumptions were/are behind the model, and I would respectfully suggest neither do you. Understanding the assumptions are critical in interpreting the results, but are too easily ignored, forgotten or overlooked
The initial 500,000 figure was an upper bound, assuming no changes and widely reported without nuance. Given that, at the outset, the death rate appeared to be around 1-2% this wasn’t an outrageous worst case number.
@PF
With respect, that’s not what I said. The 500k figure was headline grabbing and that’s what it did. IIRC Ferguson reported further figures, generally decreasing, but these did not get the attention the initial figure did. The model did/does have scope for change.
All models are an approximation, economic, engineering, climate and pandemic. How useful they are depends on not just how good a representation they are of the real world but the quality of the input.
Our knowledge of how this virus behaves is developing even as we speak. It was not immediately apparent that BAME communities appear to be disproportionately affected. More recently obesity seems to be emerging as a key factor and there is an ongoing argument as to the role of children and the re-opening of schools.
There can be no validated model for singular events. The question is can we develop a framework for informed decision making. Simplistically… if we do X or Y what is the effect of the death toll?
And these points stand equally in the context of previous outbreaks. SARS, MERS, Foot & Mouth
I know sweet FA about Ferguson’s model, but it’s just a model. The mistake is to treat it as “truth”. Dismissing his model as worthless because it doesn’t adhere to a set of arbitrary standards (which are themselves fluid and the subject of continual research and adaptation) is as silly as taking it as literal, complete and accurate representation of a country of 55-65 million people.
Look to the people who took the Prof’s work, made the decisions then stood back and said “but.. but.. science”.
Please note I am not seeking to defend Prof. Ferguson as such. He could be an incompetent idiot for all I know. By the same token he could be a genius, hamstrung by a lack of resources and cooperation. My point is… It’s just a f*cking model and reading too much into one, either way, makes you the fool not the author.
Fwiw someone I trust in these matters has seen the original code (they had a mate on the cleanup operation it seems) and they told me the code was basically fine. Not the kind of thing you release as a commercial product, but actually better written than a lot of the analytics-type code written by professional quants, which is a more apt corporate-world comparison. (And the world they know, so I trust their judgment.)
My personal view is perhaps a bit contradictory. As others have said above, the main issue here is likely to be in the model specification, rather than the code implementation, and it’s a shame that the model itself was not fully laid out and specified so it could be critiqued. Other modelling groups using different approaches have come up with similar results but there’s potentially a risk that there’s “herding” going on, ie without a clear target to aim for have modellers just decided their model seems more likely to be right if it yields results similar to other modellers?
On the other hand, even though it seems unlikely in this case there are really substantial issues with the codebase – I doubt anyone is going to discover an if statement got nested incorrectly and now the prediction is ten times smaller or whatever, and those people aiming to replicate the model in Python will likely succeed and get similar results (though I’m not sure what the point of this is, as without a model specification to follow separate from the released codebase all they’re doing is a language translation exercise) – why on earth has it come to this? What if there had been a problem? Should we really be making decisions with thousands of lives and hundreds of billions of pounds at stake on the basis of code that the government doesn’t own, hadn’t previously been exposed to outside critique, and hadn’t had quality assurance?
I would prefer it if pandemic modelling, since it deals with a potentially severe threat to the nation (the fatality rate from COVID could have been far worse, like H1N1 we have got lucky in some ways) was dealt with in-house by the government, perhaps at PHE, and treated as the kind of software project that gets specifications and testing. They could release their code for it publicly for outside scrutiny (in fact PHE do publicly open source quite a lot of code) and also compare their results to models developed at academic centres which might use more ‘bleeding edge’ techniques. This way if the government’s advice turned out to be completely wrong due to some daft error, at least it’s their own daft error and one which they had a chance to catch and correct – and if there were an avoidable error due to insufficient quality assurance built into the modelling programme, in principle someone can be held responsible for it. Not ideal, and I doubt the responsibility-holding would work out, but surely better than relying on results from academic code you haven’t seen that might have been churned out by grad students.
TMB,
Isn’t it that Ferguson has been so consistently wrong about all his predictions: Foot & Mouth, Mad Cow, SARS, H1N1 and on and on? His trousers should have been removed years ago to keep him out of trouble.
Something that Jim and others were pointing out right at the start of the lockdown, but is only now of interest to the MSM.
I also note the the MSM is claiming that UK discovered the blood-clogging properties virus but I remember watching a YouTube video pointing out the problem some weeks ago that I think was linked here by Rhoda Klapp.
The lack of curiosity and group think mentality of the MSM has a lot to answer for, they were the ones pushing for a full lockdown on the simple premise that other countries had, so when Ferguson published his model the Government really had nowhere to turn.
flint,
“Nor is programming to modern standards any guarantee of “correctness”. Programming & functional mistakes appear in modern code all the time, as any computer user can attest.”
That’s true, but the reason we program to certain standards is that they help to produce better code. We don’t adopt these practices as theatre, we do so because they are more likely to improve accuracy. If you have functions with void returns and global variables, the code is more likely to have errors than functions returning an object. if you use 6 levels of nesting, you’re more likely to get errors than re-organising it. If you have 400+ line functions, you get more bugs.
And when people say “modern” about most of these standards, that not really accurate. “not ancient” would be better. The last 15000 line monolith program I saw was started in 1987. The last program I saw stuffed with single line variables was from the 1970s. We considered nesting beyond 3 levels as generally bad around the mid-1980s. 400 line functions have never really been a good idea. I think it’s more accurate to say that Ferguson was an amateur doing the sloppy practices of amateurs.
Anyone for Twelve Monkeys?
‘But the real problem at the heart here is the assumption made.’
No. The real problem is reality. Screw the model. It doesn’t matter now. It was used to get some idea of what was going to happen. Yeah, it was off some. But that was months ago. Government can see now what has actually happened.
YET GOVERNMENT IS STILL ACTING BASED ON THE MODELS!
The problem is government not changing based on what has now been observed.
We begin with two problems. (i) The model is NBG – Tim’s point. (ii) Its implementation, the code, is crap – and not just crap by modern standards but crap by the standards of the 1960s.
Even if (ii) were solved by a wave of a magic wand, (i) would remain.
The stunning things to me are (a) People confused this shit with “science”, (b) Nobody seemed to notice that Ferguson’s record of predictions was abysmal, and (c) It doesn’t seem to have occurred to anyone to look at the code and use its state to diagnose the extent to which Ferguson’s work was conscientious, diligent, honest, and intelligent.
Points (b) and (c) are inexcusable – what the hell are the government’s scientists and scientific advisors doing if they are not trying to assess just those characteristics?
The scientific model is open to scientific debate. The issue is the code that ran the model is shit is because each time you run it, it gives a different answer, even if the randomised parts are given the same seed. Run it on different machines with the same hardware, it gives more variation. Run on different hardware, it gives even more variation.
Imperial College just averaged all the results of multiple runs over time to give an answer. Any model, even if based on crap scientific theory, should at least give a consistent answer when run time and again. Only then can you show any changes/improvements to the model haven’t fucked up the stuff already coded and criticise or improve the model. Clearly, this basic tenet of coding wasn’t applied.
Flint
“On and off.” With respect, that’s not what I said.
Fair point, my mistake. But others, who have reviewed the model, have had laid such criticisms at it, and which is fundamental to any subsequent analysis or discussion.
The mistake is to treat it as “truth”. … Look to the people who took the Prof’s work, made the decisions then stood back and said “but.. but.. science”.
Agreed. And which was the point I was then making (“should have been prepared to stand up, etc”).
@Mr Ecks: ’ The Omega Man–1971 with Charlton Heston–the best IMO.’
Certainly is. And the one with the best score too.
The always reliable Ron Grainer wrote the musical score–very catchy “action”-type music.Esp the bit near the start where the albino photo-phobic pseudo-zombies in sunglasses (at night) and monks robes try to set fire to Heston’s car. Very good.
https://www.youtube.com/watch?v=dk7bUsc854I
Now some action like that would have made the LD worthwhile–esp if Theo and other members of BlueLabour were the photophobes.
Mr Womby–Yes –I clean forgot 12 Monkeys. Though Blojo seems to have them in the Cabinet so no worries.
Not to take over the thread but this is interesting:
https://www.youtube.com/watch?v=wWD8gW9cbYQ
Never was “a little knowledge is a dangerous thing” so apposite
Regardless of the details, we need to make sure that we are trusting science, not scientists.
If the work is not fully published its not science, it’s one man’s opinion. And if it could have been published 13 years ago but wasn’t that’s very fishy indeed. If a man is hiding something he generally has something to hide.
Plus if the public paid for the work they have a right to see it, or their money back.
I suspect that nothing done by government would have affected the outcome, other perhaps than diverting healthcare resources to the new disease. The social distancing etc. was driven by fear, not government decree. The closing of schools was forced by fear. I suspect that had the medical profession been privately run it would have naturally diverted its efforts to the new disease.
flint said:
“IIRC the Ferguson model predicted a top bound of 500,000 deaths if nothing was done. To validate the model the UK would have had to continue as if nothing was happening – not just no lockdown but full attendance at work, concerts, sports fixtures etc. How many deaths from CV19 that would have occurred without the lockdown can only be a matter of conjecture.”
Actually the important figure from Ferguson was not the 500,000 deaths if nothing was done, but that we’d still have 250,000 deaths under voluntary action – more hand washing, working from home if practical, encouraging self-isolation for the most vulnerable, etc.
That’s the figure that says whether we need to go further and have compulsory lock-down and business closures.
And that’s the figure that, as Tim is pointing out, seems to be based on completely false assumptions about human behaviour and how much people would follow voluntary guidelines.
Nothing to do with whether Ferguson’s code is any good, or whether his epidemiological assumptions are right (although those are also important points), but that he doesn’t understand human nature.
He could even be right about the 500,000 with no action (although his past record doesn’t give any confidence) and the 30,000 with full lockdown, but still be wrong about the middle figure – how many of those lives will be saved by voluntary action.
Someone said that he’s a known socialist – I wonder if that has warped his view of human nature?
@RichardT
Would probably have been better to run the model a couple of times with different assumptions: X% of people wash their hands regularly, Y% of people will obey social distancing and so on. Then leave it for the behavioural scientists and messaging people to argue about which scenario it’s practically possibe to attain.
The other thing is – and this isn’t anyone’s fault as such, more that there’s no way you can write a model that captures everything that goes on in society – you can’t just naively say “the best results come from implementing the most extreme measures for the longest time, so let’s do just that”, because the model doesn’t take into account all the additional harms caused by those measures or their long-term sustainability, psychologically, economically and so on.
“(b) Nobody seemed to notice that Ferguson’s record of predictions was abysmal, and (c) It doesn’t seem to have occurred to anyone to look at the code and use its state to diagnose the extent to which Ferguson’s work was conscientious, diligent, honest, and intelligent.”
This, I contend, was Alice Gast’s job. She should be fired as president of Imperial College.
Another thing about the ‘model.’ Ferguson looked at the output and ratified it. He, world renowned epidemiologist, look at the numbers and decided they were reasonable – in line with what he thought. At that point, it doesn’t really matter where the numbers came from.
The average of crap is crap.
Making multiple runs of Ferguson’s model, the taking the average, is junk.
Same as IPCC climate models: they average 25 obviously wrong climate models, and declare the result good.
dearieme,
“The stunning things to me are (a) People confused this shit with “science”, (b) Nobody seemed to notice that Ferguson’s record of predictions was abysmal, and (c) It doesn’t seem to have occurred to anyone to look at the code and use its state to diagnose the extent to which Ferguson’s work was conscientious, diligent, honest, and intelligent.”
Who is going to say to do that? A former economist at the BofE (and let’s not talk about predictions there), or his boss, a former journalist? And that’s what nearly all politicians are like. They have no experience in management. None of the thinking of using known things that work, rapidly monitoring to confirm models.
The only reason I voted for BoJo was that the other guy was even worse.
Yes. The lesson is that computer modelling of an epidemic disease requires at least three different types of skill/ knowledge: of the disease, of epidemics, of modelling, and of programming. Doing expert level work requires experts in all those areas. It could still be wildly wrong but at least it will be wrong for good reasons. What does not work is programming proficiency of the sort that says ‘I cranked out a few 000 lines of Fortran for my PhD thesis and got it to compile so I know programming.’ In fact, thinking Gell- Mann paradox, if the programming is crap, how do we know the other areas are not crap too? The Romans used to say ‘falsus in uno, falsus in omnibus’ does that apply here? Grrr
I and many others must have PII to cover me in the event of a cock-up. It’s a few million quid in my case.
Christ alive. Imagine making a cock-up like this? Imagine the insurance premiums …
I used to think Two Jags’ NHS database was a model of the sheer unaccounted-for uselessness of such people. But this is in a class of its own – as is the political decision-making deriving from it.
Ultimately, the Cabinet did not have to follow what Ferguson said.
@Lud
“Ultimately, the Cabinet did not have to follow what Ferguson said.”
To be fair they were being given very similar results by other modellers who had used different methods (though as I said before, herding is a risk with all collections of predictions, particularly of one-off events where you can’t defensively check your prediction against previous occurrences so checking against other people’s predictions becomes your main sanity-check – alternatively you might pick a somewhat mismatched past event as your benchmark, eg Spanish Flu, because you think that’s the best you can do, and that can also skew results). Being given advice from multiple sources as to how bad things might get makes it very hard for government to stand up and say no, we aren’t going to do that. It would be very publicly saying “I’m prepared to risk hundreds of thousands of deaths because I as a non-scientist believe the claims of the scientists, however esteemed, are bunkum”. I don’t think that’s politically feasible.
If there had been a bust-up at SAGE and strong voices had said “treat the models with caution, our own statistical estimates are that the worst-case is X and if you take particular sets of measures you can achieve Y or Z” then you could go with that instead. If it is full-on “politicians versus THE SCIENCE” right across the media then you’ve lost. Note that Sweden had some top public science officials who were more sceptical about modelling and lockdown and more optimistic about behaviour change – whether they’re right or wrong, it’s their existence and prominence which made their government’s chosen path possible.
MBE
No dissent – SAGE / others etc
Were they mostly “on the payroll” (in some form or other, be it public grant funding, whatever)?
@PF
Science funding is primarily from the government/EU though for pandemics a lot of specialists also get grants from the Gates Foundation and various international bodies..
According to those who attend, SAGE is supposedly quite heated at times and that seems credible given the ego and reputation jostling that must be involved. But I think there must be political and practical incentives to settle on “the consensus” or “The Science” – for one thing inability to feed the government useful actionable advice (rather than “on the one hand, and on the other, and then on the third hand” as economists tend to do) wouldn’t reflect well on the scientists personally and academia in general. For another, mixed messages about conflicting opinions would risk undermining public confidence in and adherence to any resultant plan. So it isn’t very surprising that a lot of cracks get papered over.
There’s also the issue that in times of great uncertainty there can be an evidence vacuum and that can leave you more reliant than you would like to be on evidence you have doubts about and are currently unable to verify. At least the modellers are sticking some projections on the line and saying “we have grounds to believe this is what will happen under the following scenarios”. Even if you don’t find their grounds convincing, do you have anything stronger to go on? Particularly if several modelling teams are pointing in roughly the same direction?
the briefest summary is that it was all crap from the get go.
When the model was run there was a dearth of data. Ro was estimated from Chinese lies, CFR was unknown, etc.
The model could have been a model of how modelling should be done and it would still have output garbage, because any assumptions necessary as parameters were pulled out of thin air or Ferguson’s arse.
MBE,
I can forgive the politicians for trusting the model, but when the results come in for the control group, you should be re-assessing the value of the model. Sweden is so far out as to be render it worthless.
And it’s not just Sweden. Georgia and South Dakota haven’t behaved as the modellers said.
https://www.thedailybeast.com/ending-coronavirus-lockdowns-in-mississippi-georgia-and-florida-could-doom-thousands?ref=home
It’s barely above the lowest DOOM figures from Harvard/MIT figures and well within the figures they predicted if lockdown was done.
I wish watching people rediscover an essential truth was more entertaining.
Trying to model a poorly understood, rapidly evolving phenomenon is always a disaster waiting to happen. Having watched very bright, competent economists attempt to develop predictive models for economic activity since my undergraduate days (40+ years ago), and watching each of them crash and burn doing so, I am still baffled that supposedly bright people think of modelling as anything other than a supposed expert’s wild guess.
To me, the only thing of interest here is the indication that the coding of the model suggests that Ferguson and his team are far less competent than assumed. Again, I am baffled. Anyone who went to college ran into at least one professor (and usually more than one) who, upon closer examination (for example, in the classroom) turned out to be something of the village idiot. One thing I did learn at college was that having a Ph.D. and being tagged as an “expert” did not guarantee even basic competence.
I suspect Ferguson will be found to be a village idiot who had enough credentials to shut down any sober evaluation of his actual level of competence.
“Ultimately, the Cabinet did not have to follow what Ferguson said.”
Of course they did! Politicians must accept what the state epidemiologist tells them.
Initially.
MBE: to feed the government useful actionable advice
It would be excellent if the advice were actionable in the original sense of that word. In a couple of months we will learn that everybody concerned performed amazingly well against all the odds and but for that everything would be so much worse. A shame about the economy, of course, but it was a price worth paying.
Hmm… I saw at least one comment in this horde that assumes that if you run the type of model used for this several times you get ( more or less) the same results.
It. Does. Not.
This type of model is in essence a “walking ant” model **: you cannot predict where the ant will end up, even under the same set of conditions. You can run it [x] times and see if there’s a pattern to the outcomes though.
Especially when you, like in the particular application of epidemiology, let 100’s if not 1000’s of ants run around and bump into each other.
Then you vary some of the conditions and do it all over again.
The thing is… These models give you probability patterns, not numbers ***. How to get from those patterns to numbers is a Dark Art in and of itself. And as always, a probability is only useful if you also know the deviation…
The real trick is not in writing the model. The base model is used in many fields. Epidemiology just uses different variables.
The trick is also not in interpreting the model. People working in the fields those models are used in are supposed to be trained in using and interpreting them. YMMV as to how good they are in it, but still.
The real trick is in presenting the results in a way that can be understood by people who have had no training in the pitfalls of statistics, and probability, and proper mathematics. Like Liberal Arts and Grievance Studies graduates (aka. Journalists and Politicians).
And that’s where Ferguson went wrong ( again) … Where other national science institutes in Europe brought the Bad News leading with “average probability, with a range of..” , he led with “Worst Case Scenario”. (again)
With, of course, predictable results.
It’s not the way he codes, or the assumptions he made doing the modelling. They aren’t that far Out There, and the results in and of themself aren’t that odd or extreme.
It’s the way he presented them that makes him Bloody Stupid Johnson. Especially considering the ..rather sensationalist and partisan.. nature of the British Press, and the Kneejerking politicians tend to do being well aware of that fact.
And he either ignored that little fact, or he’s so dense that he hasn’t picked up on it in all the years he’s been around. In which case he should have been assigned a Handler anyway.
** Required reading, really: The Science of Discworld ( Stewart, Cohen, Pratchett ) . It has a whole section dedicated to exactly this kind of mathematical modelling, and its applications, with references for further reading.
Best primer ever. And it features Rincewind as the Egregious Professor of Cruel and Unusual Geography as a bonus.
*** Those pretty, psychedelic Mandelbrot images is where they assign a colour to a (compound) probability.
If we are to ‘follow the science’ then why the fuss over schools, the science says closing schools has minimal impact
I recall that right at the beginning, when Ferguson’s code was cited there was a pretty noisy group from Cambridge who were warning that his estimates were… let’s just say, wildly at variance from theirs. Tough spot for a government, though – either way, you will be crucified by hindsight, won’t you?
dcardno said:
“I recall that right at the beginning, when Ferguson’s code was cited there was a pretty noisy group from Cambridge who were warning that his estimates were… let’s just say, wildly at variance from theirs”
The Oxford lot were pretty openly saying he was plucking his numbers out of his arse. If the government had any bottle they’d have ramped that up a bit, presented the science as confused, and pointed out that we couldn’t stay locked down for 18 months or whatever Ferguson was demanding.
RichardT – were they from Oxford? I misremembered that, but yes, they were very critical of Ferguson’s work, and his track record. The problem for a government is that if you had 2,000 grannies / grandpas dead no one would acknowledge that at least the economy had survived, so that their children and grandchildren could continue to live in comfort. On the other hand, if the government “does something” it’s only those who have read Bastiat who would complain about the opportunity cost.
One of the significant problems with Ferguson’s code was that it used multi-threading to improve performance, without any real understanding of how to code for such an environment. As a result, it could (and did) produce different results when run multiple times, even starting with the same initial conditions. The team were well aware of this, and just ran it several times and took an average (I assume they also checked that the results only differed by a few %, not a factor of 10).
This isn’t unexpected – I couldn’t write C code that I’d be confident of running multi-threaded – but at least I know that I can’t do it (and I know enough to find someone who can). And in any case, it doesn’t really matter. Ferguson could have written perfect code to implement his perfect understanding of how such diseases spread, and it would still need to be built on base data that was almost completely unknown – number of cases, reproduction rate, mortality …
(skipping to the end)
As Rue says, it crapness of the code is irrelevant, what’s the model it’s implementing? We still haven’t see sight of the model itself, just a horrendous implementation of it.
I wrote automata programs for fun & profit in the ’80s. If the model is “bug’s food is the eight squares containing grass around the bug” but the code does b+d+f+h then the implementation is flawed, it’s only added four squares. If the model is “food is the 12 squares around the bug”, then no amount of correct code will give the right result, as the model is flawed – it’s impossible to have 12 squares surrounding a square.
I modeled power cost forecasting in the 1980s. It was 100% accurate.
All behaviors were known and programmed. Forecasting pandemics and climate is impossible, as all is not known.
They start with about 20 parameters, which are known to an accuracy of typically +/-50%.
Then they add some to others, and subtract some from others.
And they multiply some together, and divide some by others.
And sometimes they’ll raise one to the power of another.
Then they’ll do it all a few hundred times and take the average, because they never get the same output twice, for reasons that are only approximately understood.
But the output number is always >0 and always less than infinity- which is exactly what you’d expect.
So why are people complaining? 😛
Because outbreaks follow a fairly standard pattern, the only thing you are actually trying to predict is the peak. Once you have that you have the curve. So the question is, can your model predict the peak before you get there? My guess that it is highly unlikely you will ever have good enough data in time.
No, the crapness of the code is very relevant. What Ferguson is using to get his doom-laden predictions seems to be not much more than a random number generator, and a crap one at that.
Fwiw I think Timmy is wrong about “no commercial organisation would use, release or base a decision upon it”. There’s a difference between the “release” bit where you want properly engineered code that stands on its own two feet and the “use” and “base a decision on” bit, where you trust that the code’s authors can use it – possibly with some judicious editing, scripting etc – to churn out the relevant results. In my consulting I often get to see the analytics and modelling code firms make business decisions on and it is often very shoddy indeed. Still, who in an ideal world wants decisions about thousands of lives, hundreds of billions of pounds and unprecedented restrictions of liberty to be based on code the decision maker has no access to and has not had QA? Even if you think the code probably works, in so far as this stuff can be expected to work, that’s clearly not ideal.
@RLJ Predicting peaks seems to be very tough, hence some of the less glorious results in the history of epidemic forecasting. It’s not just Ferguson and Imperial, when an LSHTM team worked on the West African Ebola outbreak they also ended up with big overestimates and their real-time forecasts were only accurate to about one or two weeks ahead. Which suggests the problems are pretty inherent. https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1006785
BiW
I don’t believe Ferguson was acting in good faith. He wanted a scary number for political reasons and that is what his model provided.
MyBurningEars
Cheers.
Indeed. And I mused out loud the other day about the effects of a private prosecution of Ferguson and Imperial for manslaughter, corporate or otherwise.
The experts who write the standards defining how safety-critical software should be developed, meet in public here. There’s some good stuff there, if you’re up to following the discussion. They agree that software like Ferguson’s should be considered safety critical – errors in it are capable of killing at least as many people as faulty software in an airliner or nuclear reactor.
@Chris Miller
Thanks, very relevant!
Paywalled in Full
“Neil Ferguson’s Imperial model could be the most devastating software mistake of all time
David Richards and Konstantin Boudnik, 16 May 2020 • 1:22pm
In the history of expensive software mistakes, Mariner 1 was probably the most notorious. The unmanned spacecraft was destroyed seconds after launch from Cape Canaveral in 1962 when it veered dangerously off-course due to a line of dodgy code.
But nobody died and the only hits were to Nasa’s budget and pride. Imperial College’s modelling of non-pharmaceutical interventions for Covid-19 which helped persuade the UK and other countries to bring in draconian lockdowns will supersede the failed Venus space probe and could go down in history as the most devastating software mistake of all time, in terms of economic costs and lives lost.
Since publication of Imperial’s microsimulation model, those of us with a professional and personal interest in software development have studied the code on which policymakers based their fateful decision to mothball our multi-trillion pound economy and plunge millions of people into poverty and hardship. And we were profoundly disturbed at what we discovered. The model appears to be totally unreliable and you wouldn’t stake your life on it.
First though, a few words on our credentials. I am David Richards, founder and chief executive of WANdisco, a global leader in Big Data software that is jointly headquartered in Silicon Valley and Sheffield. My co-author is Dr Konstantin ‘Cos’ Boudnik, vice-president of architecture at WANdisco, author of 17 US patents in distributed computing and a veteran developer of the Apache Hadoop framework that allows computers to solve problems using vast amounts of data.
Imperial’s model appears to be based on a programming language called Fortran, which was old news 20 years ago and, guess what, was the code used for Mariner 1. This outdated language contains inherent problems with its grammar and the way it assigns values, which can give way to multiple design flaws and numerical inaccuracies. One file alone in the Imperial model contained 15,000 lines of code.
Try unravelling that tangled, buggy mess, which looks more like a bowl of angel hair pasta than a finely tuned piece of programming. Industry best practice would have 500 separate files instead. In our commercial reality, we would fire anyone for developing code like this and any business that relied on it to produce software for sale would likely go bust.
The approach ignores widely accepted computer science principles known as “separation of concerns”, which date back to the early 70s and are essential to the design and architecture of successful software systems. The principles guard against what developers call CACE: Changing Anything Changes Everything.
Without this separation, it is impossible to carry out rigorous testing of individual parts to ensure full working order of the whole. Testing allows for guarantees. It is what you do on a conveyer belt in a car factory. Each and every component is tested for integrity in order to pass strict quality controls.
Only then is the car deemed safe to go on the road. As a result, Imperial’s model is vulnerable to producing wildly different and conflicting outputs based on the same initial set of parameters. Run it on different computers and you would likely get different results. In other words, it is non-deterministic.
As such, it is fundamentally unreliable. It screams the question as to why our Government did not get a second opinion before swallowing Imperial’s prescription.
Ultimately, this is a computer science problem and where are the computer scientists in the room? Our leaders did not have the grounding in computer science to challenge the ideas and so were susceptible to the academics. I suspect the Government saw what was happening in Italy with its overwhelmed hospitals and panicked.
It chose a blunt instrument instead of a scalpel and now there is going to be a huge strain on society. Defenders of the Imperial model argue that because the problem – a global pandemic – is dynamic, then the solution should share the same stochastic, non-deterministic quality.
We disagree. Models must be capable of passing the basic scientific test of producing the same results given the same initial set of parameters. Otherwise, there is simply no way of knowing whether they will be reliable.
Indeed, many global industries successfully use deterministic models that factor in randomness. No surgeon would put a pacemaker into a cardiac patient knowing it was based on an arguably unpredictable approach for fear of jeopardising the Hippocratic oath. Why on earth would the Government place its trust in the same when the entire wellbeing of our nation is at stake?
David Richards, founder and chief executive of WANdisco and Dr Konstantin Boudnik is the company’s vice-president of architecture
https://www.telegraph.co.uk/technology/2020/05/16/neil-fergusons-imperial-model-could-devastating-software-mistake/ ”
– Remember, this is from ICL “tidied and cleaned up code” released. ICL & Ferguson still refusing to release orignal code, despite it being paid for by Taxpayers via Gov’t MRC. When will BoJo send cops in to sequester?
– ICL Announce £5Million funding from Huawei – great timing there
So now we have people defending models which are shit by suggesting the quality of the code does not affect the results of the model.
I despair that anyone could be so arrogant & stupid.
Oh and the reason the original Ferguson code has not been released is that it is currently being rewritten by a group of Physics PostDocs in Edinburgh.
“Imperial’s model appears to be based on a programming language called Fortran, which was old news 20 years ago and, guess what, was the code used for Mariner 1. This outdated language contains inherent problems with its grammar and the way it assigns values, which can give way to multiple design flaws and numerical inaccuracies.”
Elitist bullshit. I ran a factory with Fortran. Fortran 4+, actually. Worked great! Many great things were done with Fortran for many years.
I’m not convinced Ferguson got his numbers from his ‘model.’ I think it quite possible that he made them up, then claimed they came from a model to give them credence.
@Gamecock
Yes, I wrote complex actuarial calculations in Fortran77 back in the early 80s. Many people still use it for complicated arithmetic because it’s generally easier to understand the equations than writing equivalent code in C or Java. You can write good or bad code in any computer language.
The various takedowns of Ferguson’s code that I’ve read have suggested that it was written in C. But maybe it was in Fortran originally and then run through some type of translator.
@Gamecock
Yep, Fortan, like Cobal, Basic, Ada, Pascal is OK, it’s the coding that matters – GIGO
I found ‘different numbers in, same result out’ most worrying. Suggests model designed to always predict catastrophy, same as Ferguson’s BSE, SARS etc models
@Chris
Agree
Suggests more ICL obfuscation than “cleaning up and tidying released code”
Is ICL comitting Fraud?