Microsoft has blamed EU rules for enabling a faulty security update to cause the world’s biggest IT outage.
The software giant said a 2009 agreement with the European Commission meant it was unable to make security changes that would have blocked the CrowdStrike update that triggered widespread travel and healthcare chaos on Friday.
CrowdStrike’s Falcon system, designed to prevent cyber attacks, has privileged access to a key part of a computer known as the kernel.
I think I’ve this right. M can’t ban access to the kernel – like Apple does – because of the agreement. Therefore folk have access to the kernel.
It’s thanks to EU……
The kernel of an operating system is generally that part that controls the hardware, and manages access to said hardware. That access can be laxer (ie more privileged) or tighter. There’s absolutely no reason why a competent operating system vendor can’t block a third-party component from doing something stupid.
(In some systems, the part that the user interacts with is called the shell.)
Yeah but the change wasn’t in the actual kernel. It was a. .sys file that is called when the system reboots that was faulty and stopped the operating system from completing the startup procedure. I’ve had similar things happen to me on Unix machines in the distant past.
Everything about this farrago smacks of massive incompetence.
CrowdStrike sending out untested parches to all customers at the same time is bad enough, but the number of large corporates that allow these untestec patches to update automatically is just utterly reprehensible.
MS is being disingenuous about this matter. Windoze should be robust enough to ignore rogue modules that don’t load and not abend the entire boot sequence.
Totally agree with BiW and Otto. I’ve done a lot of kernel programming over the years and the first rule is “don’t screw up”. Crowdstrike screwed up. The second rule is “test thoroughly”. From what I’ve seen elsewhere if Crowdstrike did test it was perfunctory. As for MS, there are ways to give kernel access that mitigate potential damage but MS just opened the gates and let any old crap in.
Ottokring,
“Everything about this farrago smacks of massive incompetence.
CrowdStrike sending out untested parches to all customers at the same time is bad enough, but the number of large corporates that allow these untestec patches to update automatically is just utterly reprehensible.”
The thing is, Crowdstrike view this as a data file not a software patch, so for companies that turned off automatic updating of Crowdstrike, they still got this update.
And it wasn’t even untested. It was just a file of zero bytes. Someone sent out the wrong file somehow.
“MS is being disingenuous about this matter. Windoze should be robust enough to ignore rogue modules that don’t load and not abend the entire boot sequence.”
The driver powering it was fine, it was the data file that was sent out, Microsoft does have protections around device drivers or duff DLLs, but there’s not much that Microsoft can do much about a rogue data file that the driver uses, that contains some sort of Crowdstrike cooked up p-code that the driver then reads executes in a custom way. There’s a good technical explanation to me in this video.
https://www.youtube.com/watch?v=wAzEJxOo1ts
The EU has very little to do with what caused the BSOD’s, and Micro$oft knows this well enough.
All the EU requires is that M$ ( and for that matter any OS supplier..) does not deny access to the kernel for legitimate third-party software.
Any security software has to interact with, and partially control, the kernel to work properly. You simply need that level of access to monitor stuff effectively, even more so nowadays.
Which also means that updating this kind of software is potentially very risky, and it very much isn’t the first time that a security suite flattens entire networks due to a botched update.
Especially since those updates are automatically marked as “Trusted” and “High Priority” as far as the kernel is concerned. Because they need to be.
Which is why every security software company has learned to check and doublecheck before pushing out stuff. Trust is everything in security. Botched updates very much evaporate that trust.
Crowdstrike ( what’s in a name… ) botched up their review process, Windoze did exactly what it was supposed to do, and all hell broke loose because this particular suite happens to be used extensively in certain branches of industry..
you’d have had the same problems if…say… AVG had botched up like that ( and they did in the past….) except that that would mostly have hit consumer PC’s..
But the EU? The worst they did was telling OS developers that they do have to give access to their OS.
To prevent lock-in by those oh-so-well-behaved OS developers who would never, ever dream of such a thing….
Not really. It’s only a matter of time until M$ releases a patch that bricks every windows system on the planet.
The effects of that will make illegal lockdown look like a fun party time.
When you have a founder who’s not very bright, does that colour the whole organisation and its philosophy?
Bill is a good imitator of Mr Turpin, but aside from that is a few somethings short of a whole thing…
WB
Which makes it even worse. If a driver has null or garbage data ( as opposed to wrong ) then the daemon should simply close down and log it as an ‘event’.
Microsoft has released separate products for EU/non-EU markets in the past. In the mid-2000s, the EU demanded that Microsoft remove their default Windows Media Player; then a couple of years later they demanded a browser choice popup. This year, the EU demanded a version of Windows with fewer ties to Microsoft services (Bing, OneDrive, etc).
The precedent is thus set. Microsoft could have created a separate version of Windows for the EU, if they felt strongly about this issue. That they didn’t implies that they didn’t feel it was worth it.
To my surprise, I find myself arguing MS’s corner!
You can either provide a locked-down system, where no one can supply applications or changes, without your approval and say so (and paying a toll for the privilege), or you can provide a free for all.
The former is the Apple model, and the EU take legal action because it is anti-competitive.
The latter is Microsoft Windows, and the EU complain and take legal action…
You cannot have both closed systems allowing anyone’s access, and the guarantee of stability of a walled, anticompetive garden.
The best option is neither: open source, where anyone has access, but also anyone can earn brownie points by showing and fixing the vulnerabilities. Not perfect, but better than the alternatives.
Think of it as Evolution in Action.
M$ doesn’t even think it worth issuing an English version of Windows, so we have to suffer Websterian Dyslexia – ‘favorites’ is a particularly nasty example. I can’t fathom that, as they do a Greek version for 11 million Greeks, complete with a different alphabet, and versions for people who write from right to left in squiggles in their diatribes against the Yids.
Personally, I’m happy to believe that the place is run so that its founder can gorge on McDonalds and pontificate, but I really do believe than an excess consumption of cannabinoids and anal intercourse have a lot to do with their corporate ‘culture’.
Every computing cock-up is accompanied by screeches from IT people saying it wasn’t us, guv, it was the customer. I doubt it. In my experience many computer people are modestly clever people who imagine they are terribly clever. And anyway “the customer” is often, for practical purposes, the IT people at the customer company.
Still, whomsoever is to blame, I am always fascinated by the extent to which these bozos don’t test stuff properly.
WB has it right, it was a crap data file, not a software update. The primary fault is with CrowdStrike who don’t have sufficient (any?) validation for the new data file. Decent software would check the new file, and if crap revert to the previous one.
It’s not easy for MS to guard against what CS did. The CS driver software is flagged as ‘must load with the OS kernel’. If it screws during load, the OS load process is buggered.
dearieme,
Testing is a trade off. In most cases, cost. You want things tested to the accuracy of medical software, you’re going to pay medical software prices. Most companies don’t want to pay for that.
In the case of Cloudstrike I get the impression that speed was a factor. They turn around changes for new threats in hours.
To be fair to Microsoft — which I absolutely hate doing, it’s not reciprocated — there’s a massive difference between not installing some additional bloat on versions of Windows bound for particular regions and a fundamentally different way of handling stuff in the kernel.
The more pressing question is why any organisation is using Windows for anything remotely important. Sure, for people who do doodles in Word or pretend to be clever in Excel it’s fine, and if you want to write/deploy bad software fast then it’s brilliant. But for stuff that will stop the company’s basic functioning if it goes down? For kiosks that you want to be as light-weight and robust as possible? Whoever made a decision to run any of that on Windows is in urgent need of a P45.
Well I can’t help noticing that Microsoft issued a Windows 10 update which wouldn’t install because it required a larger recovery partition than that which the standard Windows installation created on one’s hard drive.
Wonderful! The word from Microsoft was that they were working on a solution to the snake eating its own tail and a “fix” would be issued in due course.
The next word some months later was that it was time to upgrade to W11 and that there would be no “fix” after all. (I found a youtube video to describe how to resize the partition ).
Why bore you with this? Well although I can read the exchanges between computer folk which are delightful and mysterious like the dawn chorus, this technology is entirely man made and thus within the scope of man’s comprehension.
What beats me is that people who manage to make a pig’s ear of the technology feel that they have the insights as well as the means to interfere massively with human lives or the planetary eco-system so I would happily unleash a galaxy of Captain Potatoes at maximum malevolence upon Bill Gates. Obvs they would have to be programmed to self-destruct once the job was done.
No flaw in that idea, is there?
Security software isn’t the only kind of application (in this case understandably) demanding kernel level access. Two other things have become much more common in the last decade:
1) Anti-cheat software to stop people getting assistance on computer games (especially online multiplayer),
2) Exam proctoring software to stop people cheating on online exams, particularly at university or for professional exams (also does creepy stuff with your webcam).
I’m waiting for the first really big scandal involving these. But would suggest avoiding if you can help it, even if it means travelling to sit distance learning exams in person.
I’ve been out of the IT security game for over a decade (and CloudStrike haven’t been around as long as that), but folk I know who look at security ‘solutions’ for major corporates tell me their examination of ClownStrike left them mightily unimpressed. Lots of buzzwords (AI! Cloud!!!) to impress the C-suites, but very little substance. And no concept of testing at all, it would appear – “move fast and break things” may work in some environments, but mission-critical security software ain’t one of them.
“to stop people cheating on online exams”: the answer there is to stop holding important exams online. Exams are one of those things that are best done in an entirely traditional manner: you turn up, hand over any electronic devices, sit down at an assigned desk. You write your name on the cover page of your script. When the invigilator says “You may begin” you read the exam paper, find the easiest question, and knock it off.
Quite agree. Even in person they have problems with ringers sitting exams. Long time ago now but I went to sit a large distance learning exam in London and behind the invigilators’ desk there were posters of photos of well-known ringers (all Chinese fwiw). We did have to bring ID, in this case my passport, but the invigilators’ check was only cursory. One of the slight advantages of the online exam is the facial recognition check done by the webcam is more accurate than a human can manage (a well-trained facial recognition model substantially outperforms humans these days) but really, creepy as it may be, the in-person exams should be introducing face scanners.
M$ doesn’t even think it worth issuing an English version of Windows,
Any sensible computer system would have all the human text in a seperate file so that a human can copy/edit it to create their own version or a version for a new locale. All(most all) the GUI programs I’ve written have all the text seperate from the program so anybody can either correct my spelling or create new versions. It’s not even OS-specific, I’ve done this on RISC OS, Windows and Linux.
When I did an online exam a couple of years ago I was required to use two computers. One had a live web conference session running watching me, and the other was the one I did the exam on. I had to pan the camera around my room to show there was nobody coaching me and all reference materials were out of sight.
Matt,
“The more pressing question is why any organisation is using Windows for anything remotely important. Sure, for people who do doodles in Word or pretend to be clever in Excel it’s fine, and if you want to write/deploy bad software fast then it’s brilliant. But for stuff that will stop the company’s basic functioning if it goes down? For kiosks that you want to be as light-weight and robust as possible? Whoever made a decision to run any of that on Windows is in urgent need of a P45.”
Windows runs fine. It’s robust. Websites like Stackoverflow and some huge e-commerce sites are running on IIS on Windows servers. Why they use it over Linux is a complex question. Partly historical, partly about tools. Personally speaking, building software with Visual Studio and C# is my favourite way to build software.
@witchie forget Greek there’s a Welsh version
@WB
Agree with you on Visual Studio — I use VSCode (for Mac) in preference to Xcode.
However, with regard to using Windows in important infrastructure… the proof of the pudding is in the eating. If Stackoverflow goes down for a few hours, a few people will whinge, but it won’t open the company to huge losses and liability in the same way that an airline or payment provider being unable to operate would. The whole issue with Windows is that you have to blindly trust your vendors, you can’t really have any in-house expertise.
I work for a smallish vendor of virtual appliances. Our platform team includes several people whose hobby is to manage a well-known Linux distribution. They’re paid well, but not exorbitantly so — any company for which 24/7/365 uptime of IT infrastructure is essential could afford to keep some folks like that on payroll to ensure that a Linux/BSD-based platform that the company builds everything else on top of will be secure, can have updates rolled out in a manner that doesn’t affect operations, and doesn’t break.
The decision to use Windows for essential devices will most likely have been a matter of time-to-market and cost. It’s a valid choice in some cases, but not if you’re going to lose a billion dollars in the event of an outage.
Microsoft’s position is that they are not competent to produce an operating system which protects itself against sufficient attacks, so they should have a monopoly on producing software to make up for their incompetence, or at least set the standards and act as gatekeeper. Quite obviously, they should not have such a monopoly even if they were competent, but the quality of Windows makes it even worse.
“like Apple does”
except Apple doesn’t. You can buy add-on security software for Macs (e.g. antivirus).