Warning IT chaos could take days to fix

Cyber security experts are warning about the knock-on effects of a global cyber security outage that caused widespread disruption on Friday.

Although there is now a software fix for the issue, the manual process required will take a huge amount of work to get computers back up and running, they said.

And it could take days for big organisations to get back to normal.

The outage affected firms including GP surgeries, pharmacies and airports in the UK.

The issue was caused when an update from cyber security CrowdStrike caused Microsoft systems to "blue screen" and crash.

The problem piece of software was sent out automatically to the firm's customers overnight which is why so many were affected when they came into work this morning.

It meant their computers couldn’t be restarted.

The fix will unfortunately not be automatic, but what the industry calls a "fingers on keyboards" solution.

Researcher Kevin Beaumont said: “As systems no longer start, impacted systems will need to be started in ‘Safe Mode’, to remove the faulty update.

"This is incredibly time consuming and will take organisations days to do at scale."

Technical staff will need to go and reboot each and every computer affected, which could be a monumental task.

Crowdstrike is one of the biggest and most trusted brands in cyber security.

It has about 24,000 customers around the world and protects potentially hundreds of thousands of computers.

One struggling IT manager said the process to get computers back up and running is quick once an IT person is at the machine, but the problem is getting them to the machines.

The person, who wants to remain anonymous, is responsible for 4,000 computers in an education company and says they are working flat out.

“We have managed to fix all of our servers using the command prompt as a workaround, but for many of our PCs, not easy to do manually as we are spread out across five sites. Any PCs that are left switched on overnight are affected and we're rebuilding them,” he said.

IT experts say this manual process will be particularly hard in large organisations with thousands of computers that are potentially under resourced in IT.

Small and medium businesses without dedicated IT teams or which outsource their IT issues might also struggle.

We are seeing the larger, more resourced companies, like American Airlines, fix the problems rapidly.

Interestingly it looks like many in the US might be less affected as computers that are potentially not yet switched on can be started up to download the corrected software instead of the bad version. But that might still involve a level of manual operation.

Mr Beaumont said that one of the world’s "highest impact IT incidents" was "caused by a cybersecurity vendor.”

Ironically if a customer was affected by this it was because they followed all the usual advice that’s issued by cyber security experts – install the security updates when you receive them.

While some security companies in the past have accidentally send out a dodgy software update, we’ve never seen one at this scale and this damaging.

While this incident has caused widespread disruption, the WannaCry cyber attack in May 2017 was potentially worse.

That was a malicious cyber attack that affected an old version of Microsoft Windows and spread automatically and uncontrollably to any computer that had that old and unprotected Windows software.

It affected an estimated 300,000 computers in 150 different countries.

Famously the NHS was badly hit by it with huge disruption for days, affecting doctors, surgeries and hospitals around the country.

In that case it was an attack through to be carried out by North Korea which got out of hand.

The NotPetya attack a month after that was eerily similar in method and damage.

In contrast, the outages on Friday are likely to be a mistake and not an attack.