Aces High Bulletin Board

General Forums => The O' Club => Topic started by: Eagler on July 19, 2024, 03:59:06 PM

Title: Crowdstrike
Post by: Eagler on July 19, 2024, 03:59:06 PM
I have done software qa testing in the past..

I wouldn't want to be on the team that passed that update

Eagler
Title: Re: Crowdstrike
Post by: Busher on July 19, 2024, 04:30:23 PM
Their flight home might be delayed :rolleyes:
Title: Re: Crowdstrike
Post by: CptTrips on July 19, 2024, 05:37:24 PM
I have done software qa testing in the past..

I wouldn't want to be on the team that passed that update

Eagler

 I wouldn't want to be the coder.

Like the guy that cost that Mars probe because he forgot to convert from meters to feet.  DOh!!!!!!  10 years of how many engineers work went poof. 


Title: Re: Crowdstrike
Post by: Eagler on July 20, 2024, 05:16:36 AM
I wouldn't want to be the coder.

Like the guy that cost that Mars probe because he forgot to convert from meters to feet.  DOh!!!!!!  10 years of how many engineers work went poof.

QA should have caught that too...

Then again we contract most of our software dev to the lowest bidder regardless of the language they speak..so there's that.

Eagler
Title: Re: Crowdstrike
Post by: Spikes on July 20, 2024, 09:35:11 AM
We were down for a few hours. About 2/3 of our endpoints got hit. Half were fine after a reboot, the other half needed the sys file removed manually via backdoor.
Title: Re: Crowdstrike
Post by: Meatwad on July 20, 2024, 10:11:25 AM
If a software glitch could neuter that much stuff, some third world country now knows it can do much more harm then previously expected if it was to be done again by evil
Title: Re: Crowdstrike
Post by: Eagler on July 20, 2024, 10:40:28 AM
We were down for a few hours. About 2/3 of our endpoints got hit. Half were fine after a reboot, the other half needed the sys file removed manually via backdoor.

My youngest son at Cisco cyber security told me that was the fix the morning of the accident..it's just now getting around to all the machines to manually make the corrections..

Eagler
Title: Re: Crowdstrike
Post by: LCADolby on July 20, 2024, 12:46:06 PM
My works PCs were down all Friday. Thankfully I have used AVG since 2003 and avoided all issues.
Title: Re: Crowdstrike
Post by: icepac on July 20, 2024, 04:03:36 PM
They definitely rushed a build out into the world.
Title: Re: Crowdstrike
Post by: fd ski on July 20, 2024, 04:31:02 PM
i remember the days when each company would decide themselves when to deploy updates to windows machines. It's unthinkable to me that almost entire world just straight up downloads anything that microsoft puts out. Maybe this will sober up some IT maintenance folks out of their complacency.
Title: Re: Crowdstrike
Post by: AKIron on July 20, 2024, 04:33:08 PM
Security software. You want zero-day protection. From the user perspective. Nothing should ever be pushed out by a developer before adequately testing.
Title: Re: Crowdstrike
Post by: Vulcan on July 20, 2024, 06:18:33 PM
Security software. You want zero-day protection. From the user perspective. Nothing should ever be pushed out by a developer before adequately testing.

Exactly, Crowdstrike really screwed up on this. I'm honestly gobsmacked this got released by them, as it appears to affect 100% of machines 100% of the time. Usually it's some weird combination that triggers these problems.

That said, all major antivirus vendors have experienced similar issues over the last 20+ years. At work we do our sensor updates in stages with reasonable time gaps in between (over a period of 2 weeks iirc). We also don't update the backup failover servers. So if a bad update comes out we can mitigate the issue by failing over. Of our fleet around 20% of devices (servers and laptops) were impacted (including my laptop). Because we use bitlocker there is no backdoor without a lot of fluffing around.

So all those companies that were completely nuked by this weren't following good practices.

I also thought it was funny how Microsoft themselves were so heavily impacted. Given they tell everyone they don't need Crowdstrike and should use Windows Defender... obviously Microsoft don't believe their own sales pitch.
Title: Re: Crowdstrike
Post by: Volron on July 20, 2024, 10:56:43 PM
Then again we contract most of our software dev to the lowest bidder regardless of the language they speak..so there's that.

We do that with a lot of stuff, sadly.
Title: Re: Crowdstrike
Post by: fd ski on July 21, 2024, 06:15:08 AM
Security software. You want zero-day protection. From the user perspective. Nothing should ever be pushed out by a developer before adequately testing.

i understand what zero day protection is, however if you have 1000 machines well behind firewalls, honey pots etc running your entire business, chances of even zero day exploit affecting them are marginal. Risk of this kind of screw up is much higher then non-internet exposed machines being affected.

In reality, there should be "urgent zero day" patch application as totally separate process for other patches that should be delayed by large companies. I'm willing to bet this particular update had no zero day protections in it...
Title: Re: Crowdstrike
Post by: Eagler on July 21, 2024, 06:23:34 AM
Shows how vulnerable we really are regardless of how secure we believe we are...dangerous things can happen from all angles and resources

Imagine if it would have been power grids instead of banks and airports?

>100 heat index doesn't do nice things to patience

Eagler
Title: Re: Crowdstrike
Post by: Vulcan on July 21, 2024, 02:35:18 PM
i understand what zero day protection is, however if you have 1000 machines well behind firewalls, honey pots etc running your entire business, chances of even zero day exploit affecting them are marginal. Risk of this kind of screw up is much higher then non-internet exposed machines being affected.

In reality, there should be "urgent zero day" patch application as totally separate process for other patches that should be delayed by large companies. I'm willing to bet this particular update had no zero day protections in it...

It's all about layers. You never assume any layer is perfect, so a failure to detect in one layer is picked up by another.

Crowdstrike does it's analysis in the cloud, and nor is it a signature based scanner (it can do it, but that feature only came out recently, as in last few months). So sensor patches are usually things other zero day patches.
Title: Re: Crowdstrike
Post by: Shuffler on July 22, 2024, 12:39:10 PM
Two engineering firms we deal with have recently started using firms in India to put together their drawings. The quality has noticeably gone way down. Their cost has noticeably gone way up. Now they pay us far more.
Title: Re: Crowdstrike
Post by: Mayhem on August 07, 2024, 05:05:06 PM
I can't say Crowdstrike did me a dirty .... thanks to their screw up and the resulting overtime, I have the funds for an expensive Monstertech Sim Chair and a Virpil AH64 collective.
Title: Re: Crowdstrike
Post by: AKIron on August 07, 2024, 05:10:41 PM
You probably won't hear that a foreign actor infiltrated their organization and sabotaged that patch. Wouldn't be prudent for a security company to advertise that. If it happened.
Title: Re: Crowdstrike
Post by: DmonSlyr on August 08, 2024, 09:36:02 AM
You probably won't hear that a foreign actor infiltrated their organization and sabotaged that patch. Wouldn't be prudent for a security company to advertise that. If it happened.

I heard a rumor that it was actually Azure that suffered a major outage, which is tied to the White House AI technology, which then they called Crowdstrike to fix it immediately. CS created a patch to fix it, but failed a driver that completely rekt everything. A buddy was telling me you never release a patch update on Friday. So it was very strange that they did.

Just a rumor. Cannot confirm and could be total BS.