Author Topic: Crowdstrike  (Read 1542 times)

Offline Eagler

  • Plutonium Member
  • *******
  • Posts: 18221
Crowdstrike
« on: July 19, 2024, 03:59:06 PM »
I have done software qa testing in the past..

I wouldn't want to be on the team that passed that update

Eagler
"Masters of the Air" Scenario - JG27


Intel Core i7-13700KF | GIGABYTE Z790 AORUS Elite AX | 64GB G.Skill DDR5 | 16GB GIGABYTE RTX 4070 Ti Super | 850 watt ps | pimax Crystal Light | Warthog stick | TM1600 throttle | VKB Mk.V Rudder

Offline Busher

  • Gold Member
  • *****
  • Posts: 2150
Re: Crowdstrike
« Reply #1 on: July 19, 2024, 04:30:23 PM »
Their flight home might be delayed :rolleyes:
Being male, an accident of birth. Being a man, a matter of age. Being a gentleman, a matter of choice.

Offline CptTrips

  • Plutonium Member
  • *******
  • Posts: 8269
Re: Crowdstrike
« Reply #2 on: July 19, 2024, 05:37:24 PM »
I have done software qa testing in the past..

I wouldn't want to be on the team that passed that update

Eagler

 I wouldn't want to be the coder.

Like the guy that cost that Mars probe because he forgot to convert from meters to feet.  DOh!!!!!!  10 years of how many engineers work went poof. 


Toxic, psychotic, self-aggrandizing drama queens simply aren't worth me spending my time on.

Offline Eagler

  • Plutonium Member
  • *******
  • Posts: 18221
Re: Crowdstrike
« Reply #3 on: July 20, 2024, 05:16:36 AM »
I wouldn't want to be the coder.

Like the guy that cost that Mars probe because he forgot to convert from meters to feet.  DOh!!!!!!  10 years of how many engineers work went poof.

QA should have caught that too...

Then again we contract most of our software dev to the lowest bidder regardless of the language they speak..so there's that.

Eagler
"Masters of the Air" Scenario - JG27


Intel Core i7-13700KF | GIGABYTE Z790 AORUS Elite AX | 64GB G.Skill DDR5 | 16GB GIGABYTE RTX 4070 Ti Super | 850 watt ps | pimax Crystal Light | Warthog stick | TM1600 throttle | VKB Mk.V Rudder

Offline Spikes

  • Aces High CM Staff
  • Plutonium Member
  • *******
  • Posts: 15724
    • Twitch: Twitch Feed
Re: Crowdstrike
« Reply #4 on: July 20, 2024, 09:35:11 AM »
We were down for a few hours. About 2/3 of our endpoints got hit. Half were fine after a reboot, the other half needed the sys file removed manually via backdoor.
i7-12700k | Gigabyte Z690 GAMING X | 64GB G.Skill DDR4 | EVGA 1080ti FTW3 | H150i Capellix

FlyKommando.com

Offline Meatwad

  • Plutonium Member
  • *******
  • Posts: 12797
Re: Crowdstrike
« Reply #5 on: July 20, 2024, 10:11:25 AM »
If a software glitch could neuter that much stuff, some third world country now knows it can do much more harm then previously expected if it was to be done again by evil
See Rule 19- Do not place sausage on pizza.
I am No-Sausage-On-Pizza-Wad.
Das Funkillah - I kill hangers, therefore I am a funkiller. Coming to a vulchfest near you.
You cant tie a loop around 400000 lbs of locomotive using a 2 foot rope - Drediock on fat women

Offline Eagler

  • Plutonium Member
  • *******
  • Posts: 18221
Re: Crowdstrike
« Reply #6 on: July 20, 2024, 10:40:28 AM »
We were down for a few hours. About 2/3 of our endpoints got hit. Half were fine after a reboot, the other half needed the sys file removed manually via backdoor.

My youngest son at Cisco cyber security told me that was the fix the morning of the accident..it's just now getting around to all the machines to manually make the corrections..

Eagler
"Masters of the Air" Scenario - JG27


Intel Core i7-13700KF | GIGABYTE Z790 AORUS Elite AX | 64GB G.Skill DDR5 | 16GB GIGABYTE RTX 4070 Ti Super | 850 watt ps | pimax Crystal Light | Warthog stick | TM1600 throttle | VKB Mk.V Rudder

Offline LCADolby

  • Platinum Member
  • ******
  • Posts: 7315
Re: Crowdstrike
« Reply #7 on: July 20, 2024, 12:46:06 PM »
My works PCs were down all Friday. Thankfully I have used AVG since 2003 and avoided all issues.
JG5 "Eismeer"
YouTube+Twitch - 20Dolby10


"BE a man and shoot me in the back" - pez

Offline icepac

  • Platinum Member
  • ******
  • Posts: 6998
Re: Crowdstrike
« Reply #8 on: July 20, 2024, 04:03:36 PM »
They definitely rushed a build out into the world.

Offline fd ski

  • Silver Member
  • ****
  • Posts: 1525
      • http://www.northotwing.com/wing/
Re: Crowdstrike
« Reply #9 on: July 20, 2024, 04:31:02 PM »
i remember the days when each company would decide themselves when to deploy updates to windows machines. It's unthinkable to me that almost entire world just straight up downloads anything that microsoft puts out. Maybe this will sober up some IT maintenance folks out of their complacency.

Offline AKIron

  • Plutonium Member
  • *******
  • Posts: 12799
Re: Crowdstrike
« Reply #10 on: July 20, 2024, 04:33:08 PM »
Security software. You want zero-day protection. From the user perspective. Nothing should ever be pushed out by a developer before adequately testing.
Here we put salt on Margaritas, not sidewalks.

Offline Vulcan

  • Plutonium Member
  • *******
  • Posts: 9891
Re: Crowdstrike
« Reply #11 on: July 20, 2024, 06:18:33 PM »
Security software. You want zero-day protection. From the user perspective. Nothing should ever be pushed out by a developer before adequately testing.

Exactly, Crowdstrike really screwed up on this. I'm honestly gobsmacked this got released by them, as it appears to affect 100% of machines 100% of the time. Usually it's some weird combination that triggers these problems.

That said, all major antivirus vendors have experienced similar issues over the last 20+ years. At work we do our sensor updates in stages with reasonable time gaps in between (over a period of 2 weeks iirc). We also don't update the backup failover servers. So if a bad update comes out we can mitigate the issue by failing over. Of our fleet around 20% of devices (servers and laptops) were impacted (including my laptop). Because we use bitlocker there is no backdoor without a lot of fluffing around.

So all those companies that were completely nuked by this weren't following good practices.

I also thought it was funny how Microsoft themselves were so heavily impacted. Given they tell everyone they don't need Crowdstrike and should use Windows Defender... obviously Microsoft don't believe their own sales pitch.
« Last Edit: July 20, 2024, 06:26:04 PM by Vulcan »

Offline Volron

  • Platinum Member
  • ******
  • Posts: 5805
Re: Crowdstrike
« Reply #12 on: July 20, 2024, 10:56:43 PM »
Then again we contract most of our software dev to the lowest bidder regardless of the language they speak..so there's that.

We do that with a lot of stuff, sadly.
Quote from: hitech
Wow I find it hard to believe it has been almost 38 days since our last path. We should have release another 38 versions by now  :bhead
HiTech
Quote from: Pyro
Quote from: Jolly
What on Earth makes you think that i said that sir?!
My guess would be scotch.

Offline fd ski

  • Silver Member
  • ****
  • Posts: 1525
      • http://www.northotwing.com/wing/
Re: Crowdstrike
« Reply #13 on: July 21, 2024, 06:15:08 AM »
Security software. You want zero-day protection. From the user perspective. Nothing should ever be pushed out by a developer before adequately testing.

i understand what zero day protection is, however if you have 1000 machines well behind firewalls, honey pots etc running your entire business, chances of even zero day exploit affecting them are marginal. Risk of this kind of screw up is much higher then non-internet exposed machines being affected.

In reality, there should be "urgent zero day" patch application as totally separate process for other patches that should be delayed by large companies. I'm willing to bet this particular update had no zero day protections in it...

Offline Eagler

  • Plutonium Member
  • *******
  • Posts: 18221
Re: Crowdstrike
« Reply #14 on: July 21, 2024, 06:23:34 AM »
Shows how vulnerable we really are regardless of how secure we believe we are...dangerous things can happen from all angles and resources

Imagine if it would have been power grids instead of banks and airports?

>100 heat index doesn't do nice things to patience

Eagler
"Masters of the Air" Scenario - JG27


Intel Core i7-13700KF | GIGABYTE Z790 AORUS Elite AX | 64GB G.Skill DDR5 | 16GB GIGABYTE RTX 4070 Ti Super | 850 watt ps | pimax Crystal Light | Warthog stick | TM1600 throttle | VKB Mk.V Rudder