IT/Software career thread: Invert binary trees for dollars.
- Thread starter Traak
- Start date
-
Guest, it's time once again for the massively important and exciting FoH Asshat Tournament!
Go here and give us your nominations!Who's been the biggest Asshat in the last year? Give us your worst ones!
Siliconemelons
Naxxramas 1.0 Raider
- 11,899
- 17,876
moonarchia
The Scientific Shitlord
- 23,583
- 42,936
My company is so cheap it would make Scrooge blush. So of course we do not have the resources to test anything like that, so that stuff went live immediately.
rhinohelix
Dental Dammer
- 3,050
- 5,026
I work for a Fortune 10 company that is constantly the target of radical ideologues, state actors and cyber criminals. Normally you would have a test/dev/stage level of deployment for most things, and even for most anti-malware stuff, there is some level of testing. CrowdStrike is different, though, and is trusted to protect against 0-day attacks; for that, you don't pass go and trust their updates, assuming they aren't going to fuck you. They have made billions being the go-to provider for a huge chuck of the world in that regard. It appears they let a broken kernel-level .sys driver as part of its Falcon Sensor update out into the wild which caused BSODs on every version of Windows it touched. It didn't break Azure or AWS itself but rather all the VMs.
Easy to fix: All you had to do was manually boot into Safe/Recovery mode and delete the file. Let me say that again.
All you had to do
is
MANUALLY
Log in
to every Virtual machine in every private host and public/private cloud you had VMs/Guests.
In a world in which people don't have to pay for physical hardware, the VM estate sprawl and labor shrinkage that has taken place in the last 15 years in AMAZING: Fewer people are managing more machines than ever before.
It's an interesting moment and a wake up call for many folks.
Easy to fix: All you had to do was manually boot into Safe/Recovery mode and delete the file. Let me say that again.
All you had to do
is
MANUALLY
Log in
to every Virtual machine in every private host and public/private cloud you had VMs/Guests.
In a world in which people don't have to pay for physical hardware, the VM estate sprawl and labor shrinkage that has taken place in the last 15 years in AMAZING: Fewer people are managing more machines than ever before.
It's an interesting moment and a wake up call for many folks.
Palum
what Suineg set it to
- 26,647
- 41,586
I still maintain anyone who didn't follow any change control and auto updates everything direct into prod is an idiot begging for exactly what happened Friday. I don't understand how after having so many supply chain attacks everyone forgot that too.
- 2
- 1
rhinohelix
Dental Dammer
- 3,050
- 5,026
You're not wrong; in a perfect world, everyone should follow textbook ITIL design. I used to be a Datacenter Mgr at a mid-sized company, and we had perfect ITIL test/dev/stage environments for every app or suite; Change Management was rigorously followed, and things were done by the book unless we were in a Major Incident. 0-Day attacks weren't quite the thing back then but as I moved to larger companies, the rigorous T/D/S segmentation broke down as the environments got larger and the estates sprawled out, and were subject to business cycles. When you have a couple thousand servers, its one thing. When you have 35,000+ servers and 100,000 laptops, (and more as the size and footprint of the enterprise grows) quantity takes on a quality all its own. IRM/CyberDefense isn't my area but they determined that speed of response was more important than extensive QA testing/prevention, particularly with how well CrowdStrike was trusted.
They may want to rethink that architecture after Friday.
- 6,984
- 9,904
Turns out our infosec engineers had auto in prod on.
Thousands of servers bricked. Lolz
- 1
- 12,761
- 49,670
Another option to take exists. I'm not sure if it exists in Crowdstrike as it's been a hot minute since I was working through their UI and options but...
In most modern EDR/NGAV tools I have worked with there are options for "Auto-update, but delay X days before applying it". Which then would have bought you time to hit the "DO NOT UPDATE" button if a bad patch got out and bombed others.... This is a middle ground for those who don't have the resources to do their own full dev/test/prod motion and figure to let the rest of the world take the hit.
Giving ANY tool, even your trusted to stop zero days tool, immediate access is taking on risk. Now, question becomes if you hadn't had absolutely lightning fast coverage for zero days how many system hours would you have lost, or data lost versus what you experienced Friday.
- 1
Palum
what Suineg set it to
- 26,647
- 41,586
Yea you can set host groups to n-1 on auto and then push updates I'm pretty sure.Another option to take exists. I'm not sure if it exists in Crowdstrike as it's been a hot minute since I was working through their UI and options but...
In most modern EDR/NGAV tools I have worked with there are options for "Auto-update, but delay X days before applying it". Which then would have bought you time to hit the "DO NOT UPDATE" button if a bad patch got out and bombed others.... This is a middle ground for those who don't have the resources to do their own full dev/test/prod motion and figure to let the rest of the world take the hit.
Giving ANY tool, even your trusted to stop zero days tool, immediate access is taking on risk. Now, question becomes if you hadn't had absolutely lightning fast coverage for zero days how many system hours would you have lost, or data lost versus what you experienced Friday.
- 1
Khane
Got something right about marriage
- 20,352
- 14,030
My company doesn't auto push updates for anything except for, apparently, crowdstrike because it went out to every single machine in our company on Friday including all of our personal laptops lol.
I couldn't even get anyone from the help desk on the line to get me my bitlocker key so I could delete the offending file and get past the BSODs until Sunday. It was hilarious.
I couldn't even get anyone from the help desk on the line to get me my bitlocker key so I could delete the offending file and get past the BSODs until Sunday. It was hilarious.
- 1
rhinohelix
Dental Dammer
- 3,050
- 5,026
I am not on the frontlines but the CyberDefense war is real, yo. Nobody wants to be Maersk, Colonial Pipeline or fall victim to a Solar Winds™ style extended attack.
- 1
Palum
what Suineg set it to
- 26,647
- 41,586