IT/Software career thread: Invert binary trees for dollars.

  • Guest, it's time once again for the massively important and exciting FoH Asshat Tournament!



    Go here and give us your nominations!
    Who's been the biggest Asshat in the last year? Give us your worst ones!

Voyce

Shit Lord Supreme
<Donor>
8,463
30,558
I have awoken to a World of Shit and Piss


Edit: Actually not that bad, just more annoying bullshit, who pushes a Patch out on a Friday?

Was right the first time, tons of crap still nonfunctional, can't get other things going yet, due to integrated dependencies
 
Last edited:

Ossoi

Potato del Grande
<Rickshaw Potatoes>
17,848
8,777
I got to train station and their card machines weren't working. Am traveling to a music festival and no idea if their card machines will work plus it was meant to be a cashless event


Ironically I wasn't really feeling motivated to attend and put my ticket on a resell site, but decided to go and booked a non refundable hotel.
 

TomServo

<Bronze Donator>
7,176
10,510
I got to train station and their card machines weren't working. Am traveling to a music festival and no idea if their card machines will work plus it was meant to be a cashless event


Ironically I wasn't really feeling motivated to attend and put my ticket on a resell site, but decided to go and booked a non refundable hotel.
no one gives a fuck what you do, ya autistic bastard
 

fred sanford

<Gold Donor>
1,737
5,079
1721413430033.png


Can't have a breach if your PC is down, checkmate hackers
 
  • 2Galaxy Brain
Reactions: 1 users

Palum

what Suineg set it to
27,213
42,971
So final tally was 1 UAT server and about 12% of our PCs impacted. Besides on call staff who were working last night all my IT folks started at 8 and went home on time.

Anyone who was crippled by this was incompetent and unprepared.

I was also told "you were right" six times today independently.
 
  • 1Pants on Fire!
Reactions: 1 user

Szlia

Member
6,634
1,376
Layman question as the news I saw were not very clear about what Crowdstrike does exactly: is it fair to put that in the big "digital protection being more damaging than what it is supposed to protect you against" pile?
 

Palum

what Suineg set it to
27,213
42,971
Layman question as the news I saw were not very clear about what Crowdstrike does exactly: is it fair to put that in the big "digital protection being more damaging than what it is supposed to protect you against" pile?

Kind of. The piece that got updated was their Falcon EDR (Endpoint Detection and Response) tool. It's basically like antivirus on crack, and it can be setup to detect AND quarantine/fix attacks. The problem is this level of response requires system access far in excess of your typical windows defender/antivirus sort of thing. That level of access and integration into the OS means things can go very bad.
 

TJT

Mr. Poopybutthole
<Gold Donor>
43,061
110,051
Oh wow it really forces you to manually mess around with the registry to fix it?

Gyat dyamn.

Also, this is exactly why having fucking AI have total access to your infrastructure to "automate" won't happen lol. This is even worse than if you let an AI loose in your tech infrastructure to "find improvements to make."
 

Palum

what Suineg set it to
27,213
42,971
Oh wow it really forces you to manually mess around with the registry to fix it?

Gyat dyamn.

Also, this is exactly why having fucking AI have total access to your infrastructure to "automate" won't happen lol. This is even worse than if you let an AI loose in your tech infrastructure to "find improvements to make."

No, you can delete the offending file so it boots and updates back to a stable version. The problem is getting to the file...
 

fred sanford

<Gold Donor>
1,737
5,079
The IT incompetence part is that you can update manually or let it update automatically. At least that’s what I assume/heard. In our case security had it set to auto across the board. Over 1000 remote locations with roughly 20 PCs each became potatoes. We’re the service industry so it was a mad dash before locations opened. Some they could fix with a script, for others they had to recruit the entire department (with temporary access) to help use remote consoles on the PCs, for the rest we had to call the locations and walk them through it.

Yes our security team is retarded. Every time we get a good person they leave within a year. Thankfully 95% of what I own at work is on Linux.
 
  • 1Like
Reactions: 1 user

ToeMissile

Pronouns: zie/zhem/zer
<Gold Donor>
3,272
2,143
I’m pretty sure we use crowdstrike as well, I didn’t hear of any PCs with issues, but a couple of the main DB/app/services used across the company were. I didn’t bother to look into it but assumed it wasn’t a coincidence
 

Haus

<Silver Donator>
13,063
50,957
I’m pretty sure we use crowdstrike as well, I didn’t hear of any PCs with issues, but a couple of the main DB/app/services used across the company were. I didn’t bother to look into it but assumed it wasn’t a coincidence
The default "best practice" that all companies in this space tend to default to is "allow automatic updates so you always have the latest protections". What most large organizations then change that to is a "Dev/Test/Prod" approach. New content updates for EDR/AV start in "Test", you deploy to a good reprepsentative sample and make sure it's solid/safe. Usually let it bake for a week, then allow the update to the production environment.

All those affected Friday were in the "we just update automatically" crowd from what I can tell.
 
  • 3Like
Reactions: 2 users

Palum

what Suineg set it to
27,213
42,971
The default "best practice" that all companies in this space tend to default to is "allow automatic updates so you always have the latest protections". What most large organizations then change that to is a "Dev/Test/Prod" approach. New content updates for EDR/AV start in "Test", you deploy to a good reprepsentative sample and make sure it's solid/safe. Usually let it bake for a week, then allow the update to the production environment.

All those affected Friday were in the "we just update automatically" crowd from what I can tell.

Yea, for us these are prioritized but still staggered. So none of our critical stuff got hit, but we had some desktops go tits up. Not ideal but I guess the rationalization was they couldn't screw up this bad and risk/reward whatever. That will probably change at some point now...