IT/Software career thread: Invert binary trees for dollars.

  • Guest, it's time once again for the massively important and exciting FoH Asshat Tournament!



    Go here and give us your nominations!
    Who's been the biggest Asshat in the last year? Give us your worst ones!
Noodleface

Noodleface

A Mod Real Quick
38,287
15,129
Well we inherently reject the code review by not accepting it. But others can accept it.

His claim was it didn't matter because technically it was used. My argument is it's a waste of space.
 
Noodleface

Noodleface

A Mod Real Quick
38,287
15,129
I have to post here complaining about HR again

I've called them countless times (literally no idea how many) to square away shit. First I can't access ADP, then the info isn't populating, then my accruals are wrong, then I didn't get paid my first week, then my accruals were updated wrong, then I found out they're using my old EMC email address. Literally the list goes on and on, that's just what I remember. I have called them multiple times every week. But finally everything looked good. So I took a vacation

And they paid me out at my rate from December 2013. When I made $76k. I now make $126k.

Fucking unbelievable. I have a feeling these are all bandaids and something is really fucked up behind the scenes.

This is the type of shit that makes you not want to stay at a company. I might call payroll and ask someone to escalate it. There's no reason I should be constantly having to deal with HR
 
Deathwing

Deathwing

<Bronze Donator>
16,768
7,790
You'll be amazed at what you'll put up with. I mean, I feel you, I get the same feelings too. We have reviews coming up soon and if my boss lowballs me again I'm out. But it's mostly going to be because our IT department is beyond incompetent. We had a server room meltdown last August where the AC failed but the servers kept running, essentially cooking themselves.

This happened again in November. Cook my test machines once, shame on me. Cook them again, shame on you. It happened again in February. There's been repeated incidents since then where IT is so spooked that they were present to manually avert any cooking.

The most recent one was last week where the IT guy I was talking to was implying I should be thankful someone was onsite at 3 AM to prevent any damage. The building lost power, the AC didn't kick back on, but the servers kept producing heat because of UPS backups. I think the guy was too sleep deprived to recognize the incredulous look I was giving him. Am I wrong in expecting there to be automated failsafes that should have prevented just this type of situation?

What kind of AC system doesn't recover from power loss? And if it doesn't, why isn't the system built around that?

I know IT people aren't electricians, they can't fix the dirty power we're receiving(and we are). But after a year of this shit, I feel like this is now their fault.
 
TJT

TJT

Mr. Poopybutthole
<Gold Donor>
42,816
109,309
That's easy. Your server guys may be competent at generally running servers but the problem you're talking about is a facilities management issue. Which if it isn't 100% your building you're talking about dealing with a kind of MSP for the space.

Shitty all around.

if you had a really clever server guy what you would do is have some kind of janky work around where the server environment could detect if the AC has gone out. But most server guys probably wouldn't be interested or have the drive to make something like that. so you have round-the-clock monitoring because no one can be bothered to lean on the facilities people to fix the AC
 
Deathwing

Deathwing

<Bronze Donator>
16,768
7,790
TJT said:
That's easy. Your server guys may be competent at generally running servers but the problem you're talking about is a facilities management issue. Which if it isn't 100% your building you're talking about dealing with a kind of MSP for the space.

Shitty all around.

if you had a really clever server guy what you would do is have some kind of janky work around where the server environment could detect if the AC has gone out. But most server guys probably wouldn't be interested or have the drive to make something like that. so you have round-the-clock monitoring because no one can be bothered to lean on the facilities people to fix the AC
Click to expand...
I guess I can understand if they wouldn't be strictly interested in doing that because it's not the "right way". But after almost a year of this shit, why not implement a failsafe that takes a week at most. Our test system already detects if the system is in a dirty state and has the power to reboot the machine, half the code is already there.
 
chaos

chaos

Buzzfeed Editor
17,324
4,839
When I worked a service desk back in the day we had that happen, they created a dashboard for us to monitor temps and check the ac if the temps started rising so we could alert facilities. Not exactly rocket surgery. They have automation for all that shit now, the servers themselves have automation for alerting on temp and sending alerts via SMS, email, etc. Sounds like they need to get fucking good.
 
  • 1Like
Reactions: 1 user
Kiki

Kiki

Log Wizard
2,323
1,934
We have temp sensors, humidity sensors, etc. No reason they cant' get an alarm to their phone to worry about it. Also, you can have servers shut themselves down before permanent damage if you set them up right, Dell builds this right into DRAC:

215292
 
Ao-

Ao-

¯\_(ツ)_/¯
<WoW Guild Officer>
7,879
507
Anyone have experience working for one of the Big4 consulting firms? I'm tempted to do a 2yr stint but the 80% travel sort of scares me.
 
Vinen

Vinen

God is dead
2,790
495
Ao- said:
Anyone have experience working for one of the Big4 consulting firms? I'm tempted to do a 2yr stint but the 80% travel sort of scares me.
Click to expand...

They are shit companies to work for and pay terrible in general. Also within the industry there is a shift away from using them. Many of us look down on those companies now.

I've had to unwind more than one fuckup by them at customer sites.
 
TJT

TJT

Mr. Poopybutthole
<Gold Donor>
42,816
109,309
I'm by no means an infrastructure guy but the performance monitoring dashboard you're looking at there would be indicating the temperature inside the server or the ambient temperature in the room?

Maybe I am nitpicking after being a perf engineer for so long (but for applications not infrastructure). It would take considerable effort to create some logic that could separate heat generated by workload within the machine to heat buildup caused by increased room temperature and expected workload. Especially if you have a ton of shit going on in your data center. I guarantee you the server guys here are just using whatever infrastructure monitoring tools that are exclusively software. They don't hook into the facility itself like the thermostat or something.

That being said I was under the impression that too much heat would cause a machine to just shutdown rather than continue working until it burned out. Like how laptops and desktops do by default. Do servers have that design feature disabled for other reasons?

But yes, ideally they would have something that sent an alert when the room temperature surpassed Z degrees. With warnings sent at X and Y degrees.
 
Noodleface

Noodleface

A Mod Real Quick
38,287
15,129
Dells iDRAC is monitoring sensors inside the server. Whether it's caused by environmental or internal isn't really going to change the reading. They'll definitely shut down at temps

Not sure what deathwing is using

You can disable this, like one might for certain testing. But I don't see that being the norm
 
Kiki

Kiki

Log Wizard
2,323
1,934
Yes they disable shutdown because *uptime*. But you can enable it as my screenshot shows.

We have sensors in the rack in addition to the sensors on the servers themselves, and they both can send emails, etc. So having someone physically watching it is crazy to me.
 
Deathwing

Deathwing

<Bronze Donator>
16,768
7,790
Kiki Kiki has the right of it. Alerts are not good enough, what if everyone is sleeping? Uptime is stupid, I'm not going to complain about my machines being shutdown if the alternative was them getting cooked.
 
Kiki

Kiki

Log Wizard
2,323
1,934
Well I mean, i have to keep my phone by the bed and that text better wake me up or else I'm getting yelled at.
 
Noodleface

Noodleface

A Mod Real Quick
38,287
15,129
Deathwing said:
Kiki Kiki has the right of it. Alerts are not good enough, what if everyone is sleeping? Uptime is stupid, I'm not going to complain about my machines being shutdown if the alternative was them getting cooked.
Click to expand...
So why is thermal shutdown disabled is the real question , either someone manually did that or it's an inferior product
 
  • 1Like
Reactions: 1 user
Deathwing

Deathwing

<Bronze Donator>
16,768
7,790
Don't know if our servers have that option. I know for certain some of them don't. We have ancient hardware because we build installers for some esoteric platforms.
 
TJT

TJT

Mr. Poopybutthole
<Gold Donor>
42,816
109,309
Yeah that is a stupid way to do it even if uptime is the concern. Suppose you get an alert caused by a facilities issue that requires you to contact facilities to repair the AC. What the fuck are you going to do about uptime then? You have to just remotely shutdown your shit to prevent days of loss of uptime and thousands of dollars in damage to the servers. You turn a mild inconvenience into an utter catastrophe.

Your dudes would still have to get the alert. Drive to the facility to make sure that it is the AC that's dead. Contact facilities then shutdown your servers. Or wait until the warning becomes a critical alert then just remotely kill them all... then drive to the facility. Then contact facilities. Or be there 24/7.
 
Noodleface

Noodleface

A Mod Real Quick
38,287
15,129
We also have redundancy where if we lose ac/cooling the power is shut off to the lab. I know that I'm talking about a massive corporation here though
 
You must log in or register to reply here.