Nuts to that. That is why I got my own bloodwork. I was tired of seemingly arbitrary rules. A symptom is a symptom. My initial symptoms were primarily libido, energy, and fatigue. I didn't have ED, but I had absolutely no sex drive for around 2 years. It was declining as I got older, but I figured that was just par for the course. After doing some research I attributed depression and anxiety.
I also think it was causing me to be more injury prone (recovery) and I felt like my workouts weren't paying off as much as every other person purports from their training. Once I got over the "Just man up, chump" mentality and investigated it like a problem I found out what I was experiencing was not normal.
If you get panels done, just make sure to get all of the tests. My total Test was <200 so I did not have a problem justifying past that point. However, looking at LH / FSH (pituitary hormones), SHBG, Estradiol, and actual bioavailable testosterone seems pretty critical for plan of attack. You could have a decent Total Test, but because of high SHBG, it could manifest as low bio-available testosterone. Or, as in my case, my pituitary didn't seem to be releasing LH, which is what triggers testosterone production.
I would highly recommend getting cursory blood work if you think you are being affected. While I am not back to 100%, my life is tangibly better. My mood, outlook, energy, etc. all improved.
If your pituitary is not releasing LH, I would be worried about other hormones as well... HGH, cortisol etc. A deficit in any of these hormones is going to make you feel bad and slow recovery. HGH is hard to test though, need to do provocative testing.
Problems with pituitary can be due to concussion if that fits your history. Some populations of retired boxers for instance have prevalence of HGH deficiency above 50%. These deficiencies rarely come isolated.
I'm in a similar situation: I've been lifting 5X5 5 times a week for 6month with no progress since my concussion. I'm thinking of getting some tests done.