What are you exact testosterone levels? I had the same problem for awhile, got my testosterone tested, and had a result of 350 or so which is technically in the normal range. The thing is, the acceptable range of 320-1000 is huge, and though 320 might be ok if you're 70 years old, it isn't if you are 35. My normal doc dismissed low T as a problem, but when I saw an endocrinologist he explained the above, and some T-gel definitely helped.
I have no baseline though, as I've never had it checked before. I'm falling right in the mid lvl range