Yeah it's true... I have like 45 6* units now... and I only use a few of those. Pretty much the only units i'm using right now, day in and day out are..
Maxwell, Elza, Exvehl, Kuhla, Duel-SGX, Zelnite, Farlon (Arena), Melchio with Lexida (Arena).. that's it!
I'll use like Lilly Matah and Darvanshel, maybe Rowgen or Lilith and one of my healers if there is any challenging content.. meh, that's like 12 units or so I guess.