So would your hypothesis say they have visited us, but decided not to contact us or investigate because it's just another infested planet, one among many, who cares?
I don't see how that relates to "we haven't seen other intelligences" other than that, because if other intelligences are very common, it seems at least one would have sooner or later decided to piss in our pond.
Worthy of attention or not (and I despise this 'humans are garbage' self-deprecating schtick btw, we are absolutely the center of the universe TO US, regardless of the truth of that) they still would have interacted with us in some way if intelligences and interstellar travel were as common as you postulate.