A lot of seafood has a similar "squishy" texture. Crabs, lobsters, shrimp, that sort of stuff. When people say they don't like seafood, they might not mean all seafood, they just don't want to get into a nuanced discussion that's going to end unsatisfactorily.
I don't mind the taste of the aforementioned foods, broths or stocks made from them are great, just can't get away from the texture. Same reason I don't like mushrooms. I'll gladly eat most fish.
Couple complicating factors: good seafood is more expensive than good turf, sometimes even when you live on the coast. How long the food stays fresh plus price can deter a lot of people from trying the food. Thus they are left with bad impressions because they tried poor representations.