I don't think there's anything wrong with whittling down the show; it doesn't detract from how great the show is, but let's be honest. ASoIaF is perhaps the greatest fantasy literary work since Tolkien. There's a tremendous amount of depth that can't come across in the show, and that's going to disappoint people.
However, what I'm worried about is not disappointment, because the book and the show are not the same thing. What I am worried about are the people who are going to get the wrong idea about characters because of how easily they are revealed, or introduced, or how they are giving conversations they never have in the book.
For example, season 1, a few people said to me, wow Dany is such a whore, just like Cersei. Using her twat to get ahead! In the show, Khal Drogo rapes her while she cries, then asks Doreah to teach her how to make the Khal love her. So, it's not inconceivable that someone could assume she has a fucking beaten woman syndrome and now she wants to make the guy who beat her love her.
In the book obviously, we know from the very first ride that he's actually kind, and Dany lucked out. Doreah showing her pillow tricks were more to have the Khal see her as an equal, or at the very least love her so that he would listen to what she had to say. Which was pertinent since she needed to dothraki to cross the sea to take back her throne.
I simply whimper a bit inside when I see the show, because I don't want to hear 4 episodes in that "Oh my god, poor Cersei" since they've made her a more sympathetic character in the show, or "Wow, Selmy, what a turn cloak! Went over to the other side pretty fast eh?"
When you draw a direct line between two events, two characters or two mindsets people are going to assume that the encounter in question was as easy for the person as it was in the show. The quickest route between two points and all that. And the book is absolutely wonderful, in that, even if the decision was fucking stupid, you probably can't blame the character for making it, because you know why they did. And obviously I'm not as worried about reasoning, because that's always hard to show visually, but I certainly will feel bad for anyone just watching the show for not getting the full package of the characters. Everyone in this show is ridiculously complex, and it's a huge buildup most of the time in the book to lead us to how we feel about them. And more often than not, it changes.
The show is still great, I still love it, it deserves all of it's praise. But knowing how I feel about certain people, so to hear someone else in the room watching who has never read and get an entirely different idea about them, well...I have a hard time with that. YOU SHOULD HATE THAT ASSHOLE FOR ALL THE SAME REASONS I DO AND NOT BECAUSE HE PAIRED ORANGE TROUSERS WITH A RED SHIRT.