My jimmies are rustled, but it's not because I liked Robb or Cat, it's because the main driving force behind the whole show (the war for the iron throne) is now gone. Now we are just left with several side stories that, without the context of the war, all reach 'who cares?' level. Sam, girl and kid? Cripple Stark? Arya? Jon? Stannis and Melisandre? They all have zero impact on anything, so who cares? There are other forces (Danny, Mance, ice zombies), but all three are both monolithic and far away, so who cares? It's leaves us with King's Landing's soap opera, which has been great but lost 80% of its tension (no war preventing people to leave King's Landing and Joffery's asshatery behind, no war making the throne shaky, no war making treason in favor of a third party possible).
So yeah, the question for me is not 'Why should I keep watching if the characters I invest in are getting killed?' it's 'Why should I keep watching now that the story became almost totally uninteresting?'