A Storm of Swords is considered the pinnacle of the books for a reason.
The story just needs to build up again. So many powerful characters have all been killed off, it's not like you can match that pace in every book.
Book 4/5 feels like the story goes sideways instead of forwards because it has to. But that's largely irrelevant on the tv show because we're already getting some of the stuff it fills in, and all time it spends on giving us new POVs of existing characters is meaningless on the show. Like the boatloads of Cercei that we suddenly get - annoying, paranoid, borderline psychotic Cercei. It doesn't exactly make for fun reading. We already hate the twat, hearing her thoughts is like being subjected to bee torture.
In other words, it'll play out better on the show.