Since I'm not convinced Brienne goes to the quiet isle anymore in the show, then it makes sense when the director says he thinks the Hound is done, at least for now. I don't think they would bring Rory back in S5 only for a few scenes (where to the audience he's probably not supposed to be recognizable) digging dirt on quiet isle. I would be pretty shocked if they didn't introduce LSH at all though, I would think that would be a major story change.
I now also question if we're going to get White Harbor with Davos (frey pies) since Davos is now with Stannis in the North.