Travis Fimmel gets a lot better as time goes on, so I'd stay stick with it if he was your main gripe with the early seasons.* The story branches out really well from him, and a lot of the old characters end up stepping their game up a lot later on (Floki stays awesome, Rollo stays kinda awesome, Lagertha, etc.)
*Saying this more in general to anyone that decides to the pick up the show. Know a few people that said they were curious but were also given pause because the lead dude was 'the Warcraft guy.'