The biggest problem with driving is most people think they are good drivers, but they aren't. They're the fuckheads that pull into the middle lane of a highway going 40 miles an hour, the dipshit in a SUV who thinks because they are in a SUV they can drive 80 mph in a blizzard with heavy snow accumulation on the highway, the left lane bandit, the assholes that don't understand zipper merge, the people who slow to 50 MPH when they see a cop even though the speed limit is 65 and cops usually don't give a shit unless you are pushing 80, the "offisher i am only buzzed, not drunk", etc. This is why we have tens of thousands of people die or get maimed every year in MVA.
I totally get that some people like driving, but like the earlier analogy with horses, people will still be able to self-drive, but they will be the minority and restricted as to where they can do that.