But if a developer doesn't optimize specifically for the PS4, then that extra speed basically does nothing on a console. It's not like a PC where you can maybe up the resolution another notch because you have more horsepower, or up the anti-aliasing, shadows, or other graphical options. Console games rarely have options for those things. All that extra speed is going to do is up your game from running at 30FPS on a XB1 to maybe 35-40 FPS on your PS4. So no real noticable difference the vast majority of consumers. No developer will release a game that only runs at a shitty 20FPS on XB1, while it's silky smooth on a PS4. they'll downgrade graphic options to reach that minimum magical 30fps, and the PS4 version will end up getting those same lowered graphic options as well.
Now if next-gen console developers start giving us graphical options like PC games generally have, then the PS4 could actually stretch it's legs and show off that extra horsepower. But unless that happens, 95% of the time it'll just be going to waste.