The whole "it needs to be big" thing doesn"t have any basis in biology, according to studies I"ve read. They made up molds of a vagina and put variously sized penises in it and simulated a thrusting motion. It didn"t have any effect at all on where the ejaculate ended up or how quickly it got into the cervix. So long as the penis is big enough to touch the walls so the head can create a vacuum and the lip can pull out other guy"s sperm (yeah, that is why the penis is mushroom shaped, removing competing ejaculates). In fact, excessively large penises were detrimental because of the chances that a particular mate would be afraid or injured, especially if they were particularly small, limiting your mating options.
Though come to think of it, assuming humans involved in similar groups, so small girls having sex with guys with big dicks would only happen across large areas, which is a relatively recent thing. Might explain something.
Oh, and if you feel bad about having a small penis just remember, it"ll be much easier to convince a girl to take it in the ass. You may get less pussy, but more ass.