I paid off my mortgage as quickly as I could back in the 2005-2009 time frame by raising my weekly payments 15% a year and prepayments of up to 15% a year. I didn't have a huge mortgage, it was around 180k I believe. In hindsight I regret not investing that money, although given what happened in 2008, a good chunk of that money would have taken a pretty big hit. Roughly half was paid off before the market bottomed, the other half after. So I'd probably have come out in pretty good shape. I think my rate at the time was 4.25%, so it's not like I'd have needed crazy returns to be ahead. I sometimes kick around the idea of taking out money on my home line of credit and investing it. I think I'm approved for up to 225k or something like that, and the rate is around 3-4%, so again I wouldn't have to get amazing returns to be further ahead. But on the other hand, even if I manage to make a 2-3% premium that's not really going to make a significant difference to my finances in the long run, so why take the risk?
That being said, it really is nice not having to worry about a weekly or monthly mortgage payment. But on the other hand, should I end up upgrading to a house or buying a vacation/recreation property down the road, no question that I'll be putting the least amount down I can if rates are still pretty reasonable. May as well use someone else's money if you can.