I have not gone for a giant phone yet (although my G2 is pretty big) but I know tons of people who have Notes and it seems like people are a little taken aback by the size when they first get them but once they get used to the bigger screen they don't go back.
Yeah after getting used to my Note 2, I could never go much smaller. Smallest I would go would be to drop down to an S5 size maybe. I use my phone 95% of the time as a tablet, and maybe 5% of the time as a phone, so I figure I may as well get something that is more comfortable to use like a tablet I can carry with me everywhere yet be portable enough to fit in my pocket, which means bigger screen is the most important feature to me. And the Note 2 always has fit in my front pocket just fine. In fact, I have had it in my front pocket every single day for the last 2 years with no screen protector or case, and there is not a scratch on it, nor is it bent, cracked, chipped, etc. When I look at it now and it doesn't even seem that big to me compared to when I first got it.
I plan on preordering a Note 4 in the next few days actually, due to the $200 off promotions going on. But I am a little leery now with the aluminum border, incase of bending like the iPhone 6+. Plastic flexes, aluminum doesn't, it bends, but I am thinking the extra thickness should prevent this. I would wait for the Note Edge but I have a feeling that the side screen is just going to be annoying after a while. Probably cool to show off at first but probably pointless after a while. I can slide down the notification bar or use the multitask button if I need to access other stuff anyways. If I was paying full price for the Note 4 I'd wait, but assuming Note Edge will be $300, it's not worth the $200 extra.
And for anyone interested in the Note 4, Best Buy has a deal going on, where if you preorder a Note 4, and trade in any working/non-cracked smart-phone, they will give you a $200 giftcard that supposedly you can use on the purchase itself. So $100 Note 4 upgrades, or $499 off contract.
http://www.bestbuy.com/site/promo/sa...-note-4-128435
Samsung is also doing a promo, where if you preorder a Note 4, and trade in to the same retailer, a smartphone that is worth at least $10 credit, upon release, Samsung will cut you a check for the difference to equal $200 for tradein value. So if you trade in an old Razr or something, get $15, Samsung will give you a check for $185 to equal you to $200.
https://samsunggalaxypromotions.com/Note4/