Yes, absolutely. Why wouldn't I?
If you eat nothing but pasta or grains, sure that is no good. But a slice of toast for breakfast? A bowl of rice or pasta for dinner? As long as you exercise portion control,which goes for everything else as well, then why would it not be?
I've had vegetarians make a similar argument to me against eating meat. It is the same principle, though. If you eat nothing but chicken fried steak you are going to look like Andre the Giant minus about 3 feet of height. Moderation in all things.