It was quite cold when I set out on my way to work aboard the fixed gear bicycle. As I climbed, it became warm enough for me to take off my pants. So I did. When I know I'm riding, and I am not wearing some fancy pants (which, pretty much, I only do if it's to be a longish road ride) I wear the wool boxers. You can get some from your local bike shop, or you can just cut the legs off some long johns. Either works fine. And then, when you take off your pants everbody just assumes you have on cycling shorts.
But it feels like you are getting away with something.