I think when people say it, it's because they either don't expect you to be that dressed up for the occassion in question/they're not used to seeing you in that sort of attire. And when I say "you" I mean all of "us"
Last summer I had a total revamp of my wardrobe. One day I just looked in there and was like "ugh, nothing in here grabs me anymore, it's gotta go". I then restocked with tonnes of boho dresses and so on (I'm inspired by Sienna Miller, Kate Moss and more recently a bit of the punkier stuff).
However my work wardrobe still consisted of my old smart gear that I'd only bought under a year before when I was having a bit of a fashion crisis. I look at those blouses and trousers now and think "oh dear. What WAS I thinking?!"
It's like in the past year I had a fashion epiphany haha! Don't get me wrong, I've always loved fashion, I used to pore over my Mum's only ever copy of Vogue that I ever saw sat in the house for years. But when it came to me, I just don't think I previously ever had the confidence to pull stuff off - like recently I bought a bright red jumpsuit - if you'd had told me a year ago I'd buy that, I'd have laughed at you. It's coming in the post and I still wonder if it will suit me but I've decided I'm gonna go out on a limb and be daring (that's what life's about - taking some risks!)
I only recently managed to get some money together to update my work wardrobe and since people have been making comments like "wow, new top?!" or "you've really been dressing up of late" which does make me feel cringey about how I must have came across before.
To me though, I feel more comfortable being dressed up in public than dressed "down". In the house I'm happy to wear my scruffies but outside I have to feel like I'm making an effort. I think for me though, dressing "up" is a part of what I enjoy. I love to look pretty but some girls manage to still look gorgeous dressed "down"!