It's best to leave them on. I know what you mean, but it helps keep plant safer from frost and will give it a better chance next year. In winter the spent blooms can go lacy and look ok.

What I do is I trim off untidy looking dead flower heads now because it is mid-summer and there isn't going to be a problem but at the end of the flowering season when all is finished I then leave on the dead heads over winter because they do offer some protection and then prune next spring.

I do the same as Inverglen has suggested. In fact, there are a few coming off tomorrow!