You can't I'm afraid - they'll pass after a week or three, but in the meantime, just twist them off and bin them if you don't like the look of them.

Increasing the complexity of your soil ecology, say with compost and/or composttea will reduce their numbers, but probably won't eliminate them. All of the chemicals that used to be recommended for mushrooms or toadstools have been taken off the market, for good reasons.