While stones can be a nuisance, (the reason we have picturesque stone walls is to get the rocks out of the fields) they can serve a purpose to anchor plants in the ground when the wind blows. I move the worst and leave the rest.

I'd take out as many as I could - I don't know why this happens, but I've come across it many, many times in many gardens - somehow, the pebbles and rocks gather together and form a solid, impenetrable layer - the first you know is your bed's not draining properly, or things aren't growing well, and when you dig down, you think you've found a sewer pipe or something, and it turns out to be a solid mass of rock, pebbles and stone that have "stuck" together. I've experienced this so often I now remove whatever I find wherever possible - though I suspect this might only be a problem where the soil is clay (as it is here in this part of London, by and large).

We live on a free draining silty soil, Bamboo, and so my experience is exactly the opposite of yours :-).
Unless you are in a similar situation to Bamboo, Peaches, I would just remove the larger rocks and leave the rest to provide drainage. The tail of the roots of your prizewinning carrots and parsnips will love the easy drainage.

If you make a large sieve with battons and chicken wire, and set it up at an angle of 45 degrees, you can shovel your soil up against it and the stones will run down to the bottom. It is best to have a polythene sheet to catch the soil. Put the stones in the bottom of your raised bed, cover with landscape roll. Put the soil mixed with compost on top.

And of course one can always use the bigger stones to create areas in which to grow alpines!

You could become an expert in the sorts of plants that would actually like these conditions! Find a book on gravel, scree or rockery gardens for inspiration.