I wouldn't think that the worms (possibly brandlings) will do any harm but they will eventually run out of food in the soil in your pots. I would much rather sieve them out and return them to your compost bin to carry on breaking down the material.

Thanks Jimmy, I would love to be able to sieve it as you suggest, but my compost has always been very damp and .”unsieveable”, despite me never putting water etc in there. I thought that was normal ?

if you leave the compost out to dry out a little bit then you will be able to retrieve the worms. but as to your actual query; no they wont harm your plants.
mine is always on the damp side too but it does sit directly on the soil.

Worms don't like sunlight so if you spread the compost in a bright spot & cover maybe a third of it with an old compost bag or something most of the worms will move under cover. Might take a while but you can then separate into two piles.

Thanks very much guys, the problem is solved.