Probably Plum moth larvae - get a pheremone trap from the garden centre for this pest.

That will help next year. Not much you can do for this year's crop.

Yes they are a nuisance. I forgot to put the trap on the tree in May so had them again a few weeks ago. However the plums I am picking now are worm free, as they were last year, so hopefully yours will be too.