Dunno about the plums but everything is rather earlier this year isn't it? I don't think my autumn rasps will be long either.

Take off a couple of plums and cut them in half, see if there's a larvae inside, or whether there has obviously been a larvae inside. You may have had a plum moth infestation, and the fruits have then succumbed to brown rot. Alternatively, Plum sawfly tends to attack the fruitlets rather than mature fruits, so if your fruits are/were still small when the problem started, could be that. Damaged fruits from that pest have a round hole bored in them, from which black excrement pellets may protrude, but if you didn't notice the damage, brown rot probably then started - this is usually a secondary fungal infection which sets in after damage to some of the fruits.

If you decide its plum moth, pheremone traps are available for these to protect the crop during next year.