When do they drop off the tree, and do they all drop off? Some fruits get dropped around early June, that's normal, but these are usually immature fruits. Are your fruits mature when they drop? If they are, maybe you have a yellow plum variety and they're actually ripe and ready when they fall. If they're not mature when they fall, and you've inspected both the fruits and tree for signs of infestation or disease, then water shortage when the fruit is ripening may be the cause.