On the assumption that, otherwise, the plant is perfectly healthy and growing well, there are two possibilities - shortage of water at a critical point in bud/flower development, or, more likely, frost damage to the buds, causing them to abort. Although this plant is fully hardy and won't be killed by frost or cold, the flowers and buds are susceptible to frost, particularly if its flowered slightly earlier than usual. If your winter's been anything like the UK, that's probably the case, because its been relatively mild all winter up to February onwards.

In the ground or in a pot? If in a pot is it big enough?
One of my favourite little trees.

Also the buds can be affected by early morning sun burning off the frost. They go brown then. If yours in facing east then I'd try to move it if you can.