You haven't mentioned what size plants you fancy, but a few low growers that will flower during this period are Iberis sempervirens, Helianthemums, Saponaria ocymoides, Chives, Thyme, Ajuga reptans. Larger plants, Cistus, Halimiocistus, and some Spiraeas flower May or June, depending on variety. Weigelas are either flowering now or just about to, along with Campanula portenschlagiana, a neat edger, and probably C. carpatica (if you like its sprawly appearance when flowering).

Whatever is in flower now at your local garden centre?! If you pick something perennial then it will resolve the problem for future years.
Or you could add structural plants such as ferns and hostas (which also flower later in the year)

Some Alliums about to start - chrisophii for instance. Also angelica archangelica, anthriscus sylvestris 'ravenswing', salvia mainacht just about to get going, lamprocapnos spectabilis, Solomon's seal, bluebells, clematis Montana, wisteria, tiarella. I've got all but the clematis and the wisteria going in my garden and I still feel like nothing is flowering! Was just bemoaning this when I saw your post and it made me think.

I went out and took 65 photos of things in flower in my garden after reading this. And I could have taken more, but I did not include any in pots.

Roses, Nepeta, Ceanothus, several kinds of iris. And chives make a nice front of border plant with their pink flowers too.

I would beware of chives on Bamboo's list unless you want a garden overrun by them. They are very pretty and useful but will seed themselves everywhere and, as they are a bulb, will be difficult to get rid of. Even if kept in a pot they are very profligate. I have real experience of this.
As for colour, if you think for next year, you can plant wallflowers in the autumn for spring colour. They smell nice too.
Strangely enough, when I studied my own garden I realised that were it not for the wallflowers there wouldn't be much colour except for the osteospermums I bought recently. However, I do have some lime green Heuchera intermixed with the same plant but with coral leaves and they do add a bright splash of colour. They look so nice I plan to buy more.
You can also try Veronica Gentianoides which has blue flowers on thin stems.

Well it probably depends how big your garden is, but there are loads of plants in flower right now, apart from the ones I've already mentioned, so here's a few more - Aquilegia, Myositis (forget me not), Cerastium tomentosum, Convulvulus cneorum, Ribes varieties, Deutzia, Choisya, Genista, Pieris, Rhododendron, Azalea, Syringa (lilacs) Aubretia, Phlox subulata, Scilla, Bergenia - shortly followed, at the end of the month and into June, by Callistemon, Geraniums, Geum, Pyrethrum, Foxglove, english marigolds (Calendula), Dianthus, Delphinium, Lupin, Veronica, Verbascum, Paeony, Roses, Philadelphus, Berberis darwinii and there are more...

Chives have never seeded around in our garden Arbuthnot - I suggested them too - they make a nice compact clump here.
How about aquilegias and geranium sanguinium? Just coming into flower here. If you can overwinter antirrhinums they will flower then too - we have a beautiful clump out just now. Omphaloides is nice if you have room. Its supposed to like a moist soil but it does OK here next to a Pieris Forest Flame and the colour comination is lovely. Erysimiums are still going strong,and Bowles Mauve flowers almost continuously through the year if its not too cold. Covered with flowers at present. Corydalis Flexuosa is just coming out too.

Thanks everyone, lots to go at there.