I'm confused - if you have plenty of buds, what happens to them? Do they drop off? A photograph of them might help in case they're not flower buds, merely new leaf buds - and I'm sure you're aware this particular Skimmia has small, pale green flowers, not the larger, dark red ones.

Do you need a female and male Bamboo?

Not that I'm aware of with this one, the flowers are hermaphrodite or unisexual, Drc