I don't know if this helps, but there were white stones like this on my gran's grave ... and every year my Dad would wash them in a bucket containing bleach, then rinse them well and put them back on the grave. They came up nice and clean :)

thats what i would suggest too. or blast with a power washer.

I had a load of Cotswold Buff covering my front garden several years ago, within similarly coloured cast-concrete retaining brick walls. It was in full sun, with well-spaced large pots of specimen plants, no should have had no shade/algae problems.
Unfortunately mine went the same as yours, which was glaringly unpleasant against the still-golden walling. Nothing I could find would bring them back to looking lovely, and in the end I gave up, ripped it all out and planted trees and hellebores instead!
I do have a decent-sized stretch of slate chippings in the back garden which is a bit more shady and damp, and they have not shown any deterioration in appearance over time. Every 2-3 years when I top them up, I do have to run the hose over them a bit because of the whitish slate dust you get in the bags, otherwise you get new-looking patches which from the house looks like a bad skin infection...
And don't forget to cut off one of the labels on the bags of any new stones you buy, so you can match them next time, it has saved me a costly and embarrassing mistake in the past!

Thanks Hywel, I have the same problem. Something else to note on the calendar for warmer days.

Many thanks for all those who have replied to me - always appreciated