Is there a tree or something above the Buddleia? If not, inspect the Buddleia's branches and leaves (including backs of leaves) thoroughly for signs of weeping, soft spots (on the woody parts, including the 'trunk) as well as aphids or scale insect.

My first thought would be potato aphids, too. If you find them…

Whatever you do, don't use a systemic insecticide as you will kill and the bees, hoverflies, moth and butterflies that are feeding on the flower-nectar.

(Try washing them off with plain water.)