Have you checked to see if there is any difference in the conditions for each pot? Eg does one side get more wind, more or less sun etc. Does one side get any drips from guttering etc? Has any animal made a habit of using one pot as a toilet? Are the Buxus in both pots doing well?

There doesn’t appear to be anything like that. Sun on both at the same time buxus seem to be ok in both pots. It really seems a mystery to me.

Have you got the same soil in each pot? Could you have a bug such as vine weevil in the failing pot? Would it be an idea to repot the buxus in the failing pot and start again with violas?

I put the same soil in both and have fertilised both pots the same. What I cannot understand is that when I moved the pot with the good violas and put it on the other side the plants started to deteriorate within a short time and the other plants that had been bad picked up within a few days.

The story you've told most definitely indicates something environmental,such as less sun or more wind on one side or any of the other things Steragram mentioned, along with its being a cold wind, or exposure to a car exhaust on a regular basis, or a fox or something brushing up against them that side because its where the fox always travels, anything like that, it's just you are not aware of the difference - the plants are out there 24/7, but you're not, so when you do have a look, its not necessarily easy for you to be aware there's a difference - but the plants are telling you there is.

Whatever it is seems to be acting at a fairly low level since the crown of the box seem unaffected. This might help you in your detective work