Is the less good side shadier or drier/wetter? I suspect one or both of these issues will be the main reason for less successful displays. Have used the same plants on both sides?
I'd also check the pH of the soil. Some times run off from the roof or the driveway can alter the soil. One bit of my garden is affected by the stuff hubby uses to wash the cars/bikes.

If you want to improve the soil add well rotted manure to help open up the texture/crumb of the soil. This will also help retain moisture and introduce beneficial soil microbes.

Perhaps a silly question but have you dog down deeply to discover if anything under the soil could be causing problem?

I meant dugof course...

Also, what did you mean by "topsoil", Adam? Some brands of topsoil are that in the simplest sense: dirt from the nearest farmer's field. That's good for filling holes, or as a basic component of a raised bed or terrace fill, but doesn't really improve soil texture, structure, or nutrient content.
Others are closer to what's called "planting mix", "soil conditiioner", or even "compost", being largely composted forest products, sometimes with a small mineral component, too. Those types will help improve poor soil, to varying degrees.

It is possible the soil has become contaminated at some point. I have a spot beside the back door of the garage where nothing will grow - I think the previous occupant cleaned his paintbrushes there. So I have a large flower pot stood there instead

It's something else! I don't think the soil is the issue, unless something catastrophic happened but you wouldn't need to ask the question if it did. I think it's the aspect (micro-climate). With my house, certain plants grow better on the north side which is cooler, damp & mostly shaded. Other plants do better on the south side which is warmer, has more sun and dries out faster. Yet other plants prefer the western side, rhododendrons, azaleas & hydrangeas for example. They like sun, but not too strong. It's a matter of the right plant in the right spot. If the ground isn't utterly barren, then don't worry about contamination.