Skimmias like acid conditions - I know the soil is full of bits of chalk and flint in Kent, which leads me to think it might be alkaline. If all the leaves look sickly pale green or yellow, then it might be chlorosis from the alkalinity, so get some Sequestrene Iron tonic, mix according to the instructions and water round the base of the plant. You will have to do this every six months if alkalinity is the problem, or grow it in a pot in ericaceous compost instead.
If your soil grows rhododendrons, pieris and Camellia successfully though, that might not be the trouble - if the Skimmia is in full sun, that can also cause yellowish leaves, it prefers dappled shady conditions.