No you can't leave the fleece on all winter it will do the camellia no good at all. Personally I would not cover at all and let the shrub take its chances.

I'm afraid I agree with Moon Grower - if its too heavy to move, it must be a good size, so the most I'd do is bubble wrap the pot to try to insulate the roots. This assumes you do not have the plant standing in an east facing position, where early morning sun can reach the plant in winter - frost on buds which then get the sun on them before they have time to defrost naturally causes buds to drop off. If you do have it in the wrong place, move it somewhere so that it does not catch any possible sunshine in the morning.
Other than being in a pot, with the risk of the compost freezing in a seriously cold spell, these are hardy shrubs and should not need fleecing.

The most common reason for bud drop on a Camellia is the drying of the roots in the summer months!! you must make sure the root ball doesn't dry out in the summer months when the buds are forming. The best place to place a pot with a camellia is a West facing wall or fence preferably protected from the south as well that way it is protected from the morning sun, as long as it gets a little sun it should be fine.

The reputation Camellia has for being a shade lover isn't entirely accurate - they don't mind full sun, but, as I described, the buds will fall if early morning sun can hit them before the frost has time to disappear.

No Camellias grow quite happily in the sun! and the only problem you would have is scorching with the morning dew on leaves and flowers ! and a very dry pot if not regularly watered which in the summer months would stop the production of buds.