Camellias are woodland plants, and as such require neutral to acid soil (a usual feature of such conditions), as you know, a sheltered site, and protection from the hot sun. Two important factors to take into consideration are 1) that they do not like the morning sun on frosted flowers (the buds and flowers mark and often rot and fall), therefore often the best site for them is north-west facing, and 2) they love a load of moisture and food in late June into July when the new buds form.
Possibly the reason for the bud and flower drop is as above (frost), or that the plants got dry, even at that time of year, and the plant thought to itself, "I can't support this lot, I'd better shed a few".
They can be grown in full sun, south or south-west facing, again, sheltered from the east, but careful attention must be paid to the water requirements, otherwise a common reaction is for buds and flowers to fall early.

It would do better in the ground if you could manage it really. They are, as previously said, sensitive to dryness at the roots. If you do put it in the ground and your soil isn't acidic then don't worry. dig a bigger hole as poss and use ericaceous compost mixed with the soil and back fill the plant. then give it an ericaceous feed [can buy it ready made as Rhododendron feed at the gc] in the late spring.
mine is treated this way as I garden on chalk and it does really well.