All I can think of is that maybe the sun is getting to it in the morning? that could cause the buds to shed..

Unfortunately, the conifer might be the problem. Camellias Needs plenty of water throughout the summer and into autumn as they're setting buds (but they don't like to be waterlogged) and yours will be in competition with the conifer for water. They will also be competing for the nutrients in the soil - but don't feed camellias after the end of July.
Also, there may be too much shade under the conifer it takes something pretty robust to compete with a full sized tree, especially conifer.

Both answers are accurate, but I think Hoya's got it, given the symptoms. Clearly it had enough water last year because flower initiation occurred and you've got buds. If the buds drop when you touch them, its likely to be a frost/sun problem - if the buds get frosted overnight, and the sun touches them before the frost has melted completely, that will cause this problem. If the sun doesn't hit them till the buds have defrosted, it's not an issue.

Thank you, all three. I've always thought the camellia was out of the early sun because of the fence but maybe I need to get up at crack of dawn to make sure! It does also get dry at times along by the fence under the conifer. If it is a question of water I can rectify it but I don't think I could move the plant successfully now and I don't want to sacrifice the conifer. I'm glad to have found out the probable causes, so thanks again!

It sounded as if it was protected from the early sun. Good luck with it, mulch now, feed in season and water plenty and I'm sure you'll have a brilliant display. Don't move it - a 20 year old camellia will go into an almighty sulk!

Take a look one morning around 9 am this week to see where the sun is. If the sun is hitting it then you just need to put a temporary screen (my mother used to hang a sheet on a washing line to keep the early sun off her camellias)

9 am, Urbanite? I'm wondering where you are, I'm on the outskirts of London and the sun was up and shining at 7.30ish this morning

Well, 7:30 is a bit late in the day for me, personally, but 9am is a perfectly civilised time to see whether the plant is likely to be getting the 'morning' sun at this time of year.