Olive trees are evergreen, so shouldn't shed their leaves, though they might if they dry out in summer, and then regrow again when conditions improve.

I am not an Olive Tree authority but I have had one for about 4 years in a 12 inch pot. Like all evergreens there, is some renewal each year. Just been out and looked. Yes there are new leaves, and plenty of flower buds forming. Some old leaves a looking a bit cream crackered which will eventually drop off. To keep it under control I have pruned back some of the longer shoots/branches last autumn.