This is the RHS advice, you'll need to move indoors if you get temps below -10˚C:

Plant in a well-drained mix of compost, such as loam-based John Innes No 3 with 20 percent added horticultural grit. You can place containers outdoors in summer and then move into a cold conservatory, porch or greenhouse over winter.

Although they can cope with dry periods, olives in containers need regular watering and feeding to produce fruit. During the growing season keep the compost moist and feed with a balanced liquid fertiliser every month. In winter, you can reduce watering, but don’t let the compost from dry out completely.

Pot size should be as large as you can manage. Mine is in one about 20inches square. Make sure that you put the pot in position before you fill it with compost.