Hi, welcome to GoY, your sandy soil would certainly meet the need for well draining soil which this plant needs, but it also needs to be fairly deep, and fertile.
Olive trees are frost hardy, so can withstand temperatures down to about minus 5 deg c, if you get lower temps than this you may need to use horticultural fleece for protection, or if you prefer, you can just pot into a larger pot using john innes no 2 with added grit, then if it gets too cold you can move your plant into shelter, Derek.