It definitely isn't an olive tree of any description, but its hard to tell what it is from the photos. Can you add another photo showing the whole plant so we can see the growth habit please, and have you ever seen it flower? If so, what were the flowers like?

These trees like well drained soil and if it is in a pot without adequate drainage holes or plugged ones you will see brown wilt in the leaves. An easy way to find out is to lift the plant out of the pot. If you see a wet rotting root ball then that was the problem. The cure is to replant in a pot with good drainage holes, use well draining potting soil and set the pot ON a flat plate with a pebble base. Why a flat plate? Because it will keep you from watering too much...if you do, the water will run over the plate and you do not want that to happen. PS Bambo is an expert in plant ID it may not be an olive but the effects of overwatering house plants and the remedy is the same.

Looks more like a Fig Tree house plant?