Welcome to Goy. Get some slabs or sleepers or something to build a raised bed and plant alpines in it or better still make an alpine house with glass Your side being gravelled will give excellent drainage. You do not say whether it is north, south, east or west facing. What is their fence made of is it panels, wire or what and what height is it. If you just want something temporarily while you decide on the next move place a few pots along the gravel.

You can get large slabs that you lay on edge to hold up the soil. Or how about old sleepers?