It would depend on your soil and how well drained the area is. If the soil stays damp and claggy then i would take the liner out. but if you are on a free-draining sandy soil then the liner - with holes in, could help to retain moisture. The same applies to the filling in. If soil is prone to wetness then get some drainage - bricks, gravel in, but if it is dry then fill with just soil. Be picky about where your soil comes from too. Quite often it is given away on 'Freecycle' or in local ads, but check where it comes from for the weeds etc. You don't want to import loads of bindweed, ground elder or horsetail!!! I would get a soil test kit as well and see how acid/alkaline the new soil is. Ideally you want neutral to just slightly acid. If it looks thin - add lots of compost, or well-rotted manure too. The local council compost can often be bought in large loads quite cheaply. Good luck!