not if you dont want to. I had 2 withing 3ft of a boundary wall. they did no harm and they grwe for 50+ years before they died.

I agree with Seaburngirl. If it will overhang a neighbours garden and they object they would be within their rights to cut it back to the boundary. Depending on the wind direction that could make it unstable. The birch give dappled shade under its canopy so no other worries.