If there is nothing precious groing in the gravel, then I would spray it with something like Pathclear or other systemic weedkiller twice a year, maybe. Not cheap, given the area of gravel you have, but a lot easier than weeding!

Hello and welcome to GOY that a lovely low maintenance garden you have made.
I presume you had a weed suppressant membrane put down before the gravel was laid?
If so the weeds are on top and are easy to pull out! but for you what I suggest, is that you get some one to come in and rake it over 1-2 times a year in the growing season this will lift any seeds/weeds trying to grow in the debri that one gets in stones. Or if you let the weeds grow a bit you can apply weed killer but I dont think that's the answer for you?
Around the fence I would use a weed killer being careful not to damage next doors garden.