Rosebay willowherb? If it has spikes of pink flowers I just hope the wind isn't blowing it towards your garden...

Me too, Derek.

PS All you can do really is learn to recognise the young seedlings and remove them before they get any bigger. They are brighter green than the mature ones in your photo and you wouldnd't think they were going to grow tall - but once you've noticed you'll recognise them in time to pull or hoe them to oblivion. But with those in the field it will be a never ending chore. because the seeds fly like dandelion seeds.
If you have access to he field and permission from the owner you could cut them down before they set seed. Pray that the wind blows the other way...

Thanks a lot but until they flower, I’m not sure that’s what they are. Meanwhile I’ve cut back a yard or so the far side of the fence.

Well could it conceivably be Eupatorium? The flowers will decide it.