Now there's a person after my own heart, you're doing exactly what I'd do, and I agree, it is very satisfying drawing out long lengths of ghastly root. You will probably get regrowth next year though, and if that happens after you've already replanted, stick in bamboo canes, let it climb up that and then, when there's a good bit of growth, spray or a paint it with glyphosate (there's really no choice) but without getting it on surrounding plants. Left alone, it should travel back to the root and kill it, but its a constant battle with the stuff.
As for the Cicely, this seeds itself quite readily, so worth gathering seed from the spent flowers if you can, otherwise, you'll have to wait till autumn to dig it out and extract bindweed roots.

Remember, Bindweed doesn't grow in the lawn, so constant chopping off the tops, as soon as you see them, will weaken it. It's a long job but it can be controlled. Just as well that it's a small garden!! It also, probably will try to creep back in under next doors fence etc! So be wary of re-invasion!

Thanks so much to both of you for the helpful ideas!