I think now would be good? I was surprised that one I planted in a chilly not very sunny or sheltered position in poor soil is thriving after 2 years, really rocketing away. : )) All I know is they won't want to be moved, and don't last for ever - maybe over 10 or 12 years out of it and you're doing well.
Go for it.

Oh, and they don't like being trimmed hard either. Maybe not at all?