It sounds like Verbascum bombyciferum, tall with yellow flowers and downy grey leaves? I wonder if there is a type of pest in the soil, eating at the roots, this would explain the wilt....though I'm not sure what it could be..........
Have a look at Figwort weevils......

This plant is not a perennial more biennial. I used to grow it. It looked very impressive and makes a great statement. Try one of the other mulleins if you want more permanence.(Verbascum olypicum) These self seed and you can pot em up and decide where to place, as they have a big footprint. The dried spires will last ages, if left in place. They come in various colours. Bombi can be hard to find in a lot of garden centres. I used to buy one each year to always have one in good fettle. Then I could't find it anywhere and it broke the chain.

Thankyou Janey and Dorjac. Very helpful answers. Maybe i will try something else in its place, to make a good show.