You don't need to take it inside over the winters, they'll be fine outside and would prefer to be in the ground.

If the foliage is yellowing it could be that it's been in too dark a place or too dry - or, perhaps overwatering since it's been brought back out ?
If the former, try giving it as much water as it will take, making sure the drainage is good in that pot.
Think about planting it out too because they'll prefer that.

If its been in shelter all this time, you need to harden it off before leaving outside all day and night - should have left it out for a few hours during the day, moved it back inside, then a few more hours the next day, over a period of a week to ten days, until finally you leave it outside overnight.