Knowing the height and spread of whatever plant you've got would be very useful in order to answer properly, but if you're calling it an 'azalea' and not a 'rhododendron' then its likely it doesn't get too large, which means its root system is less likely to cause a problem with a wall. Bear in mind, though, that no plant should be planted closer than a minimum of one foot away from any wall or fence behind it, and preferably 18 inches away.

Azaleas & Rhododendrons have very fine & shallow surface roots. That's why they dry out quickly & the ground around them must be kept evenly moist - their roots don't go down deep for water. The good news for you is that they won't damage your brick work, but I'd still give them some space to spread - a foot or two.