It's not stonework it might damage primarily, its the pointing. If the pointing and brickwork is in very good condition or new, it won't damage it - if the brick or stone is already crumbling or spalled, and the pointing needs some attention, then it can do so. Its the reason why its sometimes planted beside newly built houses, precisely because it won't cause damage, unlike ivy.