I think it is near enough to cause you to worry about it as it grows. I had to remove a tree at a similar distance when it got tall simply because I was anxious about subsidence. It sounds as though you could still move yours a bit farther away.

Have seen them grow within metres of temples in asia - they are shallow rooted and 12 foot away is fine from direct root damage. They like pretty damp conditions - though am not sure that they are that thirsty that they would cause soil to dry out and cause damage that way. The roots stay pretty close to the tree.

Though normally I'd say no tree within 45 feet of a house, I wouldn't worry about a Gingko, they're so slow growing that, unless you're under 40, by the time it is a problem you'll be long gone.