This is very common in summer, particularly when the pot has been kept on the dry side - it makes a perfect artificial ant colony home. The ants won't be feeding on the roots or causing any direct damage, but what they will do is cause large numbers of air channels throughout the pot, which will in turn allow the roots to dry out and prevent any water from penetrating the compost since it just drains straight out through the ant runs.

If you have just one pot with them in you could try simply soaking the whole pot in water (i.e immersing it completely in water - dunk it in a pond if you have one) for 30-60 mins. or so. If you have a more serious infestation - i.e. they are in several pots or in the lawn, for example, then you can use Nemasys No Ants, which is environmentally & wildlife/pet/children friendly:

http://www.ecocharlie.co.uk/garden-care/pest-and-disease-control/nemasys-no-ants.html

There are also a wide variety of ant-specific insecticides, but personally I would never use them in the garden as so many birds and other invertebrates feed on ants.

Turn the pot out and run away for a half an hour, lol, then go back, pick the plant up with its rootball and shove it in a container full of water. Now clear and clean the pot it was in (assuming its not potbound anyway) place half a folded jeycloth or some weed suppressing membrane and lay it in the bottom over the drainage holes, but not pushed into them, remove some of the compost from the now sodden, ant free rootball of the plant, replace in pot with fresh compost. Now the important bit - stand the pot on pot feet or bits of broken slate, anything to balance it upright and give it clearance beneath, so its not in contact with the ground. This will prevent further incursions by ants into any of your pots if you do it with all of them (though you won't need to turn or repot the others, just get them off the ground).