there building a nest in your pot id imagine or they are milking aphids on the plant . they unless they take to much soil out to make there nest wont hurt your plant at all but the secondary problem ie the aphids will.there tiny little black insects under the leaves .if you have them just blast them of every time you water the plants with your hose . if theres a nest in ya pot youl have to repot or put the plant in the ground .

Better make sure they're not in the pot, in the compost - all that activity might mean there's an ants nest inside. Turn it out and have a look and in future, stand the pots on something like pot feet, old bits of roof tile, anything that raises them off the ground slightly, which will discourage them. If there is an ants nest in the pot, you need to get rid of it before they decide the roots of your plant are preventing their nest getting bigger - at which point they'll munch through them to make more room.

great minds think alike bamboo they say lol