The ants will be 'farming' aphids - they take them up the plants and eat the honeydew they exude. The damage is from the aphids and diseases they allow in via 'sap-sucking' . You can get ant killer chemicals, but if you are organic/animal friendly, then use peppermint. If you have mint growing - particularly black peppermint - put loads in a big bucket or bin with water for several days. Then water the juice onto the areas where the ants are, they will move! You could also try using peppermint essential oil - and sprinkle it around the area.

I agree with Hoya. Ants don't usually bother plants (except Leafcutter ants). Get rid of the aphids, the ants will go too.

Also check backs of leaves, stems and branches - the other thing that attracts ants is scale infestation.

I would get an ant bait containing spinosad, and hose down the leaves with a strong spray of water. Don't worry about knocking off leaves--if they are going to come off with a spray of water, there was no hope for them, anyway.