It may be that they are after the honeydew secreted by sap-sucking pests. You could apply fruit-tree grease/bands to the trunk to keep the ants down but you should also check for aphids etc as they can lead to other problems.

Check for aphid or scale infestation - get rid of whichever it is of these, and the ants will disappear. They're only there to 'farm' the aphids/scale for the honeydew they produce, and won't harm your tree themselves.

unless you have them in pots and the ants have nested in them .