I've not grown limes but we have lemons and have grown kumquats.
As with all these fruits, they are at their best if allowed to ripen fully on the tree/bush, so I would leave them until they are on the point of falling off at a slight touch.
They wouldn't rot on the plant unless conditions were exceptionally adverse.
Most citrus are picked very under ripe for transportation and storage, but growing them ourselves we can leave them until perfect.

when ripe the fruit are a darker green almost as dark as the leaf.

Brilliant answers, thanks very much!