well any time you want. if you want max flavour leave until they ripen [usually red].

Always best to leave them to ripen to red, though you could pick a few and taste them to see how hot they have become.
Chilli plants usually produce masses of fruits at the end of the season. Don't forget you can put them in a big jar with oil to make chilli oil, or allow them to dry, and then put them in an electric mill to make chilli powder. Or you can just keep the dried chillies to use all winter.
Use the seed from your dried chillies to start new plants next year.
We were complaining to some Indian guests recently that the Scotch bonnet ones we grow are too hot to use, and their tip was to put the whole chilli into what we are cooking and then remove it after a while, rather than chopping it up. Just a tip if you like things hot but not scalding.