These are likely to be fungus gnats and not harmful, but just irritating. I'd guess you have used some good compost in your planting mix as they usually feed on something decomposing in the soil.
You could just spray them with a pesticide, but I'd suggest you just put a mulch around the plants of shreddings or small shredded bark to cover the compost, which would deter them. I put a thick layer of grass cuttings round all my chilli plants and they do well (planted in the soil rather than a pot though).

If they are driving you nuts, a weak tea solution--about 1/4 strength--will get most of them without harming the peppers. Whoever thought of inviting the critters to tea?!