Lack of pollination I'd imagine - they are self fertile and each flower can pollinate another, but they need movement of air to do it, so breezes and winds. You probably needed to hand pollinate.

I tap the stems to help pollinate just as you do with toms also it could be lack of feed = like any fruit plant they need to be fed

A moist bouyant atmosphere helps, in the greenhouse pollinations not a problem

Probably the door/windows are left open at times though Pam, allowing air movement from outside...

Thats true Bamboo, it has to be really bitter for me to close the window. ....
Had a robins nest on the shelves one year ?

Cause could be...

too low room temperature;

too moist compost (chillies like it on the dry side);

flowers not being fertilised (tickle every wide open flower with your fingertip);

too draughty. They hate cold draughts. if on a windowsill move the plant away from the windowsill as night falls because the air next to the window gets quite cool once the sun goes down.

They like heat and light.
As I suggest these tips, I am looking at my Carolina Reaper Chilli. It's in a 3" square black plastic pot, stands next to a table lamp (only on at night)on a low table in front of the South West facing window. It has four ripe fruits on it (3/4" x 1/2")