The fruits can fall off when there are wide differences in temperature. Are you opening the windows during the day and closing at night? From memory Lemon gets stressed when there is a 10c difference or so. That is what I think, it could also be water or nutrition but I do think it is temperature differences and that makes sense with our terrible summer!