If you have any bare earth in your small garden you could use the spent compost or potting soil as a mulch. Over time it will be no different from the rest of the soil. It is best not to reuse compost for potting up seedlings or young plants or for potting on. It may contain pests such as vine weevil eggs or plant diseases.

If you can use your compost from the flowers to grow your veg in and vice versa you may get away with it. You will need to feed the compost though as all the nutriments will have been used by this years plants. Be 'very' careful with compost from pots where you have grown any of the solanum family.

Your big risk is bugs in the compost that will overwinter and attach your plants the following year.

Thank you very much for your answers. I was hopping to get away with it but I will discard this years growing soil and start again next year.
Thanks.

I think you are being wise on this.