There are numerous toxic and not quite so toxic products you can spray on your plants, but all of them will kill other insects which are beneficial. Many will be taken up by the plant itself and ingested when you eat them.
That is why so many people believe in organic culture which concentrates on building up a healthy garden environment in which beneficial predatory insects exist alongside the pests, and help control them. For example, most wasps especially earlier in summer, will predate caterpillars, greenfly and other pests, yet how many people rush to get a spray to destroy a wasps nest, no matter how little problem it is causing.
I sympathise if you are talking about a massive infestation of caterpillars on a flowering bush or plant, but for most vegetable problems the most effective solution is still to hand pick any caterpillars you see, and more important, look under the leaves to crush the little clusters of yellow eggs which appear.

Fingers.

Yup fingers