Can I add builders sand to the ground?. Will it harm the vegetables?

If you already have builders (fine or soft) sand then this will be fine to use although if you are buying fresh sand in for the job then concreting (coarse) sand might be better. Sand sold for building/concreting is OK but you should not use beach sand as this has salt in it (and is probably illegal).

Be careful - some builders' sand DOES have salt in it! It will kill your plants, so make sure it's OK before you use it.

I tried using a bit of builders sand last year ina small, separate bed in case things went wrong. The plants were fine and I was pleasantly surprised how much it improved the clay.

Be careful as as mentioned it can contain salt. You can pick up horticultral sand which has been washed at your garden centre ( I'm sure you are aware ) silver or course. You could also pick up some grit to help break it up with the bag of peat.