Yes of course you can, but it depends which kind of cacti you are speaking of. Some cannot take cold. If you check out my blog about on Fire Island, you will see the prickly pear cacti growing alongside the pines and deciduous trees and seasonal grasses.

Some, yes, a few year round, although winter wet can be an issue Others need bringing in for winter... there's a blog below that's worth reading on this subject

http://www.nodiggardener.co.uk/2015/09/growing-hardy-cacti-outside-in-uk.html

No true cacti will survive a normal British winter except perhaps in a very sheltered south or south west position, but they are not at all frost hardy. They cannot stand damp combined with cold, a combination that causes rot.
So even at 5 degrees the damp would finish them off unless you could give them some shelter and keep them dry.Even in the house or under glass they should only be watered once a month in winter. Bamboo please would you tell me any you know about that would survive outside?

Would having them outside from May to September be okay. My main reason is so they can get more sun.

Well, they'll need sharp drainage, a hot south or southwest exposure, and a shallow overhang might be helpful, too. Here's a short list of the hardiest cacti that I know of:
Prairie Prickly Pear (Platyopuntia macrorrhiza)
Eastern Prickly Pear (Platyopuntia compressa)
Hardy Pincushion (Coryphantha vivipara)
Beehive Cactus (Pediocactus simpsonii)
Claret Cup (Echinocereus triglochidiatus)
Hardy Cholla (Cylindropuntia whipplei)

Mine have been OK in those months but not if it turns out to be a very wet summer. They do need to dry out between waterings. Hywel keeps a lot of his outside all summer in SW Wales - you might send him a PM if you want more reassurance. You didn't say whereabouts you live and it can make a fair difference although our country is so small...

Steragram- that blog I posted a link to suggests some, but it seems they are planted in particular positions, where they get little rain exposure - have a read of it.

Bamboo thank you for the cacti link. It makes for a really interesting read. My OH took on the care of my daughters cactus collection when she went off to uni. I think he probably drowned most of them but over the years he has become more skilful. He will enjoy reading this during lockdown. It will make a change from watching repeats of old films on TV. NTS Greenside Garden in south of Glasgow bury pots of cactus outside in a scree bed for the summer months.

It's not the cold, it's the wetness. Poor drainage will kill a cactus faster than anything. They want a quick draining, loose, gritty medium.

Well Bamboo, that was interesting. I'm amazed that he has had such success with them up there in Yorkshire but noticed that he had them in extremely coarse poor growing medium (you can hardly call it soil!) I wouldn't risk it myself - and certainly wouldn't want to keep moving Opuntias about unless they were kept in pots the whole time(those glochids are horrible things). Opuntia microdasys was my very first cactus when I was about 10 so i do have a soft spot for it in spite of that!
But he does say he would bring them in during the worst weather . They should be OK from late April to mid Sept as long as the weather isn't very wet.