The sap is toxic, yes. But over 50% of plants we commonly grow are poisonous if eaten.

I've got one growing in my front garden and they're no danger.
Unless you have grazing animals around you they're perfectly fine to have in your garden.
As Bamboo says ..... most garden plants are toxic if eaten.

Dogs occasionally try part of a leaf, get sick as a, well, dog, then never touch the stuff again. Horses, sheep, and small children are another matter, though I learned as a toddler that my neighbor's hedge was "icky", same as cacti are "pokey"!