No, you don't need to worry, so long as you're not intending to eat them. There's a surprisingly long list of plants which are toxic if eaten - 52% of plants grown in the average British garden fall into that category.

You would be surprised how many everyday plants in your garden are poisonous. As Bamboo says we just don't eat them.

Not too good if horses are involved..

Not many people have horses in the garden though!