Oh my goodness.
Just found this HUGE list on the cats protection site.

All these are toxic to cats.

Acocanthera (flowers, fruit)
Aconite (also called Monkshood, Wolfsbane - leaves, flowers, roots)
Acorns (all parts)
Alfalfa (also called Lucerne - foliage)
Almond (seeds)
Aloe Vera (also called Burn Plant - sap)
Alsike Clover (foliage)
Amaryllis (also called Naked Lady - bulbs)
American Yew (also called Yew - needles, seeds, bark)
Amsinckia (also called Tarweed - all above ground, especially seeds)
Andromeda Japonica (all parts)
Angel's Trumpet (also called Chalice Vine, Datura, Trumpet Vine - all parts, especially seeds)
Angel's Wings (also called Elephant Ears - leaves, stems, roots)
Antherium (also called Flamingo Lily, Painter's Palette - leaves, stems, roots)
Apple (seeds)
Apple of Peru (also called Thornapple, Flowering Tolguacha - all parts, especially seeds)
Apricot (inner seed)
Arrowgrass (foliage)
Arrowhead Vine (also called Nepthytis, Tri-Leaf Wonder - leaves, stems, roots)
Asian Lily (Liliaceae - all parts)
Asparagus Fern (shoots, berries)
Australian Nut (all parts)
Autumn Crocus (also called Crocus - all parts)
Avocado (fruit, pit, leaves)
Azalea (all parts)
Baneberry (also called Doll's Eyes - foliage, red/white berries, roots)
Bayonet Plant (foliage, flowers)
Belladonna (all parts, especially black berries)
Bird of Paradise (seeds, fruit)
Bitter Cherry (seeds)
Bitter Nightshade (also called Climbing Nightshade, Bittersweet, European Bittersweet - all parts, especially berries)
Bittersweet (also called Bitter Nightshade, Climbing Nightshade, European Bittersweet - all parts, especially berries)
Black Locust (leaves, shoots, pods, seeds, inner bark)
Black Nightshade (also called Common Nightshade, Nightshade - unripe berries)
Bleeding Heart (foliage, roots)
Bloodroot (all parts)
Blue Flag (also called Flag, Fleur-de-lis, Iris - bulbs)
Blue-Green Algae (all parts)
Bluebonnet (also called Lupine, Quaker Bonnets - all parts)
Boston Ivy (leaves, berries)
Bouncing Bet (also called Soapwort - all parts)
Boxwood (all parts)
Brackenfern; Braken Fern (also called Brake Fern - all parts)
Brake Fern (also called Brakenfern, Braken Fern - all parts)
Branching Ivy (leaves, berries)
Buckeye (also called Ohio Buckey, Horse Chestnut - buds, nuts, leaves, bark, seedlings, honey)
Buckthorn (all parts)
Buddhist Pine (all parts)
Bulbs (all species in the families Amarylliaceae, Iridaceae, Liliaceae - bulbs)
Bull Nettle (also called Carolina Nettle, Horse Nettle - all parts)
Burn Plant (also called Aloe Vera - sap)
Buttercups (also called Crowfoot (new leaves, stems)
Caladium (all parts)
Caley Pea (all parts)
Calfkill (all parts)
Calla Lily (all parts)
Candelabra Cactus (also called False Cactus - leaves, stem, milky sap)
Carolina Horsenettle (also called Bull Nettle, Horse Nettle - all parts)
Carolina Jessamine (also called Yellow Jessamine, Yellow Jasmine - all parts)
Castor Oil Plant (also called Castor Bean - all parts, especially seeds)
Castor Bean (also called Castor Oil Plant - all parts, especially seeds)
Ceriman (also called Cut-leaf Philodendron, Fruit Salad Plant, Mexican Breadfruit, Split-leaf Philodendron, Swiss Cheese (leaves, stems, roots)
Chalice Vine (also called Angel's Trumpet, Trumpet Vine - all parts)
Charming Dieffenbachia (all parts)
Cherry (also called Bitter Cherry, Choke Cherry, Pin Cherry, Wild Black Cherry - all parts)
Cherry Laurel (foliage, flowers)
Chicks (all parts)
Chinaberry Tree (berries)
Chinese Evergreen (leaves, stems, roots)
Chinese Inkberry (also called Jessamine - fruit, sap)
Chinese Lantern (leaf, unripe fruit)
Choke Cherry (seeds, bark)
Christmas Flower (also called Christmas Plant, Easter Flower, Poinsettia - leaves, stem, milky sap)
Christmas Plant (also called Christmas Flower, Easter Flower, Poinsettia - leaves, stem, milky sap)
Christmas Rose (foliage, flowers)
Chrysanthemum (also called Feverfew, Mum - all parts)
Cineria (all parts)
Clematis (all parts)
Climbing Nightshade (also called Bitter Nightshade, Bittersweet, European Bittersweet - all parts)
Clover (also called Alsike Clover, Red Clover, White Clover - foliage)
Cocklebur (seeds, seedlings, burs)
Common Burdock (burs)
Common Nightshade (also called Black Nightshade, Nightshade - unripe berries)
Common Privet (foliage, berries)
Common Tansy (foliage, flowers)
Coral Plant (all parts)
Cordatum (all parts)
Corn Lily (also called False Hellebore, Western False Hellebore - all parts)
Corn Plant (also called Cornstalk Plant - all parts)
Cornflower (all parts)
Cornstalk Plant (also called Corn Plant - all parts)
Corydalis (leaves, stems, roots)
Cowslip (new leaves, stems)
Crab's Eye (also called Jequirity Bean, Precatory Bean, Rosary Pea - beans)
Creeping Charlie (all parts)
Crocus (also called Autumn Crocus - all parts)
Croton (foliage, shoots)
Crowfoot (also called Buttercup - new leaves, stems)
Crown of Thorns (all parts)
Cuban Laurel (all parts)
Cuckoo Pint (also called Lords and Ladies - all parts)
Cultivated Bleeding Heart (leaves, stems, roots)
Cultivated Larkspur (all parts)
Cutleaf Philodendron (also called Ceriman, Fruit Salad Plant, Mexican Breadfruit, Split-leaf Philodendron, Swiss Cheese Plant - leaves, stems, roots)
Cycads (all parts)
Cyclamen (foliage, flowers, stems)
Cypress Spurge (foliage, flowers, sap)
Daffodil (also called Jonquil, Narcissus - all parts)
Daphne (berries, bark, leaves)
Datura (all parts)
Day Lily (all parts)
Deadly Nightshade (also called Belladonna, Black Nightshade, Common Nightshade - foliage, unripe fruit, sprouts)
Death Camas (also called Amanita - all parts)
Death Cap Mushroom (all parts)
Delphinium (all parts)
Destroying Angel Mushroom (also called Amanita - all parts)
Devil's Backbone (also called Kalanchoe - leaves, stems)
Devil's Ivy (also called Golden Pothos, Pothos - all parts)
Devil's Trumpet (also called Datura - all parts)
Dieffenbachia (also call Dumb Cane - all parts)
Dogbane (leaves, stems, roots)
Doll's Eyes (also called Baneberry - foliage, red/white berries, roots)
Dracaena Palm (foliage)
Dragon Tree (foliage)
Dumbcane (also called Aroids - leaves, stems, roots)
Dutchman's Breeches (also called Staggerweed - leaves, stems, roots)
Dwarf Larkspur (also called Larkspur, Poisonweed - all parts)
Easter Flower (also called Christmas Flower, Christmas Plant, Poinsettia - leaves, stem, milky sap)
Easter Lily (leaves, stems, flowers, bulbs)
Eggplant (all parts but fruit)
Elaine (all parts)
Elderberry (all parts)
Elephant Ears (also called Angel's Wings - leaves, stems, roots)
Emerald Duke (also called Majesty, Philodendron, Red Princess - all parts)
Emerald Feather (also called Emerald Fern - all parts)
Emerald Fern (also called Emerald Feather - all parts)
English Ivy (leaves, berries)
English Yew (also called Yew - needles, seeds, bark)
Ergot (fungus on seed heads of grains and grasses)
Eucalyptus (all parts)
Euphorbia (foliage, flowers, sap)
European Bittersweet (also called Bitter Nightshade, Bittersweet, Climbing Nightshade - all parts)
Everlasting Pea (all parts)
False Cactus (also called Candelabra Cactus - leaves, stem, milky sap)
False Hellbore (also called Corn Lily, Western False Hellebore - all parts)
Feverfew (also called Chrysanthemum, Mum - leaves, stalks)
Ficus (sap, peel)
Fiddle-leaf Fig (all parts)
Fiddle-Leaf Philodendron (all parts)
Fiddleneck (also called Tarweed - all parts above ground)
Flag (also called Blue Flag, Fleur-de-lis, Iris - bulbs)
Flamingo Plant (all parts)
Flax (foliage)
Fleur-de-lis (also called Blue Flag, Flag, Iris - bulbs)
Florida Beauty (all parts)
Fly Agaric (also called Amanita - all parts)
Four O'Clock (all parts)
Foxglove (leaves, stems, flowers, seeds)
Foxtail Barley (also called Squirreltail Barley, Wild Barley - seedheads)
Fruit Salad Plant (also called Ceriman, Cut-leaf Philodendron, Mexican Breadfruit, Split-leaf Philodendron, Swiss Cheese Plant - leaves, stems, roots)
Gelsemium (foliage, flowers, berries, sap)
Geranium (all parts)
German Ivy (all parts above ground)
Ghost Weed (also called Snow on the Mountain - leaves, stem, milky sap)
Giant Dumbcane (also called Dieffenbachia - all parts)
Gill-Over-The-Ground (all parts)
Glacier Ivy (leaves, berries)
Gladiola (bulbs)
Glory Lily (all parts)
Gold Dieffenbachia (all parts)
Gold Dust Dracaena (foliage)
Golden Chain (also called Laburnum - flowers, seeds)
Golden Pothos (also called Devil's Ivy, Pothos - all parts)
Grapes (all parts; also see Raisins)
Green Dragon (also called Jack-in-the-Pulpit, Indian Turnip - leaves, stems, roots)
Green False Hellebore (also called Indian Poke, White Hellebore - all parts)
Green Gold Nephthysis (all parts)
Ground Ivy (all parts)
Groundsel (also called Ragwort, Tansy Ragwort - all parts above ground)
Hahn's Self-branching English Ivy (leaves, berries)
Heartleaf (also called Parlor Ivy, Philodendron - all parts)
Heartland Philodendron (also called Philodendron - all parts)
Heavenly Bamboo (all parts)
Hellebore (foliage, flowers)
Hemlock (also called Poison Hemlock - all parts)
Henbane (seeds)
Hens-and-Chicks (all parts)
Holly (berries)
Horse Nettle (also called Bull Nettle, Carolina Horsenettle - all parts)
Horse Chestnut (also called Buckeye, Ohio Buckeye - buds, nuts, leaves, bark, seedlings, honey)
Horsebrush (foliage)
Horsehead Philodendron (all parts)
Horsetail (also called Scouringrush - all parts)
Hurricane Plant (bulbs)
Hyacinth (bulbs, leaves, flowers)
Hydrangea (all parts)
Impatiens (also called Touch-me-not - all parts)
Indian Poke (also called Green False Hellebore, White Hellebore - all parts)
Indian Rubber Plant (all parts)
Indian Turnip (also called Green Dragon, Jack-in-the-Pulpit - leaves, stems, roots)
Inkberry (also called Pokeweed - all parts)
Iris (also called Blue Flag, Flag, Fleur-de-lis - bulbs)
Ivies (all species - leaves, berries)
Jack-in-the-Pulpit (also called Green Dragon, Indian Turnip - leaves, stems, roots)
Jamestown Weed (also called Jimsonweed - all parts)
Janet Craig Dracaena (foliage)
Japanese Show Lily (all parts)
Japanese Yew (also called Yew - needles, seeds, bark)
Jasmine (foliage, flowers, sap)
Jatropha (seeds, sap)
Java Bean (also called Lima Bean - uncooked beans)
Jequirity Bean (also called Crab's Eye, Precatory Bean, Rosary Pea - beans)
Jerusalem Cherry (all parts)
Jessamine (also called Chinese Inkberry - fruit, sap)
Jimson Weed (also called Jamestown Weed - all parts)
Johnson Grass (leaves, stems)
Jonquil (also called Daffodil, Narcissus - all parts)
Juniper (needles, stems, berries)
Kalanchoe (also called Devil's Backbone - leaves, stems)
Klamath Weed (also called St. Johnswort - all parts)
Laburnum (also called Golden Chain - flowers, seeds)
Lace Fern (all parts)
Lacy Tree Philodendron (all parts)
Lambkill (also called Sheep Laurel - all parts)
Lantana (also called Lantana Camara, Red Sage, West Indian Lantana, Yellow Sage - foliage, flowers, berries)
Lantana Camara (also called Red Sage, Yellow Sage - foliage, flowers, berries)
Larkspur (all parts)
Laurel (all parts)
Lilies (all species - all parts)
Lily-of-the-Valley (all parts)
Lima Bean (also called Java Bean - uncooked beans)
Locoweed (all parts)
Lords and Ladies (also called Cuckoo Pint - all parts)
Lucerne (also called Alfalfa - foliage)
Lupine (also called Bluebonnet, Quaker Bonnets - all parts)
Macadamia Nut (all parts)
Madagascar Dragon Tree (foliage)
Majesty (also called Emerald Duke, Philodendron, Red Princess - all parts)
Mandrake (also called Mayapple - all but ripe fruit)
Marble Queen (all parts)
Marigold (also called Marsh Marigold - new leaves, stems)
Marsh Marigold (also called Marigold - new leaves, stems)
Mauna Loa Peace Lily (also called Peace Lily - all parts)
Mayapple (also called Mandrake - all but ripe fruit)
Mescal Bean (also called Texas Mountain Laurel - all parts)
Mexican Breadfruit (also called Ceriman, Cut-leaf Philodendron, Fruit Salad Plant, Split-leaf Philodendron, Swiss Cheese Plant - leaves, stems, roots)
Mexican Poppy (also called Prickly Poppy - all parts)
Milk Bush (also called Euphorbia, Tinsel Tree - all parts)
Milkweed (leaves, stems, roots)
Milo (foliage)
Miniature Croton (foliage, shoots)
Mistletoe (all parts)
Mock Orange (fruit)
Monkshood (also called Aconite, Wolfsbane - leaves, flowers, roots)
Moonseed (berries)
Morning Glory (all parts)
Mother-in-Law Tongue (also calledSnake Plant - foliage)
Mountain Laurel (also called Lambkill, Sheep Laurel - all parts)
Mushrooms (also called Amanita, Death Cap, Destroying Angel, Fly Agaric, Panther Cap, Spring Amanita - all parts)
Nap-at-Noon (also called Snowdrop, Star of Bethlehem - all parts)
Narcissus (all parts)
Needlepoint Ivy (leaves, berries)
Nephthytis (also called Arrowhead Vine, Tri-Leaf Wonder - leaves, stems, roots)
Nightshade (also called Black Nightshade, Common Nightshade, Deadly Nightshade - berries)
Nutmeg (nut)
Oaks (buds, young shoots, sprouts, acorns)
Oleander (all parts)
Onion (all parts)
Orange Day Lily (all parts)
Panda (all parts)
Panther Cap Mushroom (also called Amanita - all parts)
Parlor Ivy (also called Heartleaf, Philodendron- all parts)
Peace Lily (also called Mauna Loa Peace Lily - all parts)
Peach (pits, wilting leaves)
Pennyroyal (foliage, flowers)
Peony (foliage, flowers)
Periwinkle (all parts)
Peyote (also called Mescal - buttons)
Philodendron (also called Heartland Philodendron - leaves, stems, roots)
Pie Plant (also called Rhubarb - leaves, uncooked stems)
Pimpernel (foliage, flowers, fruit)
Pin Cherry (seeds)
Pinks (all parts)
Plumosa Fern (all parts)
Poinsettia (also called Christmas Flower, Christmas Plant, Easter Flower - leaves, stem, milky sap)
Poison Hemlock (also called Hemlock - all parts)
Poison Ivy (all parts)
Poison Oak (all parts)
Poison Weed (also called Dwarf Lakspur, Larkspur, Delphinium - all parts)
Pokeweed (also called Inkberry - all parts)
Poppy (all parts)
Potato (sprouts, vines, unripe tubers)
Pothos (also called Devil's Ivy, Golden Pothos - all parts)
Precatory Bean (also called Crab's Eye, Jequirity Bean, Rosary Pea - beans)
Prickly Poppy (also called Mexican Poppy - all parts)
Primrose (all parts)
Privet (also called Common Privet - foliage, berries)
Quaker Bonnets (also called Lupine, Blue Bonnet - all parts)
Queensland Nut (all parts)
Ragwort (also called Groundsel, Tansy Ragwort - all parts above ground)
Raisins (also see Grapes)
Red Clover (foliage)
Red Emerald (all parts)
Red Lily (all parts)
Red Margined Dracaena (also called Straight Margined Dracaena - all parts)
Red Maple (leaves)
Red Princess (also called Emerald Duke, Majesty, Philodendron - all parts)
Red Sage (foliage, flowers, berries)
Red-Margined Dracaena (foliage)
Rhododendron (also called Azalea - all parts)
Rhubarb (also called Pie Plant - leaves, uncooked stems)
Ribbon Plant (foliage)
Richweed (also called White Snakeroot, White Sanicle - leaves, flowers, stems, roots)
Rosary Pea (also called Crab's Eye, Jequirity Bean, Precatory Bean - beans)
Rosemary (foliage)
Rubrum Lily (all parts)
Saddle Leaf (also called Philodendron - all parts)
Sago Palm (all parts)
Satin Pothos (all parts)
Schefflera (also called Philodendron - all parts)
Scotch Broom (all parts)
Scouringrush (also called Horsetail - all parts)
Senecio (all parts above ground)
Sensitive Fern (all parts)
Sheep Laurel (also called Lambkill - all parts)
Silver Queen (also called Chinese Evergreen - leaves, stems, roots)
Singletary Pea (all parts)
Skunk Cabbage (leaves, stems, roots)
Snake Plant (also called Mother-in-law's Tongue - all parts)
Snapdragon (foliage, flowers)
Snow on the Mountain (also called Ghost Weed - leaves, stem, milky sap)
Snowdrop (also called Nap-at-Noon, Star of Bethlehem - all parts)
Soapwort (also called Bouncing Bet - all parts)
Sorghum (foliage)
Spathiphyllum (also called Peace Lily - leaves, stems, flowers, bulbs)
Split-leaf Philodendron (also called Ceriman, Cut-leaf Philodendron, Fruit Salad Plant, Mexican Breadfruit, Swiss Cheese Plant - leaves, stems, roots)
Spotted Cowbane (also called Water Hemlock, Spotted Water Hemlock - all parts)
Spotted Dumb Cane (also called Dieffenbachia - all parts)
Spotted Water Hemlock (also called Spotted Cowbane, Water Hemlock - all parts)
Spring Amanita (also called Amanita - all parts)
Spurges (also called Euphorbia, Milk Bush, Tinsel Tree - all parts)
Squirrelcorn (leaves, stems, roots)
Squirreltail Barley (also called Foxtail Barley, Wild Barley - seedheads)
St. Johnswort (also called Klamath Weed - all parts)
Staggerweed (also called Bleeding Heart, Dutchman's Breeches - leaves, stems, roots
Star Jasmine (foliage, flowers)
Star of Bethlehem (also called Snowdrop, Nap-at-Noon - all parts)
Stargazer Lily (all parts)
Stinging Nettle (also called Wood Nettle - leaves, stems)
String of Pearls (all parts above ground)
Straight Margined Dracaena (also called Red Margined Dracaena - all parts)
Striped Dracaena (foliage)
Sudan Grass (all parts)
Sweet Cherry (seeds)
Sweet Pea (all parts)
Sweetheart Ivy (leaves, berries)
Swiss Cheese Plant (also called Ceriman, Cut-leaf Philodendron, Fruit Salad Plant, Mexican Breadfruit, Split-leaf Philodendron - leaves, stems, roots)
Syngonium (all parts)
Tangier Pea (all parts)
Tansy Ragwort (also called Grounsel, Ragwort - all parts above ground)
Taro Vine (leaves, stems, roots)
Tarweed (also called Amsinckia - all parts above ground)
Texas Mountain Laurel (also called Mescal Bean - all parts)
Thornapple (also called Apple of Peru, Flowering Tolguacha - all parts)
Tiger Lily (leaves, stems, flowers, bulbs)
Tinsel Tree (also called Euphorbia, Milk Bush - all parts)
Tobacco (leaves)
Tolguacha - flowering (also called Apple of Peru, Thornapple - all parts)
Tomato (foliage, vines, green fruit)
Touch-me-not (also called Impatiens - all parts)
Tree Philodendron (leaves, stems, roots)
Tri-Leaf Wonder (also called Arrowhead Vine, Nepthytis - leaves, stems, roots)
Trillium (foliage)
Tropic Snow Dieffenbachia (also called Dieffenbachia - all parts)
Trumpet Lily (all parts)
Trumpet Vine (also called Angel's Trumpet, Chalice Vine - all parts)
Tulip (bulbs)
Tung Oil Tree (all parts)
Umbrella Plant (all parts)
Variable Dieffenbachia (all parts)
Variegated Philodendron (all parts)
Variegated Wandering Jew (leaves)
Velvet Lupine (all parts)
Venus Flytrap (all parts)
Verbena (foliage, flowers)
Vinca Vine (all parts)
Virginia Creeper (sap)
Walnuts (hulls)
Wandering Jew (leaves)
Warneckei Dracaena (all parts)
Water Hemlock (also called Spotted Cowbane, Spotted Water Hemlock - all parts)
West Indian Lantana (foliage, flowers, berries)
White Clover (foliage)
White Hellebore (also called Green False Hellebore, Indian Poke - all parts)
White Sanicle (also called White Snakeroot, Richweed - leaves, flowers, stems, roots)
White Snakeroot (also called White Sanicle, Richweed - leaves, flowers, stems, roots)
Wild Barley (also called Foxtail Barley, Squirreltail Barley - seedheads)
Wild Black Cherry (seeds)
Wild Bleeding Heart (leaves, stems, roots)
Wisteria (also called Chinese Wisteria, Japanese Wisteria - seeds, pods)
Wolfsbane (also called Aconite, Monkshood - leaves, flowers, roots)
Wood Lily (all parts)
Wood Nettle (leaves, stems)
Yellow Jasmine (also called Carolina Jessamine, Yellow Jessamine - all parts)
Yellow Oleander (also called Yellow Be-Still Tree - all parts)
Yellow Sage (foliage, flowers, berries)
Yellow Star Thistle (foliage, flowers)
Yesterday, Today, Tomorrow (all parts)
Yews (needles, seeds, bark)
Yucca (all parts)

I think most animals have an instinct as to which plants to leave well alone,so wouldn't worry too much.I don't have a cat myself,but lots of members have,and will have a lot of these plants in their gardens.I think you will just have to decide if you want to grow them or not...

We have two siamese and grow some of the toxic plants. The cats have far more sense than to eat them. You are more likely to have problems with small children eating toxic plants. The only 'plant' the girls eat is field grass when they need to cleanse their stomachs

Thank you both.
I don't feel quite so guilty now LOL
I have a few off that list in my garden and the boys seem nice and healthy still.
I was beginning to think i'd have to give away my beloved azalea.

My cat was very picky about what she chewed on, so I never had any problems that way. The main trouble comes with bored indoor cats (or dogs) chewing on random houseplants.

Generally animals know what to avoid... it is us humans that eat things that are toxic!

Sometimes I wonder just how adaptive losing our sense of smell was!

Yup!