Seems a bit mysterious - if rhododendrons and foxgloves abounded there, clearly there wasn't anything wrong with the soil then, so unless you've added something toxic, or accidentally spilled something like deisel or sodium chlorate there, or used any long lasting weed treatment, I can only conclude that you must be choosing plants which don't like the conditions, and which possibly aren't kept sufficiently watered. If rhododendrons grew happily there, I assume its shady?

Try NOT putting manure and well rotted compost in any holes you prepare, but instead, cultivate the ground generally and mix in the compost as you go. Too rich a feed can also be damaging to plants. As B says, if plants grew there before, they will grow there again, and I expect the soil is on the acidic side. If it's shady, try hydrangeas, and deciduous azaleas, leucothoe, skimmias, ferns, and where its sunnier, what about blueberries, to enhance your breakfast cereal! Phil J