The major concern isn't just the Teflon IN your pan, it's the chemicals that have shown up in the atmosphere as a result of Teflon manufacture. I don't own any Teflon cooking equipment (I mainly use cast iron), but I won't refuse to eat food cooked in a Teflon skillet either. The way I see it is that it hasn't killed me yet, so I'm not going to worry too much about it; there are other aspects of Teflon that I don't like personally.
Here are some "tin foil hat" style talks about Teflon; there's a lot of truth behind them, but sometimes it's tough to see for all the anti-modernity hysterics.The Silent Killer, by Joanie DossTeflon linked to birth defects and illness; but is it safe to use in cooking?
Don't know about the silicone, though... I use the high temperature-rated ones for high temperature stuff, and I use the white, non-high-temperature rated ones for everything else. The older style, semi-flexible non-temperature-rated ones are nicer, but I'm not sure if they're in danger of breaking down or causing me any harm.
There's a point (for me) where the convenience factor does outweigh most of the risks, given the fact that we're not talking about risks of immediate, grievous bodily harm. It all depends on your perspective.