The server's conduct was unacceptable. I hate saying this, but whether or not the oysters were problematic, "depends."
A "gold" worm isn't a worm at all, but a sign the oyster has recently eaten. A few red or green worms on gulf oysters, are probably harmless. However, they're a sign the oyster wasn't thoroughly cleaned before shipping and wasn't properly shucked and cleaned before plating. Most worms you see with oysters aren't actually dangerous and were on the outside of the shell, or in shell tubes or crevices until bad handling knocked them inside.
The worms should have been cleaned before the oysters were shipped. At the restaurant, the oysters should have been rinsed in cold water at the time of shucking for the specific purpose of removing worms and grit. I imagine the oysters were gritty, too.
Back to the server: "Normal" in fresh seafood -- except in the sense that they are normal in "worm infested fresh sea food" -- is BS. Although I suppose it would have been a bad sign to find old, listless, dessicated worms instead of young, plump active ones. The restaurant needs to retrain their shucker; and the server should have removed your oysters from the table, removed the order from your bill, and comped you with a different appetizer and a bottle of wine. The answer to, "Miss, there are worms in my food," isn't, "Ain't you never et oysters? That's normal, be-yotch." So much for "fine dining." You were punked.
You should call the the restaurant immediately and let them know you were served worm infested oysters and you aren't happy about it. And ask them how they dealt with the problem; and if they are just now becoming aware, how they plan to deal with it.