Yikes. You definitely should not be eating there!
Japanese cooks normally begin every preparation of seafood with something called mizuarai, which means "water-washing." This does not just mean rinsing stuff, it means fish are rinsed, gutted, rinsed, scaled, rinsed, boned, and rinsed. It's very wasteful of water, and there have been discussions among major chefs about how to adapt the methodology to a changing world. Shellfish are generally rinsed somewhat less than this, but they're rinsed a lot.
In fact, some eaters from other places find this practice kind of irritating, because fresh, briny flavors get removed. It's not so easy to find a nice fresh briny oyster in Japan, to be honest, because they get rinsed so much.
So if you're getting worms on your oysters in a Japanese restaurant, that tells me a bunch of things:
1. They're not paying much attention to what they're serving,
2. They're not handling their seafood terribly well, and
3. They're not knowledgeable at the Basic Japanese Seafood Preparation 101 level.
I wouldn't go back. Ever. God only knows what else they're serving, and how.