True. But in Japan, they argue that it's largely a matter of weight, age, and fat content. You really should take it easy on the tuna, for example, which is wildly overfished anyway. But tai (sea bream) and other ultra-popular fish for sashimi are often quite small and low in fat; tai for example is rarely much over 2 feet long, and is a very light white-fleshed fish. Fish like this have neither the time nor the flesh structures for serious mercury exposure and absorption, and as a result you can eat them without much concern. Small blue fish like anchovies, smelts, and sardines are very fatty, but so small that the absorption is minimal. Salmon is almost never eaten as sashimi in Japan, because of parasites, but it is commonly thought that because it spends a good part of its life in running fresh water it probably has relatively little exposure, whereas tuna is a deep-sea fish and hits mercury all the time.
So there's one good reason to pass up the toro, now that we can't afford it. It's full of mercury. So you're not poor or cheap, you're saving your life!
As to fish compared with other things, it must be said that Mad Cow is pretty deadly, and nobody knows how much there is in the American herd, because it's illegal to test for it appropriately. Chicken is full of salmonella. Pork is probably safe, I suppose.
Personally, in America I don't buy even medium-sized fish, because it's always so awful unless you pay a fortune. I buy small "trash fish" from Chinese markets, and these guys are so small that the mercury is surely trivial.
(All of which should be ignored if you're pregnant or nursing, by the way, because this makes mercury deposit in the body in extremely dangerous ways from the baby's perspective. Do what your doctor tells you.)