It could matter only in certain kinds of baking, in which case you should be working by weight anyway.
For example, if you were making a sabayon-based sauce, e.g. hollandaise, and you got a double-yolked egg, you'd just adjust the quantity of fat when you beat it in at the end based on how thick you wanted the sauce to be. It's not that precise. But if you're making some sort of pastry that has to be just so, you need to be working by weight because of the flour and especially because of the very hygroscopic ingredients (salt, sugar, honey, etc.), and in that case it makes no difference whether the yolks were large or small to begin with.