I think you're spot on. Even a strict meritocracy is ultimately unfair, since it's not really fair that some people are better at some things than other people.

Reminds me a lot of an article I wrote a few weeks ago, titled "There Are No Fair Fights" (https://medium.com/@nebulasaurus/there-are-no-fair-fights-c4ade3e4a4d8)