Perhaps a height tax would act as a "gateway" tax for a government, making taxes based on demographic characteristics seem natural and dangerously expanding the scope for government information collection and policy personalization. For instance, much the same analysis as we performed above could, in principle, be applied to characteristics such as skin color, gender, and physical attractiveness, each of which is a (relatively) inelastic characteristic that has been shown to affect economic outcomes. Even those who are comfortable with a height tax would likely be uncomfortable with a system of taxes tailored to so many personal characteristics. No matter how compelling the theory, the administrative burden and invasiveness of such a system may be too great. Moreover, democratic societies may have an interest in avoiding the taxation of specific groups as a matter of course to counter the majority’s temptation to tax minority groups.Exactly. Once we open the door to taxing on the basis of any personal characteristic that affects income, the potential for political exploitation of the tax system is limitless. Since there is no clear, objective way to quantify the advantages from having a particular height or skin color (Mankiw's paper is an approximation whose framework makes many debatable assumptions), a group's raw power in the political process will be the ultimate judge of "fairness," and this arrangement is unlikely to produce fair results at all. It will also lead our already troubled political institutions to spend even less time on issues of broad social importance, as it becomes more profitable to wrangle over the innumerable details of redistributive policy.*
I'm not convinced by the counterargument Mankiw's paper provides:
A counterargument to this concern is that modern tax systems already condition on a great deal of personal information, such as number of children, marital status, and personal disabilities, without conditioning on many others. To argue that a height tax would lead to an over-reaching tax policy while these conditional taxes do not, one would have to believe that a height tax would trigger a descent down a slippery slope for tax policy. It seems more natural to think that a height tax could be endorsed on its own merits while taxes based on gender, for instance, could be resisted for the reasons currently applicable.Why isn't this right? The core of any "slippery slope" argument has to be popular perception. If enough voters feel that height taxes are no more reasonable than taxes on race or birthplace, then enactment of a height tax will legitimize these other taxes in the public discourse, even if Mankiw has convinced himself that they should be viewed differently. And in this case, I think that the perception is clear: most people view height taxes as categorically different from income taxes or child credits. In fact, this is why Mankiw's paper provides such a disconcerting thought experiment in the first place.
Ironically, therefore, the "height tax" argument is flawed as a critique of utilitarianism precisely because so many people have difficulty understanding its reasoning. Since most voters won't appreciate its logical basis, and instead will view it as an open invitation to seek group-level favors through the political process, its overall impact will be negative. With these broader consequences in mind, prudent utilitarianism suggests that we avoid a height tax, and there's no contradiction at all.
*Yes, I realize that one could use the same arguments to oppose standard income-based redistributive taxation. The difference is that including personal qualities like height will vastly expand the purview of tax policy, resulting in adverse political consequences more serious than those caused by standard redistribution, while achieving far slimmer gains in return.