You don't think philosophers have ever struggled with this question? So let's take this one step further.
Sue decides to go on a date. Does she pick a date at random? Just any ole guy would do? Or does she become highly selective in who she picks? Suppose the date goes well, she falls in love and gets married. Is she not making a vow to discriminate against all other men on earth for the rest of her life?
If we're going to make the claim that racism is morally wrong, then logically, Sue here would be doing something far worse. She's declaring her intention to discriminate against all men of all races for the rest of her life, except for one person. Is that immoral?
Alternatively, we can make this even simpler. If racism is indeed immoral, then logically, it would be just as immoral for Robert not to hand over all his money and property to randos on the internet. Since racism is indeed the natural byproduct of kin selection, and since resource scarcity exists, and society depends upon altruism which creates a free rider vulnerability, then logically we must conclude that the opposite of the grave evil of racism would be to maximize altruism. Therefore, Robert should give away all his money and property to complete strangers, completely ignoring the harm it would cause himself and his family.
Now let's suppose everyone followed that same exercise that Robert is engaged in. What would we call that? We'd call it communism. And under a communist system people die, they starve to death and are reduced to extreme poverty. You cannot eliminate racism and still have personal property. You can either have one or the either.
Altruism can only arise if it is beneficial to one's own reproductive fitness. John can either spread his genes by having lots of babies, but Bob could also spread his genes by sacrificing his life to save his sister's life allowing her to live on and have her own babies, genes that Bob share in common with his sister.
If you detach Kin Selection from altruism, then what happens is that by instinct human being switch from one survival mode to another. In a homogenous society, where everyone shares a common ancestry and lineage, then you can get everyone sacrificing for one another. But if a society switches to a multicultural one, then eventually this sacrifice stops happening and the rates of altruism plummets. Eventually, everyone switches from a high trust society to a low trust society. They go into survival mode.
In a low trust society, you might see things like businessmen destroying the companies they work for so it doesn't survive so they can strip as much wealth from it as possible for themselves. You might see politicians engaged in rampant corruption, where institutions harm the population so they can enrich themselves so their own families can be wealthy, not caring that it impoverishes society as a whole. You might see cops refusing to uphold the law and instead take bribes from criminals to help them commit crime instead.
You cannot just wish altruism into existence. Either you foster the conditions by which it can flourish, or you can destroy it. Racism is an evolutionary cheat code to maintain a society. Get rid of racism, and the bonds that hold that society together also crumbles. It becomes everyone fending for themselves. And when everyone fends for themselves, the overall survival rate plummets. Evolutionary fitness declines.
Any moral code that naturally produces death is not a moral code at all. You cannot claim something is immoral if it directly caused people to die. Racism therefore can not be immoral.