I think it's actually a lot worse for a black man to say it - but that was Ali's game plan, taunt them with the absolute diabolical to disturb their boxing, and a lot of the time it worked.
I'm going by the Thrilla in Manilla documentary. An outstanding documentary showcasing the fierce rivalry between Joe and Ali and the intensity of their 3rd encounter. It certainly throws another darker perspective onto his flamboyant persona that we're more familiar with, and quotes a lot of the racist aggression he launched on his opponents.
RIP all the same lol