I should also mention that fighting is serious. Bad shit can happen, you can end up seriously injured, or you could seriously injure him and end up in jail. It sounds like it's arranged, and he probably won't let off on you unless you stand up to him.
Before the fight, let him know that he better lose. Say this with balls. Tell him if he, on some off chance, happens to get some bolo punch in there and win, then he will be in a whole new world of hurt. Tell him fuck that, no matter if he wins or loses, if he decides to make this confrontation physical and doesn't lay off the shit talking, that you will haunt him for the rest of his life. Tell him that if he decides to fight you today, then he will have to watch his ass every time he turns a corner, every time he leaves his house, every time he is alone at night, cause he will never know when you are going to strike. Let him know that you will be watching him. Let him know that you are not afraid to break his kneecaps with a baseball bat. Let him know that if he pursues this, he will end up in the hospital and crippled one way or another, either today or a month from now. Let him know that if he decides to fuck with you, he is fucking with somebody who can hold a grudge for a long, long time, and that the rest of his life will be spent regretting the day he ever messed with you. Let him know that you are not afraid to go to prison just to mess with his life. Let him know that if he ever lays his hands on you or talks shit again, he better be prepared to kill you because you will never stop.
I think you get the point. Act crazy. Be crazy. Shit like this has worked for me before when I've been ripped off before, and much bigger, tougher, and meaner motherfuckers than me have been scared off by this act. I don't know if it is an act or not, actually, I haven't been in a physical confrontation since I was in middle school, but I think if somebody really did lay their hands on me I would be so upset that I would have no choice but to make their life a living hell. Fortunately I know how to get my crazy eyes going and appear sincere with this shit, because I don't want to fight anybody. I like my teeth, and my unbroken nose, and my freedom.
Intimidation on that level is so much more satisfying than punching someone though. If you convince someone who is willing to fight that you are crazy enough, they will not want to fight you unless they are that crazy as well (so use your best judgment, of course. I wouldn't say this to a hood dealer who sold me some fake bag of heroin, I would just cut my losses, but if it is some punk suburban kid trying to act tough, then yeah, this is what I would do). Watching somebody back down from what they previously thought was a weaker opponent on sheer intimidation will bring you tears of joy for the rest of your life. It will make him a joke for the rest of his life. Fighting will just bring pain, and not necessarily end the actual situation. It will bring a lot of shame and humiliation to one of you, and one of you will have to choose whether to continue this fight for the rest of your life (or the amount of time before one of you moves away) or walk away in defeat. And considering this guy appears to be a shit talking narcissist who needs to put people down to keep himself up, if you lose he will talk shit about it to you and people you know forever.
Intimidation on that level, if you can get him to really believe that you are that crazy and vengeful, is the perfect solution to your scenario. Trust me, the look in his eyes when he realizes that you are a psycho and he just made a huge mistake will be priceless. Take whatever drugs you need to look crazy. Remain calm and look right into his eyes the entire time. Let your eyes tell him that he does not want to go down this road. And if he does, I guess you have to come through on your word. That way at least the next motherfucker will know what not to do.
