But your body is composed of mostly caustic chemicals.
Accurately speaking, it's not, most of your body is chemically neutral... the only caustic bit is the stomach acid in your guts.
Compounds can be irritating to mucous membranes 4 seperate ways:
1.
Mechanical irritation: this is when you have chunks/shards/"crystals" of hard material that can cause irritation when you snort it or rub it on your gums, simply by acting like the grit on sandpaper and mechanically wearing away your mucous membranes - if enough pressure is applied, and before it dissolves. Finer powders will have much less mechanical irritation effect, but they will dissolve faster so can potentially have stronger burns via the other factors. Solutions of dissolved compounds do not usually have this factor to worry about.
2.
Acidity: This is simple, it's a measure of how acid (or contrarily, how basic) the compound is. Anything outside of a pretty small pH rnage centered on 7 will sting. Too high or too low a pH will cause permanent damage and extreme pain. In commercial drug formulations, small amounts of sodium hydroxide and/or hydrochloric acid are used to help adjust the pH of drug solutions to around 7.
Most drug salts sold these days are slghtly acidic when dissolved in solution, but there are some exceptions. Freebase drugs are, in general, basic (opposite of acidic) and are irritating in their own special way.
3.
Osmotic pressure. Osmosis is an effect where water will migrate across a permeable membrane from lower salt concentration to higher salt/dissolved solid concentration. If you've ever had a bath in the tub and had your fingers get all wrinkly, or had to rinse your mouth with concentrated salt water for a dental problem and noticed its strange feeling on the tissues in your mouth, you've experienced osmosis happening to you first-hand. (If you reverse the process, you can use pressure to separate water and salts from each other, forming Reverse Osmosis, a common form of water purification). Anyway, it just so happens that if you have solutions in water that don't match the same osmolarity (same concentration of dissolved ions), your mucous membrane cells (or blood cells in Iv usage, muscle tissue in IM usage) will either sell up or shrink down depending on how different the salt/solid concentration. This can cause your cells to be temporarily disrupted, and in extreme cases, burst and die. It's also why saline solution for injection is 0.9% sodium chloride - that's isotonic with blood.
4.
The drugs can also sometimes actually directly activate receptors that are responsible for causing pain or local irritation. For instance, morphine and other histamine-releasing opioids do this, by releasing histamine from mast cells along the path of the morphine. Hence the reason IV bolus morphine causes "pins and needles".