If we're not counting Caffeine, Alcohol, Cigarettes and Weed like you said then Nutmeg was the first substance I tried other than those in my young foolishness

I'm not sure whether to count it though since I didn't really ever get much of any effects from it other than a couple of times getting some mild ones.
If we count the first solidly active substance other than the ones mentioned above, then
Ketamine would be my first.
TL;DR story of my first time with Ketamine: I was hanging out with some friends from college at a park in the city skipping class, wanted to go pick up some bud but my main dealers weren't around. It was then we ran into another friend from college who came to sit with us and he asked if it was okay if he did a line of Ketamine there - we had no reason to object, and I remember laughing as he spilled the line on the ground and proceeded to snort it anyway from the concrete. It was this funny behaviour and his silliness that ensued that got me interested in Ketamine and finding out what it was all about - I'd read about it in the past but hadn't been particularly interested.
Eventually we sat down in an old parking garage which looked a right tip, there was broken glass everywhere, mud, shit, piss, spilled booze, and I thought he must have been pretty damn high to not mind the conditions of where we were sitting - I mean I've been in some bad places but I could barely hold my gag reflex in at that point. We sat down and a friend came over and sorted my weed situation, then at that point the guy with the Ketamine asked if anyone else wanted a line, I said I hadn't tried it and was interested, so he made a nice line for me.
I snorted it and waited, it seemed like ages was going by with no effects (contrary to all my future Ketamine experiences, where when snorted it would hit basically instantly), in this time the guy who gave me the Ketamine and his girlfriend proceeded to give me some tips, tell me what to expect, and tell me stuff like how to stop myself from holing if it was a bit strong etc, as well as warning me about the drip I was about to experience.
Sure enough, the drip hit moments later and the taste wasn't great but it wasn't really bad either, certainly not as bad as they'd made out, I remember it felt a little sandy going down the back of my throat. At this point I knew the effects would come soon. I was getting tired of sitting around near an ocean of piss though and decided to stand up and stretch my legs. At that point I remember saying something like "Woah" and being drawn to the lights like a moth.
I went on to smoke a couple joints with my friends and didn't take any more Ketamine that day but spent the next hour or two in a pleasant "bubble" as we used to call it, I still remember a few silly moments from that day like needing to piss and taking a piss in the middle of the pavement because I thought it was a bush, and nearly pissing on someone's kid. Oh and going into the pound store and joking that "If there's 2 drinks for a pound, then one must be free, since everything is a pound here" - to which one of the others proceeded to pinch a drink and sneak it right through the big detectors. Was a simple grab but being on Ketamine it felt like some Derren Brown shit.
Next up after that were
Cocaine,
Methylphenidate (Ritalin) and
Mephedrone, in that order.