I think people may become tolerant to the drug itself, but also the experience. The last thing will indeed be permanent, the first thing not.
Your brain will recover when you stop using a substance for a long enough time (at least when you didn't fry your brain in the first place by using E every day for several years for example.) The thing that will stay is the memory of the experience. Before you ever used E you didn't know what to expect in detail. Everything you experienced the first time was special and amazing. After your first time using you will know what to expect, so it will surprise you less, and thus be less magical.
I think it's kind of like a brilliant movie: when you watch it the first time, it's amazing. When you watch it again, it's still amazing, but less. When you watch it everyday, it won't be any good any more. If you stop watching it for 2 years and then watch it again, it will be amazing again, but still less amazing than the first time you watched it.
How special your times on E after the first time using will be depends on the setting if you ask me. If you have a lot of great people around you, at a great festival or whatever rocks your boat, the experience will be enhanced. It will then still be possible to have amazing times on E. When it comes to the physical tolerance moderation is the key (my opinion.) Make sure your body has enough time to recover, and you will be able to enjoy the drug without a tolerance buildup.