^Yes four weeks would definitely allow more time for neurological function to recalibrate, or in other words go through the gradual processes required for regaining normal levels of functioning. However, even in spite of your infrequency of use, more time would be needed for you to return to a pre-exposure state.
My main point though is that basically all recreational drugs are enchantingly euphoric in the beginning- during the "honeymoon" phase. But this all-fun-consequence-free stage is short-lived, and once it's over it's gone forever, regardless of whatever counter measures you take. Even if you stop for years and return to your normal pre-exposure physiological state, once you resume, yeah you'll get a few good highs at first (before you rapidly adjust), but it will never be the same as it was in the beginning "honeymoon" phase. It is for precisely this reason- seduced by this powerful nostalgia bound to those first magical highs- that we chase until we lose our bearings and end up in over our heads, ultimately disappointed, as well as dependent if not addicted.
If these four or five times are the only times you've ever taken amphetamine, then your euphoria window may not have shut just yet, though it's likely larger doses will be required. Also I suspect any other form of amp- Adderall, Dexedrine, etc would produce much more pronounced and positive effects. I've not taken lisdexamphetamine but I'm familiar with its pharmacological/pharmacokinetic properties; diet, metabolism, what is in your system (stomach acidity, alkalinity, fats and sugars, alcohol or other substances that exert significant impact on the liver/kidneys, etc) when you take it and anything consumed afterwards, and the like can drastically affect the metabolic process(es) necessary for the prodrug to become active, and additionally how well it is absorbed. I've also been told it sucks and that 70mg is vastly inferior to 20-30mg of Adderall.