Acetone is a bad solvent for a variety of reasons.
1. Acetone is miscible in water and is actually somewhat soluble in brine so it makes a bad extraction solvent. It won't separate out as cleanly and completely as something like heptane. Methyl ethyl ketone (2-butanone) is better if you need to use a polar ketone solvent because it has limited water solubility.
2. Acetone reacts with itself in the presence of acids or bases to make aldol condensation products which impart off colors and scents. Higher temperatures or long exposures to acid/base/dehydrating agents stronger than magnesium sulfate will encourage formation of these side products... and more water! So don't dry your acetone with molecular sieves or anything harsh like sodium metal.
3. Acetone is actually quite polar compared to something like xylene, hexane, diethyl ether etc. A better aprotic polar solvent for extractions is dichloromethane/chloroform. (either work good for mescaline) Limonene is also a good nonpolar solvent, but if you use that you must salt the mescaline out because limonene has a high boiling point and low volatility.
Sekio's tek: Lab grade dichloromethane is probably the most ideal, you can start with from boiled San Pedro cactus juice, add lye to pH 11, and then just extract repeatedly with dichloromethane which forms a layer on the bottom. Combine the dichloromethane extracts, wash with a little cold brine, and let the dichloromethane evaporate, you're left with crystals of mescaline carbonate. It absorbs CO2 and water from the air and forms a salt all on its own, isn't that great? Dichloromethane isn't good to inhale or drink in large quantities but it is non flammable and otherwise pretty safe. It should be easy enough to buy it from chemical suppliers.
You would have to describe what exactly happened during the process. I would imagine if you managed to get acetone to form a phase that separated out by using large amounts of salt or something it could work with reduced purity and yield. I think that if you didn't work quickly your acetone solution would come out colored from yellow to brown/black, would have a strong minty/camphor off smell, and may not pull as much of the alkaloids as you would expect - either by forming an emulsion or simply because the solvent does nto partition out effectively.
I know for a fact that extraction teks are not something you can just make random substitutions with. Specific chemicals are used for a reason. For instance replacing naphtha with Varsol/Stoddard solvent/petroleum distillate (yes they are both hydrocarbons) is an excellent way to waste mimosa root bark and end up with exactly nothing in the end.
Xylene is not even close to acetone. Both are flammable solvents but that is where the similarity ends. Acetone is quite polar and is non-aromatic, has a tendency to draw water from the air and is water-miscible itself. It also has a lower boiling point than xylene.
Xylene is not all that toxic. It takes a long time to evaporate though. If you want a food safe tek I would use limonene as your nonpolar solvent; extracted mescaline should still be of high purity. It really depends on the cactus you use, not the extraction method.
If you actually pulled some alkaloids and can check via TLC or GCMS or HPLC or something (I imagine the end dried result is going to be a sticky paste, gum, or resin with significant water and other plant compounds contaminating it) then there's no reason you can't re-extract the crude goop.
