Of course it's feasible. It just takes time, money and effort like most things.
The best advice I can give you is to ground yourself in basic chemistry principles like molarity and functional groups, basic biology (if you're into the drug-effects side of things I strongly suggest learnign about neurons, receptors, the synapse, protien synthesis, cellular biology etc..) and then read as much as you can. Another important thing: don't force it. Read what's interesting to you and eventually you will accumulate knolwedge.
Vogel's organic chemistry textbook is a wonderful reference for synthetic processes. The Merck index or equivalent is an indespensible refrence for the physical and chemical properties of compounds. There are also no end of pages on Pubmed/Wikipedia/Chemspider on organic molecules too.
2/3 of O-chem is just the terminology and theory, I think. The other 1/3 is actual lab practice that you won't get without breaking glassware and burning reactions.
The reason it's generally considered "hard" to learn O-chem effectively is due to the fact that in some jurisdictions (e.g. Texas) actual practice of chemistry without a license is forbidden. In those which it's not outright illegal, lab equipment/glassware/reagents can be very expensive. And there's always local laws regarding venting, waste disposal, zoning etc.
Organic chemistry is really just cooking for drug nerds, with a bit of math and physics sprinkled on top.