Yeah, I'd say you definitely can't tell just by looking. I think the more you talk to someone, the more you're able to gage how straight they are. Although, like SirTophamHat said, the only foolproof giveaway is when they get high with you, haha.
It's more than just avoiding drugs. You gotta be as touph and healthy as possible. It's a warrior cult.
Straight edge as a culture actually contains a lot of movements but these are the things most likely you will see to someone into the culture.