Well if something like a panic attack can mess up your heart I would say its definitely not safe and you'd be playing with fire to an extent.
I know nothing of heart conditions, but based on some nasty experiences I've had I would imagine a panic attack of paranoia and anxiety which elevates your heart rate and blood pressure to high levels would be something that you woudln't want to have happen, and it can happen, and I'd wager some would say will happen eventually (maybe not your first time, maybe not ever, but the possibility is there).
What happens if you ever get extremely frightened?