According to the history book that I read, there were alkaloids other than nicotine which were responsible for the hallucinations, but since this happened a couple of hundred years ago there is no way to be certain. GC/MS, NMR, FTIR and HPLC technology came along much later.
I think it's a shame we don't have any of the hallucinogenic tobacco cultivars left to sample, test, and assay. So, basically, who knows?
(The name of the book btw is Seeds of Wealth: Five Plants That Made Men Rich by Henry Hobhouse.)