Depending on what conversion chart you use, 1mg hydromorphone is supposed to be roughly equal in analgesia to about 60mg codeine (in reality it's going to vary substantially from person to person). The OP didn't say how much codeine they were taking, but using that conversion you linked, if they were taking the maximum dose of Tylenol 3, which is 10-12 tablets in 24 hours (not that that is wise to do chronically, even the prescribing information says you should not take Tylenol 3 for more than 5 days in a row, it's not meant for chronic use), at 30mg codeine per tablet that would be 300-360mg of codeine per day (and 3000-3600mg of APAP), so the conversion would work out to around 2-3 of the 3mg Hydromorph Contin per day.
If the OP is just taking lower doses of Tylenol 3 only as needed then that would be different.
BUT, I don't think that people were saying it was a big jump because they feared it would kill him. There is more to consider than just potency, it's not like all opioids are the same and the only difference is how many mg you have to take. 360mg oral codeine is listed as being of equivalent analgesia to ~10-12mg pure IV heroin, one wouldn't be surprised if anyone said it was a big jump to go from codeine to heroin, regardless of the dose. Obviously that's a more extreme example than switching to oral extended release hydromorphone, but just trying to explain that it's not like all opioids are the same aside from potency. There are plenty of opioids considered in between the ladder from Tylenol 3s and hydromorphone. If Tylenol 3 and Hydromorph Contin were the only choices, and constant opioid use was required then yes I'd go with the Hydromorph Contin. But in reality there are other meds in between. We also don't even know if the OP requires continuous pain relief.