The reason for your problem is your lathe and not you. There is an angular misalignment between the tailstock centerline and the spindle centerline. Unfortunately, there is no easy fix nor for that matter any easy way to determine how much angular misalignment exists.
It appears that most turners mistakenly assume that if the points match on a drive center and tailstock center that everything is hunky-dory. However, that check only tells you how much radial misalignment exists. It does not give you any clue about the amount of angular misalignment.
For lots of things, angular misalignment doesn't matter, but for a few things such as drilling on the lathe in which the drill chuck is rigidly mounted in either the headstock or tailstock, the angular misalignment can be an issue.
One other bit of machining advice regarding drilling on the lathe -- even though you see turners sometimes drilling from the tailstock with the work piece rotating, it is always better to drill from the headstock. The reason is that angular misalignment will cause binding due to side loading of the drill which leads to the hole being wallowed out larger than the drill diameter when the work piece is rotating and the drill is stationary. On the other hand, when drilling from the headstock and the work piece fixed, the hole size will be true and circular even though angular misalignment might mean that it is not always concentric with the exterior diameter.
The best solution in your case sounds like the earlier suggestion of drilling the hole and then mounting the workpiece on a mandrel that runs between centers.