2 Suggestions
1. Before you sharpen, clean your bit(s). Because of their wide surface contact area, Forstner's tend to load up on pitch, lignin and sugars quite easily. Even if you sharpen, the sticky film will smoke and drag in the hole. A little oven cleaner takes it right off, then water rinse and dry.
2. Consider using a spade bit instead. "They wander and don't cut straight," she said. Ah, all too true when the bit's off center. "Huh?" he said. If you carefully examine the grind and wings on many such bits, you'll find a number are not ground properly (even fresh from the factory) with the cutting wings a) with 1 spur point longer than the other, b) with one side at a different angle from the centerline, c) one side longer than the other putting the point and shaft off center in the hole, d) , one side sharper than the other, e) all of the above, any one of which will cause the bit to "wander as you wonder."
If you regrind a bit (part pun) the tool will cut straight without the hassels you have with forstners.
m