Two schools on buffing. One buffs the wood, the other the finish. You're buffing the wood. One coat of oil, or maybe even four, isn't enough to put finish on the surface, and without resin to give it some backbone, it's going to buff away quickly anyway.
Since you're experimenting with buffing the wood, why not try a good burnish instead? With a good burnish you'll harden the surface, contract the pores and lay down the fuzz without gumming up anything. Burnishing on the lathe is one way to give the wood a glow, but the hardened surface is a bit reluctant to accept high viscosity treatments afterward. Cut your oil with turps or mineral spirits to get it to run in.