Goes with the wood. The whole willow family, including the true poplars, are fuzzy and fine-fibered. Sand along the grain after a P400, then dampen the piece, let dry, and sand again. Sanding round-and-round in either or both directions won't make a bit of difference.
If it's still giving you the wrong feel, use thinned shellac or lacquer as a sanding sealer as suggested, where you sand the finish rather than just the wood.
The good news is that the grain reversals that give you the fuzzies can be quite attractive with the proper finish.