Dremel-sized fine rubberized abrasive drums/wheels, such as you can get online at WT Tool, will give you greater control and take away the stress. They can be easily shaped also as they wear rather rather quickly if used roughly.
I can't insert pictures from my computer here but the shaping I did to the Rosewood handle here:
was done with a fine Dremel sanding drum, still it was too rough and removed material too fast ( and I should have used some tape to protect the blade).
Look closely and you'll see that the complex radius at the front is not completely fair, but I could have done a much better job if I had switched to the fine abrasives after initial roughing.
You can also make fine sanding blocks out of skinny sticks, shaped any way efficacious for a particular task, to do finish-shaping work, using double-sided transfer tape or spray adhesive, etc, to mount the sandpaper. Kind of a like gross approximation of jeweler's files. You can use 80 grit paper here for shaping, it's not as though your abrasive is moving at hundreds or thousands of feet/min.