When trying to hit a size with a drill, you have to step drill it. Spot with a center drill, and then drill through with at least 2 sizes smaller. Then you drill with the size you want. Sort of a reaming operation with a drill. That will still leave you slightly oversize - a 1/8" drill will make about a .127 to .129 hole - but it should be straighter and smoother.
I use the rule of thumb that a drill size leaves a running fit hole. If you want a press fit, you go one size smaller. I rarely use reamers except on bores where I need a smooth hole.