Moving it very little isn't a problem at all. In fact, in Leo Mazzone's book Pitch Like a Pro, it explains that Maddux points his toes towards his target on every pitch (right for inside, left for outside). It's not noticeable to the naked eye, and it attributes to his crazy accuracy.
At the same time, if the direction of your foot is not your target, then you're typically throwing across the body. In that case, it's more bad
...just not as safe I guess.