Just move your awareness back your meditation object. It's only a distraction because you make it so - in fact, it may be a sign of and calm mind a body.
I do meditation on the abdomen mahasi style, it's just the object that comes most naturally to me. I felt for some time that unless the abdomen was going in a smooth motion where the belly was going in and out it was going wrong. It annoyed me, a lot. To the point where when the abdomen got to the point of near disappearance I would get frustrated because I would then have nothing clear to focus on. Fixing posture helped, but ultimatly you have to realise there isn't a right or wrong way to breathe. Mindfulness in all states is what matters, as well as keeping well away from restless or distracted mind-states.
"If beings knew, as I know, the results of giving & sharing, they would not eat without having given, nor would the stain of miserliness overcome their minds. Even if it were their last bite, their last mouthful, they would not eat without having shared."