I knew a teacher that when teaching beginners meditation said that it consisted of two steps.
Step one, stay on your object of concentration.
Step two, return to your object of concentration.
He said as long as you are doing those two steps (Stay and Return) you are doing it "right". So it sounds to me like you are doing ok.
“The truth knocks on the door and you say, "Go away, I'm looking for the truth," and so it goes away. Puzzling.” ― Robert M. Pirsig