Developing breathing is not the purpose of the meditation, the purpose is to develop awarenesss.
If you are noticing the minds tendenency to drift when there is nothing for it to latch onto then that's good, that's the point where you need to train the mind, that's where you need to develop awareness.
So this is where you should put your effort now, make a point of maintaining awareness through the gap between breaths. Don't worry each time you fail, the next breath is a new opportunity.
"Whenever we feel that we are definitely right, so much so that we refuse to open up to anything or anybody else, right there we are wrong. It becomes wrong view. When suffering arises, where does it arise from? The cause is wrong view, the fruit of that being suffering. If it was right view it wouldn't cause suffering." - Ajahn Chah
"Remember you dont meditate to get anything, but to get rid of things. We do it, not with desire, but with letting go. If you want anything, you wont find it." - Ajahn Chah