This is what we in the business refer to as a “meditation obstacle.”
It’s actually not uncommon to have experiences like this when meditating. I get this frequently when I do the body scan–my body will tense up where I am scanning, and when I notice, I have to consciously release that tension. I have never had it happen when following the breath, but I don’t see why it couldn’t happen. Just notice that it’s happening, and notice that it’s frustrating you, and let go of it, and keep looking for sensations. If your mind goes back to it ten times a second, just notice that that’s happening, accept that it’s happening, and keep going. Try to be amused rather than annoyed, if you can pull it off.
It may feel like just the opposite, but my experience is that anything like this that happens in meditation is a goldmine, because it’s so predictable, and you can be ready for it. It’s a great opportunity to practice not following sensory distractions. It may or may not stop, but either way, you will eventually stop being distracted by it.