You did not mention what brand/type of rifle or what stock.
One issue; unless the stock has been straight-lined by an expert, it is likely when the gun moves back from recoil, it is not moving exactly straight back. There is enough dwell time of the bullet in the barrel, that any slight difference in vertical or horizontal movement of the rifle, while the bullet is traveling down the barrel, will cause inaccuracy on the target.
Also, some rifles shoot better locked down, while other rifles shoot better using free recoil, likely due to differences in the vibration characteristics of different rifles. This is why testing, practice, and consistency are so important in rimfire bench shooting.
These are just a couple of my thoughts. I am no expert. There are others that use this forum that have forgotten more than I know.
--Mike McAllister