Is it only for some people because I have thrown a 2 and 4 seam fastball and yeah 4 seam doesn't move like the 2 seam but is there really a different in speed by a couple mph
It really has a lot to do with how you throw it. If you are just doing some subtle IFP (index finger pressure) with no real turnover then there may be almost no perceivable difference in velocity to home, but if you get aggressive with your finger pressure and really turn it over and it becomes more of a sinker there can be a couple mph difference. I am guessing arm slot and hand size could affect this as well.
Of the total force applied to the ball, any force used to spin the ball is force NOT used to accelerate it. So, the more you alter how you throw your 2-seam to create movement, the slower it will be as compared to your 4-seam.