My main reason is that I have what appears like residual fat imediately below both of my nipples. Good grief you are very far from being the only man to suffer from this but the seriousness of it depends a great deal on your age about which you give no clue. However, your particular problem was addressed many years ago by the late great Steve Reeves, who advocated only incline chest exercises for the problem of saggy, fatty lower pectorals. Nipple pain Q.
You can also build up your chest muscles by doing a full-body exercise like rowing or swimming for at least 15 minutes. This is archive material from the MHF's malehealth.