You should have cleaned them with soap and water, then a wax & grease remover, then scuffed, clean again, adhesion promoter, then paint. No idea how the primer will hold up for you. It will probably be fine. If not, it's easy and cheap enough for you to do them over.