Not only is there a risk to the bearing, but also to the paint I suspect, from oil burning off from within the bearing! Isn't there also a seal that will have to come out?
Update...I removed oil seal, retainer ring and pressed out the old bearing. Decided to buy a new bearing. It's a cheap part and figured...why not...I'm already in there. Also, it costs about $100 in gaskets and replacements every single time you remove the inner primary. So I'm replacing any "inspect for wear" parts.