Sounds like both your bearing balls are seized around the rudder shaft and turning in the housing. Their toroid shape will not allow them to drop out of the housing that way, so you will need considerable downward force to force them out of the seized bearings. Access probably prohibits any sort of relief cuts or holes to be made in the bearing ball - no choice but to bang on the top until it comes out.
Obviously everything needs to be removed from the rudder shaft first (steering quadrant, autopilot tiller arm and rudder position sensor in most cases). Then you bang on the top, with a sturdy piece of wood to protect the shaft top from the hammer.
The oil can swell up acetal, so you may have made it harder on yourself with that. Nothing to be done about it now. On some boats the top bearing housing can be unbolted, but since it comes out from the top, not the bottom, that wouldn't help you very much either.
So just apply sufficient force and fix whatever breaks (if anything) later - it needs to come out!