I think someone must have used "Extra Lock" and that can be difficult to remove. The set screw will only pinch the tread to stop it getting undone, no thread-lock is needed. It should be a straight forward removal. If you broke the set screw you're going to need to drill it out and use an "easy-out" and heat, quite difficult though as the thick rudder stock will dissipate heat very fast for it to be effective. Otherwise drill the screw out and re-tap the hole, alternatively replace the nut.
Don't forget to support the rudder blade before you start undoing the nut, otherwise it will be damaged!