When I serviced mine, I checked the tension too and it was way too high. The Selden method (lugagge scale + winch handle) works fine, but you must ensure you have the right length winch handle (or adjust the math if you don't). Also make sure the threads are well lubricated first or all you're measuring is the friction of the adjustment screw
If you've read the whole manual, you know the foil inside the mast is designed so it bends and lays against the inside of the gap when the sail is unfurled and powered up. This is how it is meant to work as that gives it a nice stable resting position that is supported the whole length of the mast (even better than a fancy batten car system).
The "tongue" of the next turn being squeezed out of the gap happens if the sail was not furled tightly enough. It also happens if you "unfurl" the sail by pulling the wrong end of the endless loop or by turning the winch at the mast - do not "help" in this way, all that does is make a tight furl loose from the inside, which ends with the same problem. When you do furl the sail, you must have a bit of tension on the outhaul so the sail gets a tight furl, which then prevents this from happening. This is in addition to keeping the leech tight enough (by applying some kicker/vang) as you've already found out.
On my boat the outhaul is on the left coachroof winch and the furling loop on the right, so I can furl by pulling with my right hand while easing the outhaul in my left with a turn or two around the left winch. Gives a nice tight furl and no problems.