Here are some thoughts I'll share with you.
Don't use aluminum pop rivets, they're much to soft, and the origin of the problem in the first place. If using rivets, use 3/16" Monel or Stainless Steel.
Unless you want to completely disassemble the rail to clean underneath it, replace only one rivet at a time for ease of alignment and operation.
If using machine screws, #10-24 or #10-32 in stainless should be adequate.
One central issue is how to minimize the size of the rivet heads, screw heads, nuts, or rivet shanks inside and outside.
Small bumps under the rubber rail will hide easily, but inside you want to keep the fastener heads from sticking out and becoming meat hooks or ugly bumpkins.
I'd first try to back rivet the rail from the inside so only the heads of the rivets would show, but this would require some help from a second person and some exact length rivets that just barely make a strong bond with the aluminum molding rail on the outside. I'd also use a Monel or stainless washer to secure the fastener fully rather than just riveting them to the soft aluminum, providing the height of the completed package would fit neatly under the rubber insert. Technically, zinc chromate paste should be used at any dissimilar metal junction like this, but using a weather seal compound would take precedence and using both would be difficult.
If using machine screws, the nuts would have to go on the inside, and the screw length should be just long enough to set firmly into the nylon part of the NyLock nuts. The NyLocks are smooth enough that their finish should be acceptable as long as the ends of the screws don't protrude.
Be sure to seal the hole and whichever fastener you use up with a sealant, e.g., 3M 4200, 5200, or something equally waterproof and able to seal the shafts against the sides of the holes, and of course, the hollow openings of the rivet shafts if you use them.
Some experimentation will probably be necessary to find the optimum solution.
Another thought: the rubber rail frequently pops out when "kissing" the dock with a firm landing approach.
I'd consider embedding the rubber insert in the aluminum molding with a rubber sealant, perhaps like a windshield sealer, to prevent these "nuisance" dislodgings of the rubber insert.
Just some thoughts to sort through from the bottom of the Cracker Jack Box!
Joe. :teeth :thup