I've used both compounds (rubbing compounds) and sandpaper (wet/dry). I've had good results with both. That makes sense, as essentially what both methods are doing is the same thing: removing a very thin layer of (oxidized) gelcoat to get down to the good (shiny) stuff.
The one caveat is that gelcoat on any boat is only so thick, so there comes a point when there is no longer enough "good stuff" left under the oxidation. Then if it were me I would likely use a two-part paint to restore the boat (depending on boat/use of course).
Whether or not said point is reached depends on how thick the original gelcoat was, and how many times or how aggressively it was rubbed/sanded in the past.
For compounds I've used both Meguiars and 3M brands. They each make varying grades from mild (sometimes combined with wax) to aggressive. The first time I tried it - after doing part of a rather oxidized boat by hand, and then with a "hobby' buffer and not seeing a lot of difference, a friend came by with a "real" buffer (a big Milwaukee with a Superbuff type pad, but there are other brands now too), and .... instant shine! I borrowed that buffer for the rest of the job (oh my aching shoulders :cry) , and rented one since then for compounding jobs (until buying one).
For wet-sanding, you have to sort of feel it out as to where to start. Somewhere in the 600 range maybe, give or take. Then work up to 1000 or even 1500 for a nice shine.
You may want to test in an inconspicuous area (or maybe even find an old boat or dinghy to try it out on first) .