I have not taken the rubber type material off, but is is nice to have on. I have owned a number of boat with "Destroyer" wheels, and the metal is cold and hard to the touch--so I ended up covering them with laced on leather...
I would wonder what the underlying SS is like, and if it is polished as a purpose built SS wheel is? Also the diameter of the pipe may not be as comfortable on your hand grip.