Here is the definitive information on epoxy and mat. This is from West Systems "Epoxy works":
Can WEST SYSTEM® epoxy be used to wet out chopped strand mat? The answer is yes. The fiberglass strand in mat wets out with epoxy, but the binder holding things together does not dissolve. (It does get put into suspension and is sealed in the cured epoxy.) This undissolved binder causes the wet-out mat to remain a bit stiff compared to wet out with a styrene-based resin. For gently curving or flat projects like cabin soles or plywood decks, mat and epoxy should work fine. The fabric does not wet out perfectly clear with epoxy. Wet-out clarity of mat with epoxy varies somewhat with different suppliers, but none of them wet out as clear as a good 4 oz or 6 oz fiberglass cloth.
The texture of the chopped strand mat is quite rough given its random fibers (many of which come loose when epoxy is applied). Mat requires a number of coats of epoxy to fill the profile at the surface. 879 Release Fabric can be applied over the freshly wet-out mat to compress the fibers and minimize the need for many of the buildup coats. Using release fabric in this way will result in a much smoother surface, and if you are going to allow the epoxy to cure before continuing, the surface is ready for buildup coats of epoxy or fairing putty after the release fabric is removed.
4 mil plastic sheeting can also be used over the still uncured epoxy and mat to compress the fibers. Trapped air bubbles can be removed by piercing or slitting the plastic over the bubble before the epoxy begins to cure. If the plastic was clean when it was applied, you can pull the plastic after the epoxy cures to a hard gel and apply epoxy fairing putty or buildup coats of epoxy later the same day. If you allow the epoxy to cure hard before pulling the plastic, be sure to sand the surface completely dull before applying more epoxy.
When choosing chopped strand mat to be used with epoxy, look for mat that is soft and pliable. Some forms are quite stiff and may cause problems wetting out with epoxy. Stiff mat is often older stock and the fibers may take more time to wet out with epoxy.
It answers most of your questions. The vinyl ester resin is often used these days as a first few layers of laminate to help prevent blistering. It is less permeable to water than is Polyester resin. Vinyl Ester is also a better adhesive on secondary bonding. Polyester is poor in secondary bonding and Epoxy is best.
I have never molded anything (except flat surfaces like table tops and trays) with epoxy--I have always used polyester for molding using CSM (and cloth) . I use epoxy for repair because if its better secondary bonding. Some people use polyester--I have seen several of these repairs fail.
Laminating resin does not have wax in it--you can get it to cure by applying it against the waxed surface of a mold, by covering with occlusive material, or spraying with PVA (which is what I have used). Finish resin has wax which floats to the surface and occludes the air (oxygen)
With carbon fibers and some other exotics boats are manufactured with epoxies. (Think the multi million dollar America's cup or Around the World solo racing boats.) For the most part most other production boats are polyester.
What are you molding? You will have to have a 'non stick" surface that you are laying up against. You can highly wax with molding wax or spray wax, There are tapes which are used specifically for epoxy molds (like table tops etc--"Tuck tape", house sheathing tape or Epoxy molding tape.). You are going to have to make the mold--and either cover it with tape or release sheeting or use something like HDPE (Starboard type of material) with epoxy mold release sprayed over it.
With any mold, you have to have the surface perfect, and then waxed--that way the item you are molding will have a perfect finish. The finish can be sanded and painted--or even gel coated. A vinyl Ester resin gel coat is available. Or..you can use polyester gel coat over epoxy--either by using several techniques, or a tie coat. All of this depends on what you are molding and what the external appearance needs to be.
If it were going to be subject to stresses and out in the elements, etc I might consider spraying polyester gel coat and then laying up polyester resin or vinyl ester resin.