If much of one's driving is w/o the trailer on and if the truck is being used primarily for human transportation during that time and if one doesn't tow too many total miles in a year, I think the solution is a full size used pickup and a car that's gets decent gas mileage. For quite awhile, we were towing with an Isuzu Trooper that in practice was being used 95% of the time for single person transportation. Admittedly, we bought the Trooper prior to the boat and at a time in our life when we where carting scouts and gear around and when my wife was not working full time. But not too long after getting the boat, things changed.
Hence, last year I picked up a used F250 with a 351, a low rear end and reasonably low miles (116k). I paid $2500 and put about 1500 in for a new rear seal and some other initial maintenance. The truck is now used only for boat towing and misc chores the require a truck (lumber hauling, etc). This year we got rid of the Trooper and bought a deisel Jetta - 37 MPG city and damn near 50MPG highway (real life numbers). Even with modest driving of about 12,000 miles per year to work and back, the Jetta save us about 530 gals of gas = about $1600/year in fuel. Now that doesn't in any way make up for the additional expense of the new vehicle but I do feel better about not wasting fuel (and the Jetta is really nice to be in).