Hi bshillam,
Swantown Marina in Olympia is a nice facility. (No place to wash your trailer, though; we had to haul to a car wash to rinse.) This is just an observation on my part... that is a long run from Olympia to Poulsbo for only 4 days. Certainly can be done with your boat, but it doesn't give you a lot of time to stop and smell the roses (or the bakery goods). There are so many cool places along the way that you will have to just breeze on by to keep your schedule.
What is your cruising style? Some folks like to get on the boat, push the throttle and go... all day long. Some like to amble along, sight-seeing along the way. Some like to stay in marinas every night, others want to swing on the hook. Do you want restaurants within an easy walk or cooking on the boat? The Puget Sound has some spectacular scenery, and the view from the water is the best. But for our style, part of the fun of cruising is getting OFF the boat, as well. Whether it's hiking in the wilderness or exploring the big city. Spend a relaxing afternoon and night in a marina, and you will meet some interesting people. Swantown in Olympia was especially entertaining, because the transient docks are near the ramp. Good showers, park-like atmosphere, and while we were there, there was a farmers' market that was an easy walk. It was easy to spend the better part of a week just day-tripping out of Olympia.
With 4 days, it might be fun to get a feel for a nice marina, a quiet cove, a state park, and whatever else tickles your fancy. All of that is available from any part of Puget Sound without having to run the whole length of it... giving you more time for sight-seeing and spending less on fuel. Please understand that I am not suggesting you should change your plans in any way, just presenting another perspective.
One way to do it is to decide what is THE most important area you want to see. Then where is the most convenient place to launch nearby? Then, decide how far you want to travel in your allotted time. You can spend two days heading out, then two days heading back. Or, you can go as far as you want on the first day out and spend the next three sight-seeing your way back. Or, you can go a ways each day and spend the last day running back (this way could be chancier if there is weather to be dealt with). Or, you could spend one day running, two days in one area, and one day running back. Or... Yeah, the possibilities are unlimited.
Any part of the Sound could easily provide 4 days of interesting exploring, without having to go far. Running from one end to the other in 4 days may give you the opportunity to run the boat a lot, but not see much along the way. Not that that's a bad thing, just good to know before you find yourself into one of those "I don't know, what do you want to do?" scenerios.
Good luck with your planning.
Best wishes,
Jim B.