I'm probably just left of center, and I don't really care if he plays golf. Hell, being POTUS is a tough, stressful, job that ages you (look at pre 9/11 photos of Bush), and I hope that he is able to relax. If playing golf does that and prevents him from keeling over from a stroke from stress, great!
I remember hearing that same "statistic" about Clinton as well; that job is a 24/7 thing that they can't take breaks from. Plus, just because someone isn't at the White House doesn't mean that they aren't working.
