Martha's Vineyard is **quite** a hike from New York. Going there will eat up an entire day, at least; if you take the Seastreak, it's a 5 hour trip, one way (
http://www.seastreak.com/marthas.aspx). I might give MV its own trip another time.
(Vineyard veteran of 23 years here...)