Go with Memorial Day! Don't worry about the hotels though. In my opinion, Memorial Day seems like one of those holidays that people kind of "spur of the moment" make plans for. Not like Christmas or anything. Your guests will have their invitations and hotel reservations long before any other travelers can take up the rooms.
Plus, the weather is nice that time of the year.