Freedom of Association, as derived from the First Amendment of the U.S. Constitution does just that. There was a famous court case, between Purdue University and Sigma Chi Fraternity, that went before the Indiana Supreme Court.
You may read about it
here.