Well, I am married to the younger brother of one of my Phi Mu sisters -- 21 years and two children later, I still think he's pretty darn cute.
My sister and I lived together for two years while undergrads.
However, even though he and I went out occasionally during college, it really was "as friends." He was (is) a Sig Ep and I went to his Homecoming dance with him one year. He also escorted me to one of my Phi Mu formals. We were both graduated and well into our 20's before we started to really date and get serious. My sister was very happy and supportive of the relationship. (And yes, she was a bridesmaid in the wedding!)
At this point in your life, I would not pursue anything romantic with him. Just be friends and don't ever view him as someone to have a "fling" with. Because of your close relationship with your Big, the dynamics are not right for you to view her brother in that light. When the fling ends, and especially if someone gets hurt, there will be a very uncomfortable situation for everyone.
My sister, her brother and I had some great times together during those college years and he and I never crossed the friend/romantic line...even though we were attracted to each other. Just wasn't the right time or dynamics for it. Later, it was and we had a great foundation of friendship supporting us.