Ok, so let me get this straight....
The only times he would use a condom is if a girl wasn't on the pill? Ok, that is not cool. Girls could very easily lie and say, "Oh yeah, don't worry, I'm on the pill!" Also, THE PILL DOES NOT PROTECT FROM STD'S!!!!
Furthermore, if you think your guy's numbers may be in the 30's
and the only time he'd wrap it up was when a girl was "on the pill" then you need to tell him that you will NOT have sex with him unless he gets tested. Tell him you'll go with him.
You need to protect yourself.
Your initial question was if you should sleep with him or not. I cannot answer that for you because that is your own very personal decision. But, the fact that you have asked us for our opinions leads me to believe that you're having doubts. If you don't want to sleep with him, you don't have to. But, if you do, be VERY careful.