Well it seems to me that if you made this phone call after you worked the agreed amount of hours, you were on YOUR time, which means you can do whatever you want. If he wanted you to continue working after those hours then he should have payed you over time or kept his mouth shut and let you go home.
If it is something that is important to you, I would seek some legal advice. It sounds like this guy's in viloation of the contract.
|