Perhaps there have been mixed signals, kitten. But what's done is done. And nothing irreversible or irreperable has been done.
Possibly it was a lapse of judgment on your part but your statement you didn't feel you could say no is well-made; a prime reason why work involvement is a bad idea. Hope no-one does get mad at you incidentally, you seem really nice.
Onward to the issue, it isn't so much worse having not declared a lack of interest initially; versus later making it clear having reflected.
There hasn't been a tumultous long relationship, you won't owe him long explanations. You won't have to marry him because you perhaps 'dated' him
. Simply speak to him to resolve the situation. You'll be okay in the end.