Shalom,
What state do you live/attend church in?
It is very unfortunate that people still think these words and phrases are acceptable.
I believe that if you were upset, you should let the pastor know that he offended you. Unfortunately, some people who use these words usually grew up around racism and negativity and SOMETIMES don't realize that it is offensive.
Now, if your pastor thinks nothing of this, then there is something seriously wrong with him.