Actually, this does not surprise me. At the time a White woman who was with a Black man would pass at Black to keep from being assaulted by her own race. And she would have been embraced by Blacks because of her "light" complexion.
My grandmother's grandmother was Irish, but that piece of info was kept within the family, since she married a Black man.
As for the great grandparents, if they were very poor are for some reason not accepted by other Whites, then they too would have been accepted by Black folx more readily. Those times were really crazy.
Now as to Harold Ford, I do not understand his reasoning for claiming this now. Maybe he is trying to show that "passing" went both ways.