I agree with your point that skin color ultimately wins out. 'Cause just cause you're black it doesn't mean you don't have European roots and just cause you're white it doesn't mean that none of your ancestors came from Africa.
Plus with being black in America I honestly think that there's an even quicker
tendency to assume an undefined African slave history. My best friend's father's side of the family is from Antigua so there's no American slave history there. Two guys I work with are black but their fathers are from Panama. Again, no American slave history. I think it would suck to grow up in the US learning "all your ancestors" came from slavery when you know full well they didn't!