I personally know more about the slavery ancestry of my Jamaican mother's background than that of my American father's.
That is a good point, we are "brown", but so are a lot of other people whom I wouldn't necessarily defined as black.
Also on wikipedia (yeah, good old "reliable internet") they discuss that there are people who are very dark brown from other parts of the word (like India)who also consider themselves black.
Then there are Africans (I remember from Real World: Voyage At Sea I think?) that an African woman told them that African-Americans aren't Africans! LOL!
So yeah, if we listened to everyone then we'd be nobody. And I'm too busy worrying about being somebody to worry about the color of my skin and the history of my people every second of everyday! So it's definitely a personal choice to go for something and stick to it and make it work for yourself.
