What Frederick Douglas thought is anecdotal. I could quote someone with a differing opinion and it would bring no more or less of anything of value to the discussion.
Here's something from snopes:
"Robert E. Lee, the commander of the Confederate Army of Northern Virginia and (from 1865) the general-in-chief of Confederate forces, neither owned slaves nor inherited any, thus it is not correct to assert that he “freed his slaves” (in 1862 or at any other time).
As in the case of Ulysses S. Grant, the slaves that Lee supposedly owned actually
belonged to his father-in-law, George Washington Parke Custis, and lived and worked on the three estates owned by Custis (Arlington, White House, and Romancoke). Upon Custis’ death in 1857, Lee did not “inherit” those slaves; rather, he carried out the directions expressed in Custis’ will regarding those slaves (and other property) according to his position as executor of Custis’ estate.
Custis’ will stipulated that all of his slaves were to be freed within five years: “… upon the legacies to my four granddaughters being paid, then I give freedom to my slaves, the said slaves to be emancipated by my executor in such manner as he deems expedient and proper, the said emancipation to be accomplished in not exceeding five years from the time of my decease.” So while Lee did technically
free those slaves at the end of 1862, it was not his choice to do so; he was required to emancipate them by the conditions of his father-in-law’s will."