How many slaveholders were there, really?

Taking the opportunity of Jillian Michaels saying stupid stuff on national televison to talk a little, in detail, about the economy of slavery and the extent of slaveholding in the antebellum South.