How did old-money people of color maintain their wealth through racist times?

There are wealthy Black, Hispanic, Asian, etc. families that have been multimillionaires for a few generations (maybe even billionaires), but my question is: how did these Black and other families of color maintain their wealth through racist times?