In a word one of the biggest things preventing more African-Americans from joining Sanghas is: culture. Take a look at this African-American church service:
And compare that to the noble silence of sitting meditation, the very quiet and solemn practice we do in sitting meditation, walking meditation, retreats, even the way we eat our food!
However, socio-economic class also has a lot to do with it and as more African-Americans come out of poverty, there will be greater participation. I have met several African-American Buddhists, but admittedly most have been middle to upper-middle class.