Here's the Washington Post last summer:
https://www.washingtonpost.com/posteverything/wp/2015/06/10/americas-swimming-pools-have-a-long-sad-racist-history/?utm_term=.3ad27b7180df
"In some cases, white swimmers imposed de facto segregation through violence and intimidation. At Pittsburgh’s Highland Park Pool, for example, white swimmers attacked black swimmers — sometimes with rocks and clubs — to prevent them from entering the pool. Police officers encouraged these attacks and typically arrested the black victims, charging them with “inciting to riot.” In attempting to explain why black swimmers were being attacked at Highland Park Pool but not at other city pools, the Pittsburgh Courier wrote: “The whole trouble seems to be due to the way Highland Park Pool is operated. It is the only city pool where men and women, girls and boys swim together. This brings the sex question into the pool and trouble is bound to arise between the races.”
The same type of trouble had no chance to arise at public swimming pools in the South and Mason-Dixon line cities such as St. Louis and Baltimore, because public officials mandated racial segregation, explicitly barring blacks from entering “whites-only” pools."
As noted in the Root article above, the biggest predictor for whether a child learns to swim is whether their parents could swim. So I fail to see how we can discount racism as contributing to the low levels of ability to swim among African Americans.