It was a poem about the bombardment of a fort at the subsequent repulsion of the British forces in the Battle of Baltimore in 1814. It was literally titled "Defence of Fort M'Henry" before it was used as the lyrics to the national anthem.

The context of the poem doesn't jibe with it being about African slaves.

We have lots of bullshit revisionist history that tries to whitewash racism, but this ain't it.