To clarify: I meant how do I do it via API calls,

If you mean at the X11 call level, I think that it's a window hint, assuming that you're talking about a borderless fullscreen window, and not true fullscreen (like, DGA or DGA2 or something, in which case you don't have a fullscreen X11 window, but rather direct access to video memory).

https://specifications.freedesktop.org/wm-spec/latest/ar01s05.html

See _NET_WM_STATE_FULLSCREEN, ATOM

If you're using a widget toolkit like gtk or something and writing the program, it'll probably have some higher-level fullscreen toggle function that'll flip that on X11. Ditto for SDL.

If you mean in a script or something, I'd maybe try looking at xprop(1) to set that hint.

I'd also add, on the "user" front, that I don't use F11 and I think that that every window manager or desktop environment that I've ever used provides some way to set a user-specified keystroke to toggle a window's fullscreen state. I've set Windows-Enter to do that for decades, on every environment I've used.