Now I don't know enough about electronics to know how wrong this is

Very, assuming the refrigerator in question typically runs on a typical power grid you'd find in the US or Europe (source: am electrical engineer)

Mainly because most compressors I'm aware of use alternating current (AC) motors, or at a minimum accept AC power. Batteries alone produce direct current (DC). The simplest way to make this work would involve an inverter (converts DC to AC). Cheap ones probably have at least a 10% conversion loss, so you're looking at an hour or two at most.

Edit: should also mention that discharging a typical lead-acid battery until it's all the way flat (realistically below ~11V) does irreparable damage. Might be cheaper to replace the contents of your fridge :)