This used to be my field before I quit academia. There are two answers both indicating towards - yes, babies remember:

1. every time we (scientists) devise a new way to ask even younger kinds (infants) whether they understand one thing or another, or whether they remember, we find out that they do. The problem is how to communicate with a nonverbal infant, let alone a newborn.

2. A lot of brain development happening in the first 6 years or so is killing a lot of neural connections and strengthening others.

The leading theory (10 years ago, when I left the field, science can move fast) proposed that this may be why adults rarely remember things before age 3 - but young children have LOADS memories of before age 3 with accounts (anecdotes) of young children having memories of the prenatal phase (“when I was in mama’s belly …”) - I call these anecdotes because I know of of many parents with these anecdotes, but no study that actually looked into validity of these stories.

The theory then simply argues that as the brain matures and kids learn all the new things they need to learn, they retire these very early memories - they simply forget. But they did have them.

Babies have memories, some going back pre-birth. They can recognise music and voices from the womb. My daughter started understanding and following instructions around 3-4 months.

What you might be talking about are adult accessable memories. At some point, the structure of memory changes. I personally suspect it's tied to the acquisition of language, but that's just my opinion. The key point is that memories from before the transition are difficult, or impossible to interpret. This sets the line for "first memories". For most it's around 2-3 years, though it can be earlier (mine is my 1st birthday).

Interestingly, the better someone's adult memory, the later their first memory tends to be. This implies multiple revisions, each one leaving older memories inaccessible, but giving better performance/sorting.

I went to New Zealand when I was 2, and I have some vague memories of it. I’m guessing that it was different enough from my normal life that my brain could retain in

Of course they have memory, how would they learn otherwise?

Forming actual memories of situations and people comes a bit later, though. Still, humans can be traumatized at any age, even a baby that can't form conscious thoughts yet.

They sure as hell do. Older memories fade away around puberty iirc because your brain gets somewhat rewired.

I remember my 2nd birthday party. I was seated at the table and bunch of noisy kids were there. I remember thinking how much I disliked the noise and chaos.

The scientific consensus is that children start forming memories after the age of 3.

That being said, some toddlers might form memories earlier or later, depends on the child and brain development in general. I suppose girls are more likely to form memories earlier cuz they develop qucker, but this is just a theory I have, I haven't actually taken any scientific steps to proving or disproving it.