The interference pattern disappears if anything becomes entangled with the which-way information at all, it doesn't need to even be an "observer" (unless you are using "observer" broadly enough that it can include even a single particle). You can replace the entire measurement device with a single particle that interacts with the particles at the slits in such a way that it becomes perfectly correlated with the which-way information that the observer has no awareness of (such as if a moat of dust interacts with the particle because the experimenter did not isolate it well) and that is sufficient for the interference pattern to disappear.