I am not a Microbial Pathogensist, so I'll just use the internet's preferred method of saying something confidently that is probably wrong on some pedantic level and having actual experts climb out of the woodwork to correct me, but...

Mold isn't a single organism, it's actually a colony of many individual microbes that work together, most of the time; there are exceptions since the word "mold" is a lay term as much as it's a scientific term, and the common usage doesn't have the same rigor applied ("if it's slimy/fuzzy, it's mold" kinda reasoning).

The colony aspect is important, because you're probably inhaling mold spores and eating tiny amounts of mold every day. The microbial aspect means that eating a whole colony has the potential to infect you even if your body kills off 99.9% of the microbes.

As to why an infection is dangerous depends on the type of mold. Some take up resisdence in specific organs and starve the organ cells of vital nutrients, others are carnivorous and eat your tissues/cells, others may eat beneficial bacteria or starve them of vital nutrients, some secrete toxins that are relatively harmless in small doses but deadly with a full-blown infection—penicillin is a great well-known example of this: it's the chemical the mold uses to kill off bacteria competing for the same resources—, sometimes it's just the immune system's response that's dangerous.

If you want an example of what such an infection does on our scale, look up everyone psychonaut's favorite: Ergotism. The Ergot fungus grows primarily on cereals/grains and secretes chemicals that are psychoactive in humans (one of which is the precursor that Dr. Albert Hoffmann first derived LSD from). Unfortunately besides mind expanding insights into the nature of reality, it also comes with a nasty infection that can cause convulsions, painful burning/tingling/freezing sensations, diarrhea, vomiting, gangrene, psychosis, and more.

That all being said, if you accidentally swallow a bite of moldy bread, you probably don't need to freak out and call poison control/EMS; just don't regularly eat moldy food and expect to have your immune system stave off a full-blown infection for long.