Part of the answer lies in the spacing of the plants relative to the width of the beds. With intensive cropping methods, growers will often put a high number of plants into the beds, and this increases the amount of root competition for space in that horizon of the soil. There's little competition below them, and often plenty of nutrient and water due to the way the beds are constructed, so it's the optimal path for the plants' roots to take.

Sometimes factors like crop rotation come into play, such as when growing carrots, parsnips, or other taprooting vegetables before another crop (sometimes seeding the new crop before pulling the root crop). The deep roots put sugars into the soil which cultivate a host of microscopic life, and the second crop can sometimes use these colonies, or can at least make use of the crumbly soil structure left over from the first.