Your personal roboport is entirely separate from the logistic network roboports' construction bots.

When you have a personal roboport equipped, bots in your inventory will use your inventory to build in your personal range.

Construction bots inside roboports will build anything in the connected logistic network (roboports in each other's green range) construction range (orange range... If I'm remembering colours right).

You can speed up personal construction by equipping more or better roboports, and logistic network construction by adding more bots/roboports for charging and closer materials. Research also makes a big difference. To some extent early game construction is just a bit slow.

What I like to do is have my mall store everything in passive provider or filtered storage chests so it's all available for construction and personal logistics. Then I just accept that my early network is a bit slow, and do other things while waiting on them for large projects. Things I'm actively building my personal bots help with from my own inventory.

Edit: one thing to keep in mind is bots reserve ghosts when they start a build job, so if you have a large logistic network they can take a lot longer to fly over than something closer would have to finish it. For this reason some people prefer smaller, separate networks and constructing from personal or spidertron inventories.