Building an audience over time is exactly how blogs, and publishing in general, work unless you start off with a lot of advertising or endorsements. For better or worse, there's far more content than there is time for a large audience to read it all.

This gives you three choices:

• specialize and post in an existing community that's aligned with that specialization. People will nearly always engage, especially if the content is good
• specialize and start your own blog. You could even try seeding it by referring people to it from already existing specialized communities. People will know what to expect content wise and keep coming back if the subject you're talking about is interesting to them and the content is good
• don't specialize and strike out on your own. If the content is good and you stick with it your audience will eventually grow. This will probably take more time because your audience will initially be looking for content that relates to what they've seen in the past, but what you're really offering is your personality, writing style, world view, etc

Personally, if I'm looking for engagement I choose the first option.