Self hosting search engines is very hard. The scraping, indexing and storage requirements are immense. You could definitely self-host a front end (with your QoL improvements), but the back end search engines (Bing/Google/etc) will be able to track you all the same.
this post was submitted on 14 Jul 2025
19 points (100.0% liked)
Privacy
39869 readers
399 users here now
A place to discuss privacy and freedom in the digital world.
Privacy has become a very important issue in modern society, with companies and governments constantly abusing their power, more and more people are waking up to the importance of digital privacy.
In this community everyone is welcome to post links and discuss topics related to privacy.
Some Rules
- Posting a link to a website containing tracking isn't great, if contents of the website are behind a paywall maybe copy them into the post
- Don't promote proprietary software
- Try to keep things on topic
- If you have a question, please try searching for previous discussions, maybe it has already been answered
- Reposts are fine, but should have at least a couple of weeks in between so that the post can reach a new audience
- Be nice :)
Related communities
much thanks to @gary_host_laptop for the logo design :)
founded 5 years ago
MODERATORS
Are there even open source indexing software available?
There's YaCy. I've run a node for a while but it ended up filling up my server's drive just indexing german wikipedia and the results were terrible.
And it's still not private because you have to broadcast the query across the network.
Stract, Marginalia, Wiby, Mwmbl, etc
The two first are NLnet funded and the second one is one of the best developed despite it uses Java in contrast to Rust. I see the developer taking the development very seriously.
None that im aware of. There are webscrapers, and I guess you could just webscrape and dump the results into a postgres db and use it to index. But I'm guessing you'll eventually want something more tuned/custom? But even if it existed, there is the discovery problem. How do you find the sites to scrape? Bing and google both let site operators submit urls, but that isn't gonna scale to self-hosting.
Yeah, exactly. I just think that making anonymous request from google or bing is private enough for me.
That's a good point, I forgot that stuff like SearXNG are only frontends so in order to add personalisation to them you'd have to modify your queries to Bing/Google/etc I assume, rather than do what Google etc do with whatever algorithm they use for providing search results.
Self-hosting a search engine is unfortunately not feasible given the amount of data and power required for it. Not to mention access to the data (crawling yourself or using another index).
For privacy and customization there is Kagi, which is amazing and very customizable, but requires a paid subscription. You are a customer rather than the product, though.