Mwmbl - the Open Source and Non-Profit Web Search Engine
No ads, no tracking, no profit
Mwmbl is a non-profit, open source search engine
where the community determines the rankings. We aim to be a
replacement for commercial search engines such as Google and
Bing.
We have our own index powered by our community - while the index itself
is centralised, the crawling is distributed across servers run by our volunteers.
Our index is currently much smaller than those of commercial search engines, but
you can help change that by joining us!
Community
Our main community is on
Matrix but we also have a
Discord server for non-development
related discussion.
The community is responsible for crawling the web (see below) and
curating search results. We are
friendly and welcoming. Join us!
Documentation
All documentation is at https://book.mwmbl.org.
Development coverage
make coverage measures the Python and Rust code and prints a combined,
executable-line-weighted percentage. See the coverage guide
for prerequisites, exclusions, and CI details.
Crawling
If you have spare computer power and bandwidth, for now the best way you can help is
by running our command line
crawler.
If you have Firefox you can help out by installing our
extension. This
retrieves search results from commercial search engines to use to evaluate our own ranking.
It does not use or access any of
your personal data.
Why a non-profit search engine?
The motives of ad-funded search engines are at odds with providing an
optimal user experience. These sites are optimised for ad revenue,
with user experience taking second place. This means that pages are
loaded with ads which are often not clearly distinguished from search
results. Also, eitland on Hacker News
comments:
Thinking about it it seems logical that for a search engine that
practically speaking has monopoly both on users and as mattgb points
out - [to some] degree also on indexing - serving the correct answer
first is just dumb: if they can keep me going between their search
results and tech blogs with their ads embedded one, two or five
times extra that means one, two or five times more ad impressions.
But what about...?
The space of alternative search engines has expanded rapidly in recent
years. Here's a very incomplete list of some that have interested me:
Of these, YaCy is the closest in spirit to the idea of a non-profit
search engine. The index is distributed across a peer-to-peer
network. Unfortunately this design decision slows the fetching of search results.
Marginalia Search is fantastic, but our goals are different: we aim to
be a replacement for commercial search engines whereas Marginalia aims to
provide a different type of search.
All other search engines that I've come across are for-profit. Please
let me know if I've missed one!
Designing for non-profit
To be a good search engine, we need to store many items, but the cost
of running the engine is at least proportional to the number of items
stored. Our main consideration is thus to reduce the cost per item
stored.
The design is founded on the observation that most items rank for a
small set of terms. In the extreme version of this, where each item
ranks for a single term, the usual inverted index design is grossly
inefficient, since we have to store each term at least twice: once in
the index and once in the item data itself.
Our design is a giant hash map. We have a single store consisting of a
fixed number N of pages. Each page is of a fixed size (currently 4096
bytes to match a page of memory), and consists of a compressed list of
items. Given a term for which we want an item to rank, we compute a
hash of the term, a value between 0 and N - 1. The item is then stored
in the corresponding page.
To retrieve pages, we simply compute the hash of the terms in the user
query and load the corresponding pages, filter the items to those
containing the term and rank the items. Since each page is small, this
can be done very quickly.
Because we compress the list of items, we can rank for more than a
single term and maintain an index smaller than the inverted index
design. At least, that's the theory. This idea has yet to be tested on
a large scale.
How to contribute
There are multiple ways to help:
If you would like to help in any of these or other ways, thank you!
Please join our Matrix chat
server or email the main
author (email address is in the git commit history).
Development
Local Testing
For trying out the service locally see the section in the Mwmbl book.
Code style and checks
Python code is formatted and linted with ruff and type checked
with ty. Both are configured in pyproject.toml and pinned in
the dev dependency group, so everyone runs the same versions.
Install the git hooks once, after uv sync:
make install-hooks
From then on every commit is formatted and checked. To run the same checks by hand:
make check # format check + lint + type check, exactly what CI runs
make fix # reformat and apply the safe lint fixes
ty is gated on error-level diagnostics only. There is a large warning backlog - Django's
Model.objects is invisible to ty, which has no django-stubs plugin support - so the rules with
pre-existing violations are downgraded to warnings in [tool.ty.rules]. Run make typecheck-all
to see the whole backlog, and promote a rule back to error once its count reaches zero.
Using Dokku
Note: this method is not recommended as it is more involved, and your index will not include any data unless you
set up a crawler to crawl to your server. You will need to set up your own Backblaze or S3 equivalent storage, or
have access to the production keys, which we probably won't give you.
Follow the deployment instructions
Frequently Asked Question
How do you pronounce "mwmbl"?
Like "mumble". I live in
Mumbles, which is spelt
"Mwmbwls" in Welsh. But the intended meaning is "to mumble", as in
"don't search, just mwmbl!"
Star History
Acknowledgements
Many thanks to Mohamed Alm for security consultations and recommendations.
