This pet project of mine is a web search engine for discovering virtual private server (VPS) offers starting from a search query that the user can incrementally specialize to iteratively add constraints and narrow the corpus of results. Thru the past years I added 120 service providers to the data acquisition pipeline, which is run every 24 hours to refresh the entirety of the corpus and to index the VPS services in a custom C++17 database that is made searchable thru the web interface. Service add-on such as extra storage or extra memory are also indexed in the database and priced correctly. The project is split in two major halves, a backend and a web server:
1. Backend: a computer in my basement executes a ~10,000 lines Nodejs program that implements the data acquisition pipeline, the majority of these lines deal with web scraping and API fetches, followed in line count by regex and LLM processing, and finally by database indexing. The language models that I use are all large 200B+ parameters and are called via SaaS APIs; they can execute web searches to integrate missing data from the product description pages or from the APIs exposed by the VPS providers; the models are queried multiple times for each question and their responses "vote" for the correct answer until a quorum is reached; this almost completely eliminates data inaccuracies. The final product of this Nodejs program is an SQLite database file that contains the indexed data for the VPS services, which I manually copy to the web server computer (a VPS).
2. Web server: this is a lightweight VPS that runs two processes: a thin Nodejs web application that I quickly wrote without frameworks and runs off simple HTML templates, and a basic C++17 database that serves the web application. Originally the Nodejs application incorporated its own search functions that would linearly swipe thru the data arrays to produce SERPs, but as the number of indexed services grew past 300,000 VPS, the latency became unacceptable (> 1s) and easily DoS-able so I rewrote the search functions in C++17, dropping the query time to ~10ms. This custom DB allows me to easily control low-level details, such as:
2.1 to quantize numerical data to 1 or 2 bytes using a quadratic lossy compressor; the compressor is bijective and totally ordered, so the stored quantized data can be compared as-is against the quantized query parameters, reducing memory use and throughput;
2.2 to compress long strings using a trained ZSTD dictionary, which is convenient for storing millions of offer URLs in memory on a tiny VPS avoiding disk access (URLs are largely repetitive across offers from the same providers, and compress to a 1:10 ratio with a 2kB dictionary);
2.3 to swipe thru presorted indexes, each presorted by a specific criterion, so that the matched data requires no runtime sorting; in comparison tests, MariaDB would consume half of its query time on sorting;
2.4 to efficiently avoid disk access: the SQLite file produced by the backend is read only once, fully, at the start of the process, and subsequent queries are served from memory. The database is read-only and no API permits file write access or memory modification.
A few users query the search engine every day, and it currently generates snack money thru affiliate links. It is a proof of concept more than anything, and I had a lot of fun profiling the database trying out different algorithms to squeeze some decent performance out of my tiny VPS with a shared CPU.
This pet project of mine is a web search engine for discovering virtual private server (VPS) offers starting from a search query that the user can incrementally specialize to iteratively add constraints and narrow the corpus of results. Thru the past years I added 120 service providers to the data acquisition pipeline, which is run every 24 hours to refresh the entirety of the corpus and to index the VPS services in a custom C++17 database that is made searchable thru the web interface. Service add-on such as extra storage or extra memory are also indexed in the database and priced correctly. The project is split in two major halves, a backend and a web server:
1. Backend: a computer in my basement executes a ~10,000 lines Nodejs program that implements the data acquisition pipeline, the majority of these lines deal with web scraping and API fetches, followed in line count by regex and LLM processing, and finally by database indexing. The language models that I use are all large 200B+ parameters and are called via SaaS APIs; they can execute web searches to integrate missing data from the product description pages or from the APIs exposed by the VPS providers; the models are queried multiple times for each question and their responses "vote" for the correct answer until a quorum is reached; this almost completely eliminates data inaccuracies. The final product of this Nodejs program is an SQLite database file that contains the indexed data for the VPS services, which I manually copy to the web server computer (a VPS).
2. Web server: this is a lightweight VPS that runs two processes: a thin Nodejs web application that I quickly wrote without frameworks and runs off simple HTML templates, and a basic C++17 database that serves the web application. Originally the Nodejs application incorporated its own search functions that would linearly swipe thru the data arrays to produce SERPs, but as the number of indexed services grew past 300,000 VPS, the latency became unacceptable (> 1s) and easily DoS-able so I rewrote the search functions in C++17, dropping the query time to ~10ms. This custom DB allows me to easily control low-level details, such as:
2.1 to quantize numerical data to 1 or 2 bytes using a quadratic lossy compressor; the compressor is bijective and totally ordered, so the stored quantized data can be compared as-is against the quantized query parameters, reducing memory use and throughput;
2.2 to compress long strings using a trained ZSTD dictionary, which is convenient for storing millions of offer URLs in memory on a tiny VPS avoiding disk access (URLs are largely repetitive across offers from the same providers, and compress to a 1:10 ratio with a 2kB dictionary);
2.3 to swipe thru presorted indexes, each presorted by a specific criterion, so that the matched data requires no runtime sorting; in comparison tests, MariaDB would consume half of its query time on sorting;
2.4 to efficiently avoid disk access: the SQLite file produced by the backend is read only once, fully, at the start of the process, and subsequent queries are served from memory. The database is read-only and no API permits file write access or memory modification.
A few users query the search engine every day, and it currently generates snack money thru affiliate links. It is a proof of concept more than anything, and I had a lot of fun profiling the database trying out different algorithms to squeeze some decent performance out of my tiny VPS with a shared CPU.