Hi HN, I’m Andrey. I previously worked on Puppeteer and Playwright; for the last couple of years I’ve been building Flakiness.io, leading a small bootstrapped team.
Flakiness.io collects test results from CI, associates them with GitHub commits and pull requests, and cross-references them with Git history. This way, a test report can show whether a failure is new, whether a test has been flaky before, and which commit caused a test to start running 5x slower.
We built integrations for several test runners, along with an open JSON report format that we use to represent concepts shared across different runners: tests, tags, annotations, retries, steps, attachments, logs, and more.
Supporting large projects was an interesting engineering challenge: some repositories generate hundreds of thousands of test results per commit, but the data is very repetitive. Our experiments showed that it can be compressed and analyzed without decompression, so we ended up with a specialized analytics engine that can process billions of results on modest hardware while keeping the service affordable.
The report format, SDK, and runner integrations are MIT-licensed. The server is closed source, which is how we plan to make money. To avoid locking users in, our CLI [1] can export all of your data so you can process it however you want.
Hi HN, I’m Andrey. I previously worked on Puppeteer and Playwright; for the last couple of years I’ve been building Flakiness.io, leading a small bootstrapped team.
Flakiness.io collects test results from CI, associates them with GitHub commits and pull requests, and cross-references them with Git history. This way, a test report can show whether a failure is new, whether a test has been flaky before, and which commit caused a test to start running 5x slower.
We built integrations for several test runners, along with an open JSON report format that we use to represent concepts shared across different runners: tests, tags, annotations, retries, steps, attachments, logs, and more.
https://github.com/flakiness/flakiness-report
Supporting large projects was an interesting engineering challenge: some repositories generate hundreds of thousands of test results per commit, but the data is very repetitive. Our experiments showed that it can be compressed and analyzed without decompression, so we ended up with a specialized analytics engine that can process billions of results on modest hardware while keeping the service affordable.
The report format, SDK, and runner integrations are MIT-licensed. The server is closed source, which is how we plan to make money. To avoid locking users in, our CLI [1] can export all of your data so you can process it however you want.
Here is a public project to explore:
https://flakiness.io/wordpress/gutenberg
We’d love your feedback! What's missing for Flakiness.io to fit your test setup?
[1]: https://docs.flakiness.io/cli/overview/