The self-hosted install-quality report
data as of August 28, 2026 · every claim traces to a published verification transcript
Between July and August 2026 we ran the documented install path for 96 of the 98 self-hosted apps in our catalog on a fresh Ubuntu 26.04 VPS — not in a container on a laptop, but on the same class of machine our guides recommend. Every command executed for real, unattended, with a hard timeout and a wiped box between apps. Two apps (agentarea and vllm) haven't been through the harness yet, so they're excluded from every number on this page.
The result: 27 of 96 apps — 28% — failed on the first attempt. Most of those failures were our own fault, which is exactly the point: install instructions written by careful people and reviewed by other careful people still break the moment something actually runs them. The interesting part is what broke, and whose bug it was.
The numbers
Six upstream defects, with receipts
Six times, the failure wasn't ours and wasn't the reader's: the vendor's own documented path is broken on a current system. Each of these passed every static review — the compose file is valid, the tag looks real, the Makefile target reads fine. They only fail when something runs them.
Checkmate: the pinned MongoDB won't boot on a current kernel
Upstream's reference docker-compose.yaml pins mongo:8.0, and that tag's current build refuses to start on Linux kernels 6.19 and newer:
MongoDB cannot start: Linux kernel versions 6.19 and newer has a known
incompatibility with this version of MongoDB (SERVER-121912)On a 2026 Ubuntu LTS — the OS a new VPS actually ships — the whole stack dies on dependency mongodb failed to start. The fix is a one-character retag to the floating mongo:8, which resolves to a patched build. Invisible to any review; fatal on every fresh box.
TeamPass: the official .env.example pins a Docker tag that no longer exists
Follow the official quick start verbatim and docker compose up -d dies with failed to resolve reference … not found: upstream's own .env.example pins TEAMPASS_VERSION=3.1.5.2, a tag Docker Hub has since stopped serving. The compose file's own default (latest) would have worked — the example config is what breaks it.
Twenty: the recommended password generator breaks its own database connection
Twenty's .env.example warns that the Postgres password must contain no special characters — but the natural way to generate one (openssl rand -base64) produces + and /, which break the server's DSN parser. First boot: container unhealthy. Generating the password as hex instead fixes it. A footgun upstream documents and then hands you.
Dokku: the bootstrap script refuses current Ubuntu
Unsupported Linux distribution. Only the following versions are supported:
Debian [11, 12, 13], Ubuntu [22.04, 24.04]Upstream's bootstrap.sh hardcodes a distro allowlist that doesn't yet include Ubuntu 26.04. Nothing to work around — the installer exits before doing anything. If you're on a current LTS, Dokku's official path simply says no.
SigNoz: the documented install method no longer exists upstream
The long-documented deploy/docker directory doesn't exist in a current clone — as of v0.130.0 upstream replaced the install script and bundled compose files with a new Foundry-based flow. The old path doesn't fail cleanly; it just isn't there. (And the new installer drops its CLI outside PATH in a non-interactive shell, which our snippet now handles explicitly.)
Jan: the quickstart fails on the repo's own state
make quickstart in a fresh clone of janhq/server fails inside its own template-copy step:
Error: failed to copy .env template: read template: open .env.template:
no such file or directoryCopying the template explicitly and starting the full stack directly gets the infrastructure healthy, though the full application stack exceeded our harness's 15-minute cap while still pulling images — so we can vouch for the failure, and only partially for the workaround.
Honorable mention — Plausible CE: two boot-crashers from documentation drift
Found in our deploy-guide testing rather than the install-snippet sweep, same disease: current Plausible CE reads env from .env (older guidance said plausible-conf.env — following it leaves every variable blank), and TOTP_VAULT_KEY must now be exactly 32 base64-encoded bytes, where older guidance generated 48:
(ArgumentError) TOTP_VAULT_KEY must be Base64 encoded 32 bytes…
Got Base64 encoded 48 bytes.Eleven official installers can't run unattended
A separate class isn't broken at all — it just can't be automated. Eleven apps' official installers require a human at a terminal: they prompt, render TUIs, read /dev/tty directly, or run in the foreground forever.
docker run -it wizard; no TTY, no install.-ti flags need a real terminal.generate_config.sh prompts by default, though MAILCOW_HOSTNAME/MAILCOW_TZ env vars make it scriptable — that's how we verified it./dev/tty directly; unattended it loops on an email prompt and segfaults. Driven through a PTY it installs cleanly.createsuperuser wants a TTY unless you pass every argument explicitly.None of this is a defect. But if your plan is cloud-init, Ansible, or a bash-over-SSH bootstrap, these are the apps where that plan meets a prompt it can't answer.
Most of the failures were ours — and that's the point
Here's the uncomfortable stat: of the 27 first-attempt failures, 21 were bugs in our own install snippets — the commands we'd written, reviewed, and published. A missing sudo. A compose file that mounts prometheus.yml nobody ever creates. A snippet that maps 22:22 for Git-over-SSH — a port every VPS's sshd already owns, and following the original comment to move sshd off :22 could have locked a reader out of their own box. Four distinct defects in one four-line Headscale snippet.
We're publishing that number instead of hiding it because it's the strongest argument for the whole exercise: if the people writing the docs get it wrong 22% of the time, the only honest install doc is one that has actually been executed, recently, on the OS it claims to support. That's why every app page here carries an install-verified stamp with the date and OS of its last real run — see how we test.
What a real run catches — including our rig lying to us
Running installs for real cuts both ways: the test rig itself can lie to you. Six times, our harness produced failures that looked exactly like broken apps and weren't:
- An early runner executed compose YAML as shell for 18 apps and briefly “proved” 27% of the catalog was broken. The true count in that failure class: 2.
- Freeing port 53 for Pi-hole killed DNS for the whole box — the next ten apps all “failed” with
could not resolve host. - A bootstrap died on an apt lock before it ever added the test user to the
dockergroup — four of five code-hosting apps then “failed” on Docker permissions; none of those failures were real. - Stale state from a previous app — a leftover clone directory, a container still holding port 8000, an old container name — produced three more phantom failures across waves.
Every one of these was caught by reading the actual transcript instead of trusting the verdict column. It's also why we now assert DNS before every single app, wipe the box wholesale between apps, and verify the reset actually left it clean — and why we treat any new failure as a harness bug until the output proves otherwise.
Method
The harness SSHes into a throwaway GCP VM (e2-standard-2, stepped up to e2-standard-4 for the heaviest stacks; Ubuntu 26.04 LTS), executes each app's published install snippet step by step exactly as a reader would, applies a hard per-app timeout, and resets the box to a verified-clean state between apps — containers, volumes, images, build cache, working directory, and (after one memorable incident with Docker Swarm resurrecting containers we'd deleted) swarm state too. An app passes when every step exits cleanly and its service actually answers.
Failures were then adjudicated by reading the transcript: upstream defect, our own snippet bug, interactive-by-design, or harness artifact. The fixes now live in the snippets on each app's page, alongside the install-verified stamp. During the August runs we also captured measured first-boot idle RAM for eleven apps — those numbers appear on the relevant comparison pages with their measurement date.