The self-hosted install-quality report
data as of September 27, 2026 · every claim traces to a published verification transcript
Between July and August 2026 we ran the documented install path for 96 of the 98 apps then in our catalog on a fresh Ubuntu 26.04 VPS — not in a container on a laptop, but on the same class of machine our guides recommend. Every command executed for real, unattended, with a hard timeout, and Docker state and the working directory wiped between apps. Two (agentarea, vllm) hadn't been through the harness, so they're excluded from every number on this page.
The result: 27 of 96 apps — 28% — had a real defect on the first attempt. Most of those failures were our own fault, which is exactly the point: install instructions written by careful people and reviewed by other careful people still break the moment something actually runs them. The interesting part is what broke, and whose bug it was.
The numbers
Five upstream defects, with receipts
Five times, the failure wasn't ours and wasn't the reader's: the vendor's own documented path is broken on a current system. Each of these looks fine at a glance — the compose file is valid, the tag looks real, the Makefile target reads fine. Four of them failed when we ran them; the fifth, Twenty's, we found in its source files, and we say so below.
Checkmate: the pinned MongoDB won't boot on a current kernel
Upstream's reference docker-compose.yaml pins mongo:8.0, and that tag's current build refuses to start on Linux kernels 6.19 and newer:
MongoDB cannot start: Linux kernel versions 6.19 and newer has a known
incompatibility with this version of MongoDB (SERVER-121912)On a 2026 Ubuntu LTS — the OS a new VPS actually ships — the whole stack dies on dependency mongodb failed to start. And the obvious retag doesn't save you: every MongoDB 8.x build either refuses to start or predates the check that would have refused, because 8.x is what vendors the affected allocator. Our own working fix is MongoDB 7.0. (An earlier version of this report recommended the floating mongo:8 tag; that tag now resolves to 8.3.9, which refuses to start too.) Invisible to any review; fatal on every fresh Ubuntu 26.04 box (no 7.0.14+ kernel ships for it yet).
TeamPass: the official .env.example pins a Docker tag that no longer exists
Follow the official quick start verbatim and docker compose up -d dies with failed to resolve reference … not found: upstream's own .env.example pins TEAMPASS_VERSION=3.1.5.2, a tag Docker Hub has since stopped serving. The compose file's own default (latest) would have worked — the example config is what breaks it.
Twenty: the recommended password generator breaks its own database connection
We didn't hit this one by running it — our snippet generated a hex password from the start. It's from reading upstream's own files: Twenty's .env.example warns that the Postgres password must contain no special characters — but the natural way to generate one (openssl rand -base64) produces + and / — and the compose file pastes the password unescaped into the database URL, where a / ends the credentials and leaves an invalid DSN. Generating the password as hex avoids it; that is what our verified snippet does. A footgun upstream documents and then hands you.
Dokku: the bootstrap script refuses current Ubuntu
Unsupported Linux distribution. Only the following versions are supported:
Debian [11, 12, 13], Ubuntu [22.04, 24.04]Upstream's bootstrap.sh hardcodes a distro allowlist that didn't include Ubuntu 26.04 when we ran it in July 2026. Nothing to work around — the installer exits before doing anything. If you're on a current LTS, Dokku's official path simply said no.
Update, September 2026: fixed upstream. Dokku v0.38.28 (released 2026-09-17) adds 26.04 to the allowlist (dokku/dokku#8791). Our snippet now pins that release, but we haven't re-run it yet — so Dokku carries no install-verified stamp until we do.
Jan: the quickstart's first run looks for its template in the wrong directory
make quickstart in a fresh clone of janhq/server fails inside its own template-copy step:
Error: failed to copy .env template: read template: open .env.template:
no such file or directoryCopying the template explicitly and make up-infra brought Postgres, Keycloak and Kong up healthy, though the full application stack exceeded our harness's 15-minute cap while still pulling images — so we can vouch for the failure, and only partially for the workaround.
Honorable mention — Plausible CE: two boot-crashers from documentation drift
Found in our deploy-guide testing rather than the install-snippet sweep, same disease: current Plausible CE reads env from .env (our guide, written against older CE instructions, said plausible-conf.env — following it leaves every variable blank), and TOTP_VAULT_KEY must now be exactly 32 base64-encoded bytes, where our guide generated 48:
(ArgumentError) TOTP_VAULT_KEY must be Base64 encoded 32 bytes…
Got Base64 encoded 48 bytes.Honorable mention — SigNoz: the documented install path moved out from under us
Our snippet pointed at deploy/docker, which doesn't exist in a current clone: as of v0.130.0 SigNoz replaced the install script and the bundled compose files with a Foundry-based flow. Upstream did handle the move — the same commit that removed the directory added a deprecation notice and turned install.sh into a pointer to the new docs — so this is not a broken installer. It is saved instructions rotting between releases, which is the failure mode this report exists to measure. (Foundry installs foundryctl to ~/.local/bin, which Ubuntu only puts on PATH in a login shell; the installer says so on its way out, and our snippet handles it explicitly.)
Eleven official installers can't run unattended
A separate class isn't broken at all — it just can't be automated. Eleven apps' official installers require a human at a terminal: they prompt, render TUIs, read /dev/tty directly, or run in the foreground forever.
docker run -it wizard; no TTY, no install.-ti flags need a real terminal.generate_config.sh prompts by default, though MAILCOW_HOSTNAME/MAILCOW_TZ env vars make it scriptable — that's how we verified it./dev/tty directly; unattended it loops on an email prompt and segfaults. Driven through a PTY it installs cleanly.createsuperuser wants a TTY unless you pass every argument explicitly.None of this is a defect. But if your plan is cloud-init, Ansible, or a bash-over-SSH bootstrap, these are the apps where that plan meets a prompt it can't answer.
Most of the failures were ours — and that's the point
Here's the uncomfortable stat: of the 27 first-attempt failures, 22 were bugs in our own install snippets — the commands we'd written, reviewed, and published. A missing sudo. A compose file that mounts prometheus.yml nobody ever creates. A snippet that maps 22:22 for Git-over-SSH — a port every VPS's sshd already owns, and following the original comment to move sshd off :22 could have locked a reader out of their own box. Four distinct defects in one four-line Headscale snippet.
We're publishing that number instead of hiding it because it's the strongest argument for the whole exercise: if the people writing the docs get it wrong 23% of the time, the only honest install doc is one that has actually been executed, recently, on the OS it claims to support. That's why an app page carries an install-verified stamp only when a real run backs it, with that run's date and OS — and why an app whose snippet has changed since loses its stamp until it's re-run. See how we test.
What a real run catches — including our rig lying to us
Running installs for real cuts both ways: the test rig itself can lie to you. Six times, our harness produced failures that looked exactly like broken apps and weren't:
- An early runner executed compose YAML as shell for 18 apps and briefly “proved” 27% of the catalog was broken. The true count in that failure class: 2.
- Our reset script stopped systemd-resolved to free port 53 for Pi-hole, killing DNS box-wide; ten apps then “failed” with
could not resolve host. (Pi-hole's documented port-53 fix leaves DNS working — we checked.) - A bootstrap died on an apt lock before it ever added the test user to the
dockergroup — four of five code-hosting apps then “failed” on Docker permissions; none of those failures were real. - Stale state from a previous app — a leftover clone directory, a container still holding port 8000, an old container name — produced three more incidents across waves.
Every one of these was caught by reading the actual transcript instead of trusting the verdict column. It's also why we now assert DNS before every single app, wipe Docker state and the working directory between apps, and verify the reset actually left it clean — and why we treat any new failure as a harness bug until the output proves otherwise.
Method
The harness SSHes into a throwaway GCP VM (e2-standard-2, stepped up to e2-standard-4 for the heaviest stacks; Ubuntu 26.04 LTS), executes each app's published install snippet step by step exactly as a reader would, applies a hard per-app timeout, and resets the box to a verified-clean state between apps — containers, volumes, images, build cache, working directory, and (after one memorable incident with Docker Swarm resurrecting containers we'd deleted) swarm state too. An app passes when every step of its snippet exits cleanly. The harness does not probe the service itself, so where the snippet has no health check of its own we confirmed it answered by hand, and the adjudication notes say so.
Failures were then adjudicated by reading the transcript: upstream defect, our own snippet bug, interactive-by-design, or harness artifact. The fixes now live in the snippets on each app's page, alongside the install-verified stamp. During the August runs we also captured measured first-boot idle RAM for thirteen apps; a 2026-09-28 re-sweep took that to 96. Each figure is on its app page with the machine and date it was measured on.