TIL — Today I Learned
Short things I learned, discovered, or found interesting.
§Fail2ban + push notifications = self-DoS
Push notification services (ntfy, Gotify) behind a reverse proxy with fail2ban create a self-DoS loop:
- Persistent subscribers maintain long-lived connections
- Any disruption (Caddy reload, auth blip) triggers reconnection bursts
- Fail2ban sees the burst → bans your IP
- Auto-unban → instant re-ban (subscriber retries immediately)
-
bantime-incrementmakes each cycle worse
The fix: exclude the notification service from fail2ban entirely:
# In caddy-auth filter
ignoreregex = ^.*"host":"ntfy\.sbr\.pm".*$
ignoreIP doesn't help if your IP changes (travel, mobile). The service itself must be excluded from the filter.
§Remote NixOS install: don't bind-mount over /nix/store
When installing NixOS from a live USB, /nix/store is an overlayfs with a tmpfs upper
layer (RAM-backed). If the closure is too large, you might be tempted to bind-mount the
target disk's store over /nix/store.
Don't. It hides all system binaries and breaks the installer.
Instead, resize the tmpfs:
mount -o remount,size=28G /nix/.rw-store
Or better: build the closure on a remote machine and nix copy it over.
The live USB's /nix/store has a read-only squashfs lower layer (.ro-store) and a tmpfs upper layer (.rw-store). Default tmpfs is ~50% of RAM.
§Paperless: let it auto-detect, don't "optimize"
When migrating paperless-ngx to a more powerful machine (RK3588), I initially set restrictive "ARM64 optimizations" — fewer workers, single threads, OCR first page only.
This was wrong. The RK3588 has 8 cores and 32GB RAM. The defaults are already optimized: paperless auto-detects CPU count and sets workers/threads accordingly.
# DON'T do this on capable hardware:
PAPERLESS_TASK_WORKERS = "2"; # Limiting!
PAPERLESS_THREADS_PER_WORKER = "1"; # Limiting!
PAPERLESS_OCR_PAGES = "1"; # Incomplete!
# DO this: just let paperless figure it out
# (remove all worker/thread overrides)
§Always use explicit refspecs for git push
Bare git push without a refspec uses the branch's tracking configuration, which
can silently push to the wrong branch — especially with worktrees or branches created
from unexpected bases.
Always use:
git push origin branch:branch
Never:
git push # ← dangerous
With worktrees, tracking configs can diverge between the main repo and worktree checkouts. A bare git push in a worktree might push to a completely different remote branch than you expect.