Skip to main content

Troubleshooting

Troubleshoot by request path first: Dash startup, reverse proxy, node reporting, historical data, traffic accounting, and Windows updates are separate paths.

Dash Does Not Start​

View recent logs:

journalctl -u dash.service -n 200 --no-pager

Common causes:

  • Config file was not found.
  • app.listen is missing.
  • app.public_url is missing.
  • app.public_url contains a path prefix.
  • auth.jwt_signing_key is missing.
  • monitor_dash_pwd is missing or contains invalid characters.
  • PostgreSQL connection failed.
  • PostgreSQL goose_db_version does not match the binary.
  • notify-config.key is missing, too permissive, or does not match channel ciphertext.
  • Redis connection failed, ACL denies PING/INFO server, or Redis is below 6.2.0.
  • TimescaleDB is unavailable.

Validate migration manually:

env DASH_HOME=/opt/Ithiltir-dash \
  /opt/Ithiltir-dash/bin/dash migrate \
  -config /opt/Ithiltir-dash/configs/config.local.yaml \
  -debug

Check Redis separately with dash check-redis --addr 127.0.0.1:6379. Add --password-file <path> when required; the file must not have a trailing newline. This command does not read the Dash config file. A schema behind the binary must be migrated; a schema ahead of the binary cannot be used with that old binary.

Dash Update Requires Recovery​

When status contains recovery_required, run:

sudo /opt/Ithiltir-dash/bin/dash update recover

Before migration, recovery can restore the old release. After migration starts, recovery completes forward. Do not delete transaction.env or update.block manually.

Page Loads but API Fails​

Check the reverse proxy:

curl -i https://dash.example.com/api/version
curl -i https://dash.example.com/theme/active.css
curl -I https://dash.example.com/deploy/linux/install.sh

If / works but /api returns 404, the reverse proxy is probably forwarding only the SPA and not backend paths.

Node Does Not Come Online​

Node service state:

systemctl status ithiltir-node.service
journalctl -u ithiltir-node.service -n 200 --no-pager

Check target config:

/var/lib/ithiltir-node/current/ithiltir-node report list

Check Dash report endpoint reachability:

curl -i https://dash.example.com/api/node/identity -H 'X-Node-Secret: <secret>' -d '{}'

Common causes:

  • Secret mismatch.
  • Target URL is not /api/node/metrics.
  • app.public_url is not reachable from the node.
  • HTTPS certificate problem.
  • Reverse proxy does not forward /api/node/*.
  • System time is not synchronized.

/metrics Returns 503​

Local mode may return 503 before the first successful sample. Wait one collection interval.

./node local 127.0.0.1 9100 --debug
curl http://127.0.0.1:9100/metrics
curl http://127.0.0.1:9100/static

Historical Metrics Are Missing​

Check:

  • Whether the user is logged in.
  • Whether history_guest_access_mode is by_node.
  • Whether node is_guest_visible is true.
  • Whether the node already has historical data.

Daily Traffic API Returns 409​

/api/statistics/traffic/daily requires:

usage_mode=billing

Otherwise it returns:

{ "code": "traffic_daily_requires_billing", "message": "daily traffic requires billing mode" }

Manual traffic rebuild also requires Billing; Lite returns traffic_rebuild_requires_billing. Rebuild state resets to idle after Dash restart.

P95 Is null​

P95 fields have values only when p95_status=available. Common states:

  • disabled: P95 is disabled for the node.
  • lite_mode: global traffic mode is not billing.
  • insufficient_samples: not enough samples.
  • snapshot_without_p95: the monthly snapshot has no P95 value.

Windows Node Does Not Update​

Confirm the service starts the runner:

Get-Service ithiltir-node

Running ithiltir-node.exe push directly does not apply updates. The runner sets ITHILTIR_NODE_RUNNER=1 and replaces the binary.

Notification Channel Cannot Be Edited​

An invalid stored channel remains listed with config=null and can only be deleted and recreated. If Dash itself cannot start, restore the configs/notify-config.key that matches the PostgreSQL backup; never overwrite it with a new key.