> The Observability tab is where a service tells you how it's doing. It has nine sections: Monitoring, its traffic, response time, error rate, uptime and resource use over a range…
> Source: https://zedensix.com/homerun/canary/docs/observability · Site index: https://zedensix.com/llms.txt

# Observability

The **Observability** tab is where a service tells you how it's doing. It has
nine sections: **Monitoring**, its traffic, response time, error rate, uptime
and resource use over a range; **Logs**, its live container output; **Log
history**, every line it wrote across deploys, searchable; **Uptime**, its
uptime probes; **Errors**, the exceptions your app reports (see
[Error tracking](https://zedensix.com/homerun/canary/docs/error-tracking)), its failed deployments and the errors
Homerun logged about it; **Traces**, its OpenTelemetry traces (see
[Tracing](https://zedensix.com/homerun/canary/docs/tracing)); **Pipelines**, the CI runs of the repository linked to
it (see [Pipelines](https://zedensix.com/homerun/canary/docs/pipelines)); **Blocked IPs**, the addresses its blocked
paths turned away (see
[Blocked paths and IP bans](https://zedensix.com/homerun/canary/docs/blocked-paths-and-ip-bans#blocked-request-stats));
and **Health**, the healthcheck the running container was created with (see
[Services](https://zedensix.com/homerun/canary/docs/services#health)).

## Monitoring

Pick a range, **Today** (from midnight in your browser's time zone), **7 days**,
**30 days**, **12 months** or **All time**, and the section shows:

- **Requests**, **Avg response time**, **Error rate** (4xx and 5xx responses)
  and **Bandwidth served**, from Traefik's own metrics. Homerun reads them once
  a minute and keeps what each service served that minute; a minute without
  requests stores nothing. The response time is how long Traefik waited on the
  service, so it leaves out the visitor's own network. Only publicly routed
  services have traffic: a service reached only over the Docker network never
  goes through Traefik.
- **Uptime (public)** and **Uptime (network)**: the share of
  [uptime probes](#uptime) that passed in the range, with their average latency.
- **Avg CPU** and **Avg memory**, with their peaks, from the per-minute resource
  samples.

![A service's Monitoring section: its traffic, response time, error rate, uptime and resource use over the range](https://zedensix.com/images/18458056d282e8b796b384181f29f6dc9327967e.webp)

Each number also shows how it moved against the previous period of the same
length: yesterday up to the same time for **Today**, the 7, 30 or 365 days
before for the others (**All time** has nothing to compare with). The change is
green when it's good news, red when it isn't (a slower response time, a higher
error rate, lower uptime, more CPU or memory) and grey when it's neither (more
requests or bandwidth). Error rate and uptime move in percentage points, the
rest by a relative share.

Below them, charts of requests, response time, CPU and memory over the range;
hover one to read a single point. Everything is kept for a year, so **All time**
reaches back at most that far. Traffic starts being recorded when an instance
runs this version: Homerun turns Traefik's metrics on by itself (an entrypoint
inside the Traefik container, never published), which restarts Traefik once.

## Stacks and the whole instance

The same view exists one level up. A stack's **Monitoring** tab covers its
services and every substack's: requests, response time, error rate, bandwidth
and uptime summed over all of them, and CPU and memory summed minute by minute,
so the peak is the stack's real peak rather than the sum of each service's.
**Monitoring** in the sidebar covers every service on the instance, with this
host's own CPU and memory instead of the sum of the services'. Both add a **By
service** table, busiest first, that links to each service's own monitoring. The
sidebar's Monitoring page also has a **Traces** tab, the traces of Homerun's own
background jobs, and for admins a **Blocked IPs** tab (see
[Blocked paths and IP bans](https://zedensix.com/homerun/canary/docs/blocked-paths-and-ip-bans)), a **CrowdSec** tab
(see [CrowdSec](https://zedensix.com/homerun/canary/docs/blocked-paths-and-ip-bans#crowdsec)) and a **Settings** tab
for the OpenTelemetry collector (see [Tracing](https://zedensix.com/homerun/canary/docs/tracing)) and how long log
history is kept (see [Log history](#log-history)).

![A stack's Monitoring tab, with its By service table](https://zedensix.com/images/9370aa2fb8c07391957c5911b507eb0cbf10ec80.webp)

![Monitoring in the sidebar, covering every service on the instance and the host's own CPU and memory](https://zedensix.com/images/967a1d5095563d7e8a17d0c24fdb16bda71d6564.webp)

## Uptime

Every minute Homerun probes each deployed service two ways, and the tab draws
the recent results as two heartbeat strips with an uptime percentage, latency,
and the reason for the latest failure plus hints for fixing it:

- **From the network**: the container's own port, reached over the Docker
  network. It runs the service's [healthcheck command](https://zedensix.com/homerun/canary/docs/services#health) when
  it has one, opens a TCP connection for a database image, and makes an HTTP
  request otherwise.
- **From its hostname**: the public hostname Traefik publishes
  (`<slug>.<base domain>` or the service's main domain). It's skipped for a
  service that isn't DNS-resolvable, and while the base domain is a loopback
  address like `localhost`, since probing it from this machine proves nothing.

![The Uptime section, with a heartbeat strip per probe](https://zedensix.com/images/0153fb19ff603e459c6489bf72aab1b72ab09b38.webp)

Over HTTP any response under 500 counts as up, a 401 or 404 included, and a 5xx
counts as down. Each probe opens a fresh connection and retries a failed
connection once after two seconds, so a dropped or reset connection on one
request isn't counted against the service. Probing pauses while Homerun disrupts
routing on purpose, during its own boot, a Traefik restart and a self-update, so
those don't show up as outages.

A probe that fails three checks in a row (about three minutes) fires the
**Service down** event on any [notification channel](https://zedensix.com/homerun/canary/docs/notifications)
subscribed to it, and **Service recovered** once it has stayed up for 15
minutes. A failure that clears before then sends nothing. A service that keeps
falling over and coming back alerts once, not on every cycle. Results are kept
for a week; **Clear heartbeats** empties the history. Uptime also feeds
[status pages](https://zedensix.com/homerun/canary/docs/status-pages).

Probing is on by default for every service except databases and caches
(Postgres, MySQL/MariaDB, MongoDB, Redis-compatibles, RabbitMQ, Memcached),
which have nothing public to watch; turn it on for one of those if you want a
TCP check. **Turn off** in the Uptime panel's header stops both probes for that
service (the panel then says so), **Turn on** resumes them; the REST API takes
the same switch as `uptimeEnabled` on `PATCH /api/v1/services/:id`.

## Logs

The log panel live-streams a running container's stdout/stderr straight from the
browser: the server pushes each line as the container writes it over a long-
lived HTTP response, no polling and no WebSocket (SvelteKit has no WebSocket
route API; nothing here needs a client-to-server socket anyway). The panel keeps
the newest 5,000 lines, shown or held while paused, so a service that logs fast
can't grow the tab without bound; older output is in
[Log history](#log-history). The Overview tab shows the last few lines of the
same stream once a service has deployed at least once, so recent output is
visible without switching tabs.

Above the stream, **Log volume** draws the last 24 hours of stored output as a
strip of half-hour buckets, like the lanes on the Services page: brighter where
the service logged more, red where a bucket holds a line that reads as an error
(the word error, fatal, panic, exception or critical), and a diamond per deploy.
Hover or tap a bucket for its line and error counts. It reads what
[Log history](#log-history) stored, so it's empty until the worker has shipped
some output.

![The live log panel streaming a container's output](https://zedensix.com/images/9bb975ec930132380c3b5d8e7e014e3d6a1f4500.webp)

## Log history

Docker only keeps a container's output while the container exists, so a redeploy
used to take the previous revision's logs with it. Homerun's worker now copies
every deployed service's output into Postgres as it's written (standalone
containers and every task of a swarm service, on any node), and **Log history**
searches it. It does the same for every workstation on this server
([its Logs tab](https://zedensix.com/homerun/canary/docs/workstations#logs-and-self-test)) and for Homerun's own
containers ([System Logs](https://zedensix.com/homerun/canary/docs/system-logs)), whose history pages work like this
one without the Revision filter:

- **Search**: text the line contains, case-insensitive, or a POSIX regular
  expression with **Regular expression** ticked.
- **Time**: a preset (last 15 minutes to everything kept) or an exact **From**
  and **To**, in your browser's time zone.
- **Revision**: only lines written while one deployment was running. Each line
  carries the deployment that was live when it was written.
- **Replica**: one container, or one swarm task.
- **Stream**: stdout or stderr. Services Homerun deploys run with a TTY, which
  merges the two into stdout.

Results show oldest first, up to 200 lines a page, with **Older** to page back.
**Download** saves everything the current search matches (up to 50,000 lines) as
a text file.

Lines are kept for 7 days by default; an admin sets 1 to 90 days under
Monitoring → Settings → **Log history**, and an hourly pass deletes what's
older. When the worker first sees a container, it copies what Docker still holds
from inside that window, so logs written before you upgraded are there too.

The same search is `GET /api/v1/services/{serviceId}/logs/history`,
`homerun services logs <id> --history` (with `--since`, `--grep`, `--regex`,
`--deployment`) and the MCP tool `search_service_logs`.

## Errors

The **Errors** section has four views: **Issues** and **Board**, the exceptions
your app reports (see [Error tracking](https://zedensix.com/homerun/canary/docs/error-tracking)), **Failed
deployments**, and **Application errors** (persisted warn/error-level app log
lines that mention this service). The **Logs** section shows a "container
currently down" banner below the logs when the container has crashed. A deploy
that reaches running hides the errors logged before it, and **Clear errors**
does the same by hand; a note says how many are hidden and what cleared them,
with a link to show them again. Clearing on either view clears both. If a
service's container was removed outside Homerun (e.g. a manual `docker rm`), the
Logs section shows a distinct "container is gone" banner with a **Resolve**
button instead: click it to clear the stale reference so the service goes back
to its normal never-deployed state and Deploy works again.

![The Failed deployments view of the Errors section](https://zedensix.com/images/e7b876252e8410d2ba0b87107720d585d6fb042f.webp)

## Terminal

A real terminal into the live container, from the browser, only available while
the service is `running`. It runs `bash` when the image has it and `sh`
otherwise, under a TTY sized to the tab, so colours, tab completion, Ctrl-C and
full-screen tools like `vim`, `top` or `htop` work as they would over SSH. It
follows the app's light/dark theme. Open/close events are logged; individual
keystrokes/commands are not (that's a deliberate scope cut, not an oversight,
raw TTY bytes don't map cleanly to discrete commands anyway).

![A terminal into a running nginx container, from the browser](https://zedensix.com/images/659a6a2b3d1a43924f1b6c7695de8a94b4ffaa85.webp)

Leaving the tab (closing it, navigating away, or 15 minutes with no input or
output) ends the session and kills the shell along with everything started from
it, background jobs included: they get `SIGHUP`, then `SIGKILL` a second later
if they're still there. Run something meant to outlive the session as a
[cron job](https://zedensix.com/homerun/canary/docs/scheduling#cron-jobs) or in the image itself, not from the
terminal.