Telemetry

Both runtime.New (the web process) and runtime.NewQueue (the worker) build telemetry before the database, using pkg/telemetry. Logs go through log/slog; traces and metrics use OpenTelemetry, and every exporter is started and stopped with its process.

Build the provider

internal/runtime/newTelemetry translates config.Telemetry into options:

 1func newTelemetry(
 2    ctx context.Context,
 3    appCfg config.App,
 4    cfg config.Telemetry,
 5) (*telemetry.Telemetry, error) {
 6    opts := []telemetry.Option{
 7        telemetry.WithTraceSampleRate(cfg.TraceSampleRate),
 8        telemetry.WithBatchConfig(
 9            cfg.BatchSize,
10            time.Duration(cfg.BatchTimeoutMs)*time.Millisecond,
11            2048,
12        ),
13        telemetry.WithLogLevel(cfg.LogLevel),
14    }
15    if !appCfg.IsProduction() {
16        opts = append(opts, telemetry.WithConsole())
17    }
18
19    headers := telemetry.ParseHeaders(cfg.OtlpHeaders)
20    if cfg.OtlpLogsEndpoint != "" {
21        opts = append(opts, telemetry.WithOTLPLogs(cfg.OtlpLogsEndpoint, headers))
22    }
23    if cfg.OtlpTracesEndpoint != "" {
24        opts = append(opts, telemetry.WithOTLPTraces(cfg.OtlpTracesEndpoint, headers))
25    }
26    if cfg.OtlpMetricsEndpoint != "" {
27        opts = append(opts, telemetry.WithOTLPMetrics(cfg.OtlpMetricsEndpoint, headers))
28    }
29
30    return telemetry.New(ctx, cfg.ServiceName, cfg.ServiceVersion, opts...)
31}

Each OTLP endpoint is optional: an empty endpoint disables that signal. WithConsole writes human-readable logs and completed-span lines to stdout outside production. The runtime adds tel.Shutdown to its lifecycle.Stack, so exporters flush during shutdown.

Instrument HTTP, SQL, and the queue

Pass telemetry to the router, the database, and the River logger:

1router := anduin.NewRouter(
2    anduin.WithHosts(hosts),
3    anduin.WithErrorHandler(handlers.MapError(renderer)),
4    anduin.WithTelemetry(tel),
5)
6
7db, err := storage.NewPostgres(ctx, cfg.Database, storage.WithTelemetry(tel))

anduin.WithTelemetry wraps the handler with otelhttp and installs the telemetry middleware, which records the stable route pattern on spans and metrics. Request metrics come from otelhttp, not a separate API. Using the route template keeps cardinality bounded.

Configure exporters

TELEMETRY_SERVICE_NAME=andurel-site
TELEMETRY_SERVICE_VERSION=1.0.0
LOG_LEVEL=INFO
TRACE_SAMPLE_RATE=1.0
TELEMETRY_BATCH_SIZE=512
TELEMETRY_BATCH_TIMEOUT_MS=5000
OTLP_LOGS_ENDPOINT=
OTLP_METRICS_ENDPOINT=
OTLP_TRACES_ENDPOINT=https://collector.example.com
OTLP_HEADERS=Authorization=Bearer token

TELEMETRY_SERVICE_NAME defaults to a slug of the project name. TRACE_SAMPLE_RATE must be between 0 and 1; batch size and timeout must be positive; an invalid value fails startup. OTLP_HEADERS is parsed as comma-separated key=value pairs.

Instrument application work

Use telemetry.Start for a span and telemetry.Fail to record and return an error:

1ctx, span := telemetry.Start(ctx, "queue.delete_expired_tokens")
2defer span.End()
3
4deleted, err := w.tokens.DestroyExpired(ctx, time.Now())
5if err != nil {
6    return telemetry.Fail(ctx, err)
7}
8
9telemetry.Info(ctx, "deleted expired tokens", "tokens.deleted", deleted)

• telemetry.Start(ctx, name, args...) starts a span and attaches slog-shaped attributes; telemetry.From(req, name) does the same from an HTTP request.
• telemetry.Set(ctx, args...) adds attributes to the current span.
• telemetry.Info, Warn, Error, and Debug write through the process logger.
• telemetry.Fail(ctx, err) marks the span, records the error, and returns the error unchanged.

Never put passwords, tokens, session ids, full email addresses, or unbounded user input into logs or span attributes. Log the user id, or only the email domain when a correlation key is needed. Prefer stable route templates and operation names over raw URLs.