Telemetry contract

From plat5/docs at v0.6.0

This page is the contract, copied from docs/telemetry.md. Where it and the rest of this site disagree, this page is right.

Logs, traces, and metrics for Plat5 services. Plat5 does not ship a collector. Operators bring any OTLP-compatible collector (or scrape Prometheus) — Alloy, otel-collector, Grafana Cloud OTLP, etc.

Standard (summary)

Signal	Always	Opt-in
Logs	JSON to stdout	— (no OTLP logs)
Metrics scrape	Prometheus /metrics on the internal port	—
Traces OTLP	—	When endpoint is set and traces exporter includes otlp
Metrics OTLP	—	When endpoint is set and metrics exporter includes otlp

Defaults when an OTLP endpoint is set:

• Traces → OTLP on (OTEL_TRACES_EXPORTER unset defaults to otlp)
• Metrics → OTLP on (OTEL_METRICS_EXPORTER unset defaults to otlp)
• /metrics scrape → still on

Set OTEL_METRICS_EXPORTER=prometheus (or none) to push traces only and scrape metrics.

Do not scrape /metrics into the same backend you also feed via OTLP metrics for the same series.

Configuration

Prefer standard OpenTelemetry environment variables for exporters and destinations. Do not invent project-specific exporter endpoint vars as the primary API.

Destination

Variable	Purpose
OTEL_EXPORTER_OTLP_ENDPOINT	Collector base URL (e.g. http://localhost:4318). Unset = no OTLP destination.
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT	Optional full traces URL (overrides base + /v1/traces)
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT	Optional full metrics URL (overrides base + /v1/metrics)
OTEL_EXPORTER_OTLP_PROTOCOL	http/protobuf only (OTLP/HTTP). grpc is not supported.
OTEL_EXPORTER_OTLP_HEADERS	Optional headers (e.g. Grafana Cloud auth). Honored by the OTLP exporter SDKs.
OTEL_EXPORTER_OTLP_*_HEADERS	Per-signal header overrides when needed

Endpoint = where to send OTLP. When set, traces and metrics both default to OTLP push.

Exporter selection

Variable	Default	Meaning
OTEL_TRACES_EXPORTER	otlp if endpoint set, else effectively none	Comma-separated. OTLP traces only if value includes otlp and a traces destination exists.
OTEL_METRICS_EXPORTER	otlp if endpoint set	Comma-separated. OTLP metrics only if value includes otlp and a metrics destination exists. /metrics stays up regardless (see below).
OTEL_SDK_DISABLED	unset	true → no OTLP export. Stdout logs and /metrics remain.

OTEL_METRICS_EXPORTER	OTLP metrics	/metrics
unset (default)	on (needs endpoint)	on
otlp	on (needs endpoint)	on
otlp,prometheus	on	on
prometheus	off	on
none	off	on

none means no metrics push, not “disable scrape.”

Resource identity

Standard OTel (SDK env spec):

Variable	Purpose
OTEL_SERVICE_NAME	Resource service.name (takes precedence over the same key in OTEL_RESOURCE_ATTRIBUTES)
OTEL_RESOURCE_ATTRIBUTES	Comma-separated key=value resource attributes

Example:

OTEL_SERVICE_NAME=gateway
OTEL_RESOURCE_ATTRIBUTES=service.namespace=edge,service.version=1.2.3,service.instance.id=gw-1,deployment.environment=dev

Project convenience (not dedicated OTel env vars; services also read these and map them onto the same resource attributes — useful in compose):

Variable	Maps to
OTEL_SERVICE_NAMESPACE	service.namespace
OTEL_SERVICE_VERSION	service.version
OTEL_SERVICE_INSTANCE_ID	service.instance.id (fallback: HOSTNAME)
DEPLOYMENT_ENV / OTEL_DEPLOYMENT_ENV	deployment.environment

If both OTEL_RESOURCE_ATTRIBUTES and a convenience var set the same attribute, the convenience var wins. Prefer OTEL_RESOURCE_ATTRIBUTES for portable operator config; convenience vars are fine for our compose defaults. All services honor both.

DEPLOYMENT_ENV only sets this telemetry attribute; it does not change Auth behavior. Auth’s dev-only conveniences (POST /dev/token, logging login codes without SMTP) are enabled only by AUTH_DEV_MODE=true.

Other

Variable	Purpose
OTEL_TRACES_SAMPLER_RATIO	Trace sampling ratio when simple ratio sampler is used
OTEL_METRIC_EXPORT_INTERVAL	OTLP metric export interval (ms)

Default Plat5 compose does not set OTEL_EXPORTER_OTLP_ENDPOINT.

Operator recipes

Full OTLP push (e.g. Grafana Cloud / local LGTM kit)

export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
# traces + metrics OTLP on by default when endpoint is set
# Do not also scrape /metrics into the same metrics backend

Traces push + metrics scrape (no double count)

export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
export OTEL_METRICS_EXPORTER=prometheus   # opt out of metrics push
# scrape http://<host>:<internal_port>/metrics

Scrape only (no traces)

Leave OTEL_EXPORTER_OTLP_ENDPOINT unset. Scrape /metrics. Collect stdout logs in the platform.

Local all-in-one collector

docker run --rm -p 3000:3000 -p 4317:4317 -p 4318:4318 \
  --name otel-lgtm grafana/otel-lgtm:latest

Point services at http://localhost:4318 (host) or http://host.docker.internal:4318 (containers → host). Any OTLP collector on :4318 works (not part of this product).

Logs

Write JSON to stdout always (including when OTLP is enabled). No OTLP log export.

{"timestamp":"2024-01-15T10:30:00Z","level":"info","message":"request completed","route":"/widgets","method":"GET","status":200,"duration_ms":12.5}

Common fields

Field	Description
timestamp	ISO8601
level	debug, info, warn, error
message	Log message
route	HTTP route pattern
method	HTTP method
status	HTTP status code the client received (for a returned error, the status the error handler writes)
duration_ms	Request duration
request_id	Correlation ID
trace_id / span_id	When a span is active
user_id	Authenticated user ID (when known)
organization_id	Organization ID (when in org context)
member_id	Member ID (when in org context)

Field names: naming-conventions.md.

Gateway request line

The gateway writes one request completed (or request failed) line per request. On top of the common fields it says who presented the request. Each field is present only when known:

Field	When	Value
auth_type	A credential was recognized	jwt, user_apikey, member_apikey, member_session
key_prefix	An X-API-Key of a known shape was presented, valid or not	Identity’s display prefix: the wire prefix plus the first 4 characters after it, e.g. plat5-k-2-AbC9. Same value as key_prefix on identity’s key list (or a session’s token_prefix), so a line can be matched to a key
user_id	Admitted on a user route (JWT or user key)
organization_id, member_id	Admitted on an organization or member route (member key or session)

key_prefix is set before validation, so 401 and 403 lines carry it. A value of an unknown shape, or with nothing after the preview, is not logged. The key or token itself, and a JWT, are never logged. The line has no labels or role.

{"timestamp":"2026-10-08T18:30:00Z","level":"info","message":"request completed","request_id":"…","route":"/widgets","method":"GET","status":403,"duration_ms":3.1,"auth_type":"member_apikey","key_prefix":"plat5-k-2-AbC9","organization_id":"01J…","member_id":"01J…"}

Error fields (5xx only)

Field	Description
error_kind	auth, network, db, io, internal, validation
error_message	Human-readable message

Do not include error_kind for 4xx.

Traces

• Export via OTLP only (no Jaeger/Zipkin native APIs from the app).
• Propagate W3C Trace Context only (no Baggage).
• Resource attributes on every export: service.name, service.namespace, service.instance.id, service.version, deployment.environment.

HTTP server spans

HTTP span conventions for the names we emit. Do not emit deprecated names (http.method, http.status_code, http.target, http.url, http.scheme). Do not dual-write old and new.

Kind	SERVER
Name	{method} {http.route} when a route template exists; otherwise {method} only. Never the raw URI.
Attributes	http.request.method, url.path, http.route (template only, when matched), http.response.status_code (when known)
Status	Unset on 4xx. Error on 5xx with no description. Unset on 1xx/2xx/3xx unless a non-HTTP error occurred.

http.route is the matched template (/organizations/:organization_id/subscribers). Do not put the raw path there. url.path is the actual path (IDs allowed).

Do not set url.query, url.scheme, or error.type. Frameworks may add extra stable HTTP attributes; do not copy those onto other services.

Gateway creates the root span before route match: name starts as {method}, then becomes {method} {template} after match. Unmatched stays {method}. Internal child spans (gateway.request_filter) are not HTTP server spans.

Plat5 span attributes (not HTTP semconv)

• request_id on HTTP request spans (if present). Do not rename it to request.id.
• user.id on HTTP request spans when the route subject is the person — set at the gateway for ops. Gateway may also set organization.id / member.id / jwt.kid. Spans may record ids the route subject does not include. A service does not invent user.id.
• error.kind on error spans (5xx only): auth, network, db, io, internal, validation
• 4xx responses are normal business outcomes — do not set error.kind and do not mark the span as failed
• Record exceptions via span.recordException(err) / span.record_exception(err)

Logs and Prometheus metrics are not HTTP spans: log fields stay snake_case (request_id, method, status); metric names stay http_requests_total / http_request_duration_seconds.

Metrics

Scrape (always)

Expose Prometheus text format at /metrics on the internal port (not public ingress). See health-checks.md / service READMEs for ports. Optional discovery labels: container-labels.md (metrics.port, metrics.path).

OTLP (default when endpoint set)

When a metrics destination is set and OTEL_METRICS_EXPORTER is unset or includes otlp, push the same logical series via OTLP (same base endpoint → /v1/metrics unless overridden).

Cardinality

Never put high-cardinality values in metric labels (user IDs, request IDs, emails). Those belong on spans and logs.

Naming

• snake_case
• Prefix by domain: http_, db_, process_, auth_
• Suffix by type: _total, _seconds, _bytes

Metric	Type	Labels
http_requests_total	Counter	route, method, status
http_request_duration_seconds	Histogram	route, method
db_operations_total	Counter	db_system_name, db_operation_name, db_namespace
db_operation_errors_total	Counter	db_system_name, db_operation_name, db_namespace
db_operation_duration_seconds	Histogram	db_system_name, db_operation_name, db_namespace

Gateway audit writes (audit.md):

Metric	Type	Labels
audit_writes_total	Counter	write (intent, outcome), result (ok, failed, not_found, dropped)
audit_intent_duration_seconds	Histogram	result (ok, failed). Time an audited request waited on its intent, retries included

Audit service: audit_intents_total (result created, existing) and audit_outcomes_total (result applied, already_final, not_found).

An outcome dropped or failed leaves its event pending.

Histogram buckets (same across all services):

Metric	Boundaries (seconds)
http_request_duration_seconds	0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5
db_operation_duration_seconds	0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1

Do not include service_name / service_namespace in metric series if your collector already adds them from resource attributes or container labels.

Process metrics (minimum)

Always emit on the scrape path (language standard collectors preferred):

Metric	Notes
process_resident_memory_bytes	RSS
process_cpu_seconds_total	Cumulative user+system CPU
process_start_time_seconds	Unix epoch start time

When OTLP metrics are enabled, the same series should appear there too. Extra process/runtime series (heap, FDs, etc.) are optional.

Service expectations

Service	Namespace	Notes
gateway	edge	Internal /metrics; gateway sets edge span identity attrs
route-registry	edge	Internal /metrics
identity	identity	Internal /metrics

Non-goals

• Shipping or requiring a collector / Alloy config as part of the product
• OTLP log export from apps
• App push to Loki, Tempo, or Mimir native APIs
• Public exposure of /metrics
• Custom non-OTEL_* env vars as the primary telemetry API
• HTTP metric semconv (http.server.request.duration) — scrape/OTLP series stay http_requests_total / http_request_duration_seconds

References

• OTel SDK environment variables
• OTLP exporter configuration
• HTTP span semantic conventions
• Semantic conventions
• grafana/otel-lgtm