Observability

This project implements the three pillars of observability, following the pyramid model where each layer builds on the previous.

The Observability Pyramid

The pyramid reflects signal volume and cost: metrics are cheap and always-on; logs are richer but larger; traces are the most detailed and selectively sampled.

Coverage

Cross-Signal Correlation (Grafana)

Monitoring (Prometheus + Grafana)

Components

Accessing Grafana

open https://grafana.local

Credentials:

• Username: admin
• Password: from Vault → vault kv get -field=grafana_admin_password secret/k8s-provisioner/api-keys

  If Vault is disabled: admin123

Accessing Prometheus

open https://prometheus.local

Accessing Alertmanager

open https://alertmanager.local

Recommended Dashboards

Import from grafana.com: Dashboards → Import → Enter ID → Load

Kubernetes

Node

Java/JVM

Java apps need to expose metrics via Micrometer or JMX Exporter

Go

Go apps need to expose metrics via prometheus/client_golang

cert-manager

Logging (Loki + Grafana Alloy)

Components

Storage by Environment

Accessing Logs

1. Open https://grafana.local
2. Go to Explore → select Loki as datasource

LogQL Examples

# All logs from a namespace
{namespace="default"}

# Search for errors
{namespace="default"} |= "error"

# Filter by pod name pattern
{pod=~"nginx.*"}

# Parse JSON logs
{namespace="default"} | json | level="error"

# Count errors per pod over 5 minutes
sum by (pod) (count_over_time({namespace="default"} |= "error" [5m]))

Recommended Log Dashboards

Tracing (OpenTelemetry + Grafana Tempo)

Enabled via components.tracing: "otel-tempo".

Components

Automatic Trace Injection (Zero Code Changes)

Layer 1 — Istio Mesh Tracing (always active)

The Istio sidecar proxy (Envoy) forwards HTTP/gRPC traces to the OTel Collector automatically for every namespace with istio-injection: enabled.

What you get for free:

• Inbound/outbound HTTP spans with latency, status code, method, URL
• Service-to-service call graph (Service Map in Grafana)
• 100% sampling rate (configurable in Telemetry resource)

Layer 2 — OTel Operator (deep instrumentation, opt-in)

For code-level spans (database queries, function calls, custom business logic), annotate your pod:

annotations:
  instrumentation.opentelemetry.io/inject-java: "true"
  instrumentation.opentelemetry.io/inject-nodejs: "true"
  instrumentation.opentelemetry.io/inject-python: "true"
  instrumentation.opentelemetry.io/inject-go: "true"

Sending Traces Manually (SDK)

# From inside the cluster
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector.monitoring.svc:4317   # gRPC
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector.monitoring.svc:4318   # HTTP

# From any node (via DaemonSet hostPort)
OTEL_EXPORTER_OTLP_ENDPOINT=http://<node-ip>:4317

Viewing Traces in Grafana

1. Open https://grafana.local
2. Go to Explore → select Tempo as datasource
3. Search by TraceID, service name, or use the Service Graph

Instrumenting a New Application

For full trace → log correlation (“Log for this span” button in Tempo), your application must satisfy three requirements:

1. Pod label app: <name>

Alloy automatically extracts the app label from pod labels and adds it to every log stream in Loki. Without it, Tempo cannot find the matching logs.

spec:
  template:
    metadata:
      labels:
        app: my-app   # required

2. Log format with traceID=<hex>

Include the active trace ID in every log line using logfmt key traceID. Loki’s derived fields use the regex traceID=(\w+) to create a clickable link to Tempo.

// Go
traceID := span.SpanContext().TraceID().String()
log.Printf("traceID=%s level=info msg=\"handled request\"", traceID)

# Python
import logging
trace_id = trace.get_current_span().get_span_context().trace_id
logging.info(f"traceID={trace_id:032x} level=info msg=handled_request")

// Java (SLF4J + OTel SDK)
import io.opentelemetry.api.trace.Span;

String traceId = Span.current().getSpanContext().getTraceId();
logger.info("traceID={} level=info msg=handled_request", traceId);

3. Set service.name matching the app label

When initialising the OTel tracer, set the resource attribute service.name to the same value as your pod’s app label. Tempo uses this to query Loki with {app="my-app"}.

resource.NewWithAttributes(
    semconv.SchemaURL,
    semconv.ServiceName("my-app"),   // must match pod label app=my-app
)

Full checklist

See examples/otel-demo-app/ for a complete working reference.

Verifying Installation

kubectl get pods -n monitoring -l app=tempo
kubectl get pods -n monitoring -l app=otel-collector

# Check Tempo is receiving traces
kubectl port-forward -n monitoring svc/tempo 3200:3200
curl http://localhost:3200/ready