Observability Beyond Metrics: Traces and Logs

The Metrics Illusion

Dashboards full of metrics feel like observability. CPU at 40%, p99 latency at 180ms, error rate at 0.02% — all green. Then users report that checkout is broken. The error rate didn't spike because the errors are happening in a specific path for a specific cohort. The latency didn't spike because the slow requests are failing before they time out. The CPU is fine because the problem is in a downstream service.

Metrics are aggregates. They tell you that something is wrong; they rarely tell you what is wrong or where. The other two pillars of observability — traces and logs — answer those questions.

Distributed Tracing: Following a Request

A distributed trace records the journey of a single request through a distributed system. Each service creates a span representing its work. Spans form a tree, rooted at the entry point.

[Browser Request] 3.2s total
├── [API Gateway] 0.1s
├── [Auth Service] 0.08s
├── [Product Service] 0.05s
└── [Order Service] 2.9s ← bottleneck
    ├── [DB Query] 2.7s ← root cause
    └── [Cache Write] 0.2s

This trace immediately shows that the Order Service's database query is responsible for 84% of the latency. No amount of metrics dashboards would surface this as clearly.

OpenTelemetry Instrumentation

OpenTelemetry is the vendor-neutral standard for traces (and metrics and logs). Instrument once; export to any backend.

import { NodeSDK } from '@opentelemetry/sdk-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';

const sdk = new NodeSDK({
  serviceName: 'order-service',
  traceExporter: new OTLPTraceExporter({
    url: 'http://otel-collector:4318/v1/traces',
  }),
  instrumentations: [
    getNodeAutoInstrumentations()  // Auto-instruments HTTP, gRPC, DB drivers
  ],
});

sdk.start();

Auto-instrumentation captures HTTP calls, database queries, and common framework operations. For custom business logic, add manual spans:

import { trace, SpanStatusCode } from '@opentelemetry/api';

const tracer = trace.getTracer('order-service');

async function processOrder(orderId: string): Promise<void> {
  const span = tracer.startSpan('processOrder', {
    attributes: {
      'order.id': orderId,
      'order.version': '2',
    }
  });
  
  try {
    await validateOrder(orderId);
    await chargePayment(orderId);
    await updateInventory(orderId);
    
    span.setStatus({ code: SpanStatusCode.OK });
  } catch (error) {
    span.setStatus({
      code: SpanStatusCode.ERROR,
      message: error.message
    });
    span.recordException(error);
    throw error;
  } finally {
    span.end();
  }
}

What to Put in Spans

Spans without context are useless. Add attributes that let you filter and understand the request:

span.setAttributes({
  'user.id': userId,
  'user.tier': userTier,              // 'free' | 'pro' | 'enterprise'
  'order.item_count': items.length,
  'order.total_usd': total,
  'experiment.variant': abVariant,    // 'control' | 'new-checkout'
  'feature_flag.fast_path': fastPath, // Boolean flags active for this request
});

These attributes transform traces into queryable data. "Show me all traces where user.tier = 'enterprise' and error = true" becomes a supported query in any trace backend.

Structured Logging: Logs as Data

Unstructured logs are strings:

2026-06-15T14:32:01Z ERROR Failed to charge payment for order 12345: card declined

Searchable, barely. Aggregatable, not at all.

Structured logs are JSON (or key-value pairs):

{
  "timestamp": "2026-06-15T14:32:01.234Z",
  "level": "error",
  "message": "Payment charge failed",
  "service": "order-service",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "00f067aa0ba902b7",
  "order_id": "ord_12345",
  "user_id": "usr_abc",
  "payment_provider": "stripe",
  "error_code": "card_declined",
  "error_message": "Your card was declined",
  "attempt": 2
}

Every field is filterable, aggregatable, and correlatable. "How many card_declined errors did we have in the last hour by user tier?" is now a simple query.

import pino from 'pino';

const logger = pino({
  level: process.env.LOG_LEVEL || 'info',
  redact: ['user.email', 'card.number'],  // Never log PII
});

// Always bind context fields to child loggers
const orderLogger = logger.child({
  service: 'order-service',
  order_id: orderId,
  user_id: userId,
});

orderLogger.error({
  payment_provider: 'stripe',
  error_code: error.code,
  attempt: attemptCount,
}, 'Payment charge failed');

Correlating Logs and Traces

The most powerful observability pattern: include the trace ID in every log line. Then from a trace, you can jump to the logs from that specific request:

import { trace } from '@opentelemetry/api';

// Middleware to inject trace context into logs
app.use((req, res, next) => {
  const spanContext = trace.getActiveSpan()?.spanContext();
  
  req.logger = logger.child({
    trace_id: spanContext?.traceId,
    span_id: spanContext?.spanId,
    request_id: req.headers['x-request-id'],
  });
  
  next();
});

Now when you see an error in a trace, you can find all logs from that exact request in your log aggregator (Loki, Elasticsearch, CloudWatch) by filtering on trace_id.

Log Levels in Practice

Treat log levels as signal/noise contracts:

Level	Use for	Volume
error	Unexpected failures requiring investigation	Low (<1%)
warn	Expected errors, degraded paths, retry success	Medium (1-5%)
info	Key business events (order placed, user registered)	Medium (5-20%)
debug	Request/response bodies, internal state	High (suppress in prod)

Never log at debug level in production by default. Use dynamic log level adjustment (via environment variable or API endpoint) to enable debug logging for specific services during an incident.

Sampling Strategy

Full trace capture of every request is expensive. Use sampling:

import { ParentBasedSampler, TraceIdRatioBased } from '@opentelemetry/sdk-trace-base';

const sampler = new ParentBasedSampler({
  root: new TraceIdRatioBased(0.1),  // Sample 10% of traces
});

// Always sample errors (via custom sampler)
class ErrorAlwaysSampler implements Sampler {
  shouldSample(context, traceId, spanName, spanKind, attributes, links) {
    // Defer to parent; on span end, force-sample if error
    return { decision: SamplingDecision.RECORD_AND_SAMPLED };
  }
}

Tail-based sampling (decide after the request completes, always keeping errors) is ideal but requires a trace collector that can buffer and make the sampling decision at the end. OpenTelemetry Collector supports this via the tailsampling processor.

The three pillars — metrics for alerting, traces for understanding request flows, logs for forensic detail — are complementary. Build all three. Connect them with trace IDs. The goal is that any production incident can be debugged from first principles, starting with a metric alert, drilling into a trace, and reading the relevant logs — all without adding instrumentation after the fact.