When running performance tests at scale, collecting and analyzing metrics becomes a significant challenge. This is especially true when running k6 load tests in Kubernetes, where you might have dozens of pods generating thousands of metrics per second. Let’s dive into why this matters and how to solve it effectively.

The Challenge of Scale

Imagine you’re running a load test that simulates 50,000 virtual users hitting your application. To generate this load, you might need anywhere from 5 to 100 k6 pods distributed across your Kubernetes cluster. Each pod is generating its own set of metrics:

• K6 standard built-in
• Resource utilization stats
• Custom business metrics
• Protocol-specific metrics (HTTP, gRPC, WebSocket)
• Browser metrics like web-vitals

Now you’re faced with several challenges:

1. Data Volume: Each pod might generate hundreds of metrics per second
2. Metric Correlation: How do you correlate metrics from different pods?
3. Real-time Aggregation: How do you get a live view of the overall test performance?
4. Resource Overhead: How do you collect metrics without impacting the test itself?

Traditional Approaches and Their Limitations

Teams often start with simple solutions that quickly show their limitations:

1. Log Files

k6 run script.js --out json=test-results.json

• ❌ Difficult to aggregate across pods
• ❌ No real-time visibility
• ❌ Storage becomes a problem at scale

2. StatsD

export const options = {
  statsd: {
    addr: 'statsd-exporter:8125'
  }
}

• ❌ Limited metadata support
• ❌ Network overhead
• ❌ Complex aggregation rules

3. Direct Prometheus Export

annotations:
  prometheus.io/scrape: "true"
  prometheus.io/port: "5656"

• ❌ Scrape configuration complexity
• ❌ Limited to Prometheus format
• ❌ High cardinality challenges

Enter OpenTelemetry: A Modern Solution

OpenTelemetry provides a standardized way to collect, process, and export metrics that solves many of these challenges. It’s become the de facto standard for observability data collection, backed by the Cloud Native Computing Foundation (CNCF) and supported by major cloud providers and observability vendors.

Understanding the OpenTelemetry Architecture

In a Kubernetes environment running k6 load tests, the OpenTelemetry architecture typically consists of several key components:

1. Data Sources (k6 Pods)
   • Generate metrics during load testing
   • Specify output of k6 metrics as OpenTelemetry
   • (Optional) Tag metrics with pod-specific metadata
2. Collectors
   • Receive metrics from multiple k6 pods
   • Process and transform data
   • Handle buffering and retries
   • Manage data routing
3. Backend Storage
   • Time-series databases (Prometheus, InfluxDB)
   • Observability platforms (Grafana Cloud, NewRelic, Datadog)
   • Long-term storage solutions

Why OpenTelemetry for k6?

OpenTelemetry solves several critical challenges:

1. Standardization
   • Common data model across all metrics
   • Consistent metadata handling
   • Unified export protocol
2. Performance
   • Efficient binary protocol (OTLP)
   • Built-in batching and compression
   • Low overhead per metric
3. Flexibility
   • Multiple export formats
   • Plugin architecture
   • Custom processors support
4. Kubernetes Integration
   • Native pod discovery
   • Automatic metadata injection
   • Cluster-aware routing

Implementation Guide

Let’s break down the implementation into manageable steps:

1. Deploy the OpenTelemetry Collector

The OpenTelemetry Collector serves as the central hub for all your metrics. Let’s break down its configuration:

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: k6-otel-collector
spec:
  mode: deployment  # Runs as a Deployment instead of DaemonSet
  config: |
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: "0.0.0.0:4317"  # Standard OTLP gRPC port

    processors:
      batch:
        # Groups metrics into batches to reduce network overhead
        timeout: 10s          # Maximum time to wait before sending a batch
        send_batch_size: 1000 # Number of metrics per batch
      memory_limiter:
        # Prevents OOM by limiting memory usage
        check_interval: 1s    # How often to check memory usage
        limit_mib: 1500      # Maximum memory usage allowed

    exporters:
      prometheus:
        # Exports metrics in Prometheus format
        endpoint: "0.0.0.0:8889"  # Endpoint for Prometheus to scrape
      otlp:
        # Forwards metrics to another OTLP destination (like Tempo)
        endpoint: "tempo.monitoring:4317"

    service:
      pipelines:
        metrics:
          # Defines the flow of data through the collector
          receivers: [otlp]                    # Where data comes from
          processors: [memory_limiter, batch]  # How data is processed
          exporters: [prometheus, otlp]        # Where data goes

Key points about this configuration:

• The collector runs in deployment mode, allowing you to scale it independently
• It accepts OTLP data via gRPC on port 4317
• Implements batching to optimize network usage
• Uses memory limiting to prevent crashes
• Supports multiple export formats simultaneously

2. Configure k6 for OpenTelemetry Export

The k6 test script needs to be configured to work with OpenTelemetry. Here’s a detailed breakdown:

As a user you have 2 options how to enable OpenTelemetry:

1. By defining environment variable K6_OUT
2. By defining cli flag --out, -o You will need to provide the the value experimental-opentelemetry and you are on your way.

K6 OTLP provides us with some sane defaults but if you find yourself needing to tweak the configuration of OpenTelemetry you can refer to the docsor directly in source.

Next you can tweak the attributes of each metric globally or individually for each metric:

export const options = {
  ...
  tags: {
  podName: __ENV.POD_NAME // Metadata can be specified globally for metrics
  }
};

export default function () {
  // Custom metrics are automatically exported
  customMetric.add(1, {
    ...
    podName: __ENV.POD_NAME, // Or can be attached individually
  });
}

Important aspects of this configuration:

• Specify output of k6 tests
• Configure OTLP export
• Adds pod-specific metadata to metrics
• Automatically exports all k6 metrics through OTLP

3. Deploy k6 with OpenTelemetry Support

apiVersion: apps/v1
kind: Deployment
metadata:
  name: k6-load-test
spec:
  replicas: 20
  template:
    spec:
      containers:
      - name: k6
        image: grafana/k6:latest
        args:
        - run
        - --out
        - experimental-opentelemetry
        - /scripts/test.js
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name

Benefits of This Approach

1. Unified Collection: OpenTelemetry handles all metrics consistently
2. Scalability: Built-in batching and rate limiting
3. Flexibility: Export to multiple backends simultaneously
4. Rich Context: Automatic pod and container metadata
5. Standard Protocol: Wide tool support

Best Practices for Production

1. Resource Management

   • Set appropriate CPU/memory limits
   • Use horizontal pod autoscaling
   • Monitor collector performance

2. Data Sampling

   processors:
     probabilistic_sampler:
       sampling_percentage: 10

3. High Cardinality Control

   • Limit custom labels
   • Use appropriate aggregation intervals
   • Consider using exemplars for detailed analysis

4. Monitoring the Monitors

   • Set up alerts for collector issues
   • Monitor data pipeline latency
   • Track metric collection success rates

Real-world Example: Monitoring Dashboard

Here’s a sample Grafana query to visualize aggregated metrics:

sum(
  rate(k6_http_reqs_total{phase="main"}[1m])
) by (status_code, scenario)

This gives you request rates across all pods, broken down by status code and scenario.

Conclusion

Collecting metrics from distributed k6 load tests in Kubernetes doesn’t have to be complex. OpenTelemetry provides a robust, scalable solution that:

• Handles high-volume metric collection efficiently
• Provides rich contextual information
• Integrates well with existing tools
• Scales with your testing needs

By following these patterns, you can build a reliable metric collection pipeline that grows with your performance testing requirements while providing the insights you need to make informed decisions about your application’s performance.