Kubernetes Cluster Monitoring with Prometheus, Thanos and Long Term Storage

The Challenge of Monitoring Distributed Environments in Kubernetes

Managing modern applications means handling dozens of microservices running simultaneously across multiple virtual machines. In today's infrastructure ecosystem, Kubernetes has become the de facto standard for container orchestration, automating the deployment, scaling, and operation of complex systems. As this infrastructure grows, the volume of generated data regarding CPU consumption, memory, and requests per second explodes exponentially, requiring observability tools capable of keeping pace without consuming the company's entire financial budget.

To collect these metrics, the technology community widely adopted Prometheus, an open-source monitoring system that gathers real-time info and stores it locally in a highly compressed format. In practice, Prometheus acts as an untiring collector that checks every application every few seconds to ask for its current state. This decentralized model works perfectly for smaller environments, but presents an insurmountable Achilles' heel when looking at large corporate operations: local storage and long-term data retention.

The major bottleneck stems from the fact that Prometheus was designed to be fast and autonomous, keeping its data on the hard drive of the machine where it runs. When this disk fills up, older metrics are automatically deleted to make room for new ones, limiting history to a few days or weeks. Furthermore, if the physical machine hosting Prometheus suffers a crash, all accumulated history can vanish instantly, blinding the engineering team precisely when they need to understand the system's past behavior.

The Thanos Architecture for Extended Retention

When business needs demand keeping metrics for six months, a year, or longer—whether for compliance audits or long-term growth trend analysis—the standard Prometheus model falls short. This is exactly where Thanos comes in, a suite of components built to transform Prometheus into a distributed system with high availability and no practical storage limits. In practice, Thanos acts as an extra layer that attaches to existing Prometheus setups without requiring massive rewrites of the infrastructure.

The heart of this solution is a component called the Sidecar, a small program running alongside each Prometheus instance inside the Kubernetes cluster. This Sidecar reads data blocks generated by the local Prometheus continuously and sends them asynchronously to low-cost cloud object storage, such as Amazon S3, Google Cloud Storage, or any S3-compatible service. In practice, this means data gets a permanent and secure address outside the cluster, protected against catastrophic failures of local machines.

Another monumental gain of this architecture is eliminating dependence on expensive, oversized local disks. Instead of buying servers with terabytes of solid-state storage to hold history, the engineering team can maintain smaller, cheaper disks, delegating long-term retention responsibility to an elastic cloud service. This separation of compute and storage drastically reduces operational costs and simplifies disaster recovery, as system state remains isolated and securely persisted in the cloud.

Practical Implementation with Sidecars and Thanos Components

To bring this architecture to life, the first step is deploying the Thanos Sidecar binary in the exact same Kubernetes pod where Prometheus is already running. Below is a simplified manifest example illustrating how to inject the Sidecar and connect it to remote cloud storage using a configuration file:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus-thanos
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus
  template:
    metadata:
      labels:
        app: prometheus
    spec:
      containers:
        - name: prometheus
          image: prom/prometheus:v2.45.0
          args:
            - '--config.file=/etc/prometheus/prometheus.yml'
            - '--storage.tsdb.path=/prometheus'
            - '--storage.tsdb.min-block-duration=2h'
            - '--storage.tsdb.max-block-duration=2h'
        - name: thanos-sidecar
          image: quay.io/thanos/thanos:v0.31.0
          args:
            - 'sidecar'
            - '--prometheus.url=http://localhost:9090'
            - '--objstore.config-file=/etc/thanos/bucket.yml'
          volumeMounts:
            - name: config-volume
              mountPath: /etc/thanos

This technical arrangement ensures that Prometheus keeps writing local metric blocks every two hours and, immediately after a block closes, the Thanos Sidecar packages and uploads that file to the cloud bucket. The bucket.yml configuration file must contain the credentials and chosen cloud provider name, ensuring encrypted and secure communication behind the corporate network curtain.

Once data streams continuously to object storage, the next essential component to deploy is the Thanos Querier. This service acts as a centralized query endpoint that talks simultaneously to multiple Sidecars across different Kubernetes clusters while pulling historical data from long-term storage. In practice, when an engineer opens a Grafana monitoring dashboard to inspect a microsystem's behavior over the past twelve months, the Thanos Querier instantly merges real-time server disk data with old cloud-stored data, delivering a continuous, uninterrupted timeline.

Compaction, Downsampling, and Cost Optimization

Storing years of raw metrics without optimization creates severe network bandwidth consumption and query sluggishness. When a system asks for the average CPU utilization from two years ago, processing millions of individual data points second by second consumes massive memory and compute time. To solve this challenge, Thanos employs an autonomous component called the Thanos Compactor, whose primary job is organizing and streamlining cloud-stored data during off-peak operational hours.

The Compactor performs two fundamental operations: block compaction and downsampling, which means resolution reduction. In practice, downsampling takes data collected every 15 seconds and computes averages and percentiles for larger intervals, such as 5 minutes and 1 hour, preserving chart shape and trends while stripping away thousands of redundant data points. When viewing a whole day's chart, 5-minute resolution is more than enough to spot usage spikes, making responses instant and saving massive computational resources.

This optimization strategy completely alters the economics of maintaining historical data in production environments. Without downsampling, storage costs and query bandwidth consumption would grow uncontrollably as companies add new applications to the cluster. By compacting and downsampling old data, the architecture guarantees high analytical performance without demanding prohibitive investments in cloud infrastructure, perfectly balancing audit requirements with cost efficiency.

Final Considerations and Operational Best Practices

Adopting a long-term monitoring solution based on Thanos and Prometheus radically transforms the operational maturity of a company running Kubernetes. The key takeaway from this journey is that observability should not be treated as a mere system appendix, but rather as a fundamental architectural pillar supporting technical and financial decision-making. Ensuring continuous short- and long-term visibility allows teams to anticipate capacity bottlenecks before they impact end users, dramatically increasing overall technology operation resilience.

However, keeping this architecture healthy requires ongoing discipline from platform and reliability engineering teams. It is essential to monitor the monitoring tool itself—setting up alerts for Sidecar upload failures, disk space overflows on Prometheus nodes, and cloud network bottlenecks. With a solid foundation, automated backup processes, and clear retention policies, organizations gain total autonomy to scale microservices with absolute confidence, knowing no critical event will slip past the watchful eyes of observability.