Integrating Event Collectors with Databricks Zerobus

Integrating Event Collectors with Databricks Zerobus

Proof of Concept: How we used Grafana Alloy and Zerobus to collect Pod Logs from AWS EKS

Our project goal: How easy is it to integrate an open source telemetry collector for a complicated datasource like AWS EKS pod logs? Traditionally if you wanted to store these types of logs within Databricks for cybersecurity, observability, or development debugging you would need to integrate somehow with AWS S3 (leveraging Cloudwatch or your collector of choice’s cloud storage upload mechanism).

Architecture Diagram

What is Databricks Zerobus?

As described by Databricks, “Zerobus Ingest is a push-based ingestion API that writes data directly into Unity Catalog Delta tables. It is a serverless connector that automatically scales to handle incoming connections. It does not require configuring partitions or managing brokers.”

Some of the benefits of this approach:

• Performant at scale
  • Delivers near real-time data ingestion within as little as five seconds
  • Supports up to 100MB/s throughput
  • Supports over 10GB/s table throughput with concurrent workloads
• Simplifies Infrastructure
  • Eliminates unnecessary data hops through intermediate message buses and/or cloud storage buckets
  • Eliminates unnecessary data duplication while providing a mechanism for “fallback locations” in the event of issues with target table schemas
• Open Standard Architecture
  • Supports OpenTelemetry standards out of the box
  • Data stored in Delta Lake format
  • Processed with Apache Spark

Pre-Requisites

If you are interested in following along and implementing this on your own environment here are some of the tools and configurations you need ahead of time:

• Tools:
  • AWS CLI v2
  • kubectl
  • helm
  • eksctl
• Authentication:
  • AWS account access with permission to create or manage an Elastic Kubernetes Cluster
  • Databricks UC enabled workspace access with permission to create tables and schemas
  • Databricks Service Principal with a pregenerated OAuth client secret
    • This secret is only shown once - save it securely (in production we recommend the use of AWS Parameter Store)

Implementation

Now we can set up an EKS cluster in AWS. This can be done on your terminal command line via eksctl, which is a command-line tool for creating and managing Amazon EKS clusters - or directly in the AWS EKS console. We decided to use the former so we could have more control and also for debugging.

eksctl create cluster --name zerobus-grafana-alloy --region us-east-1 --nodes 2 --node-type t3.medium

Collectors

Now that we’ve covered the setup, we can dive into the more interesting parts. Let’s start with collection. How do we actually get access to the logs that EKS pods are producing? Unfortunately, you can’t directly go into the EKS console and download logs. No, you have to create a log pipeline to retrieve the data. We looked into a couple options that could be viable: the native approach is to use AWS Cloudwatch which (when working with Fargate) would require a log forwarder like Fluentbit. This can be a good option if we want the logs kept native AWS, but for our purposes we want to put data into Databricks for durable storage, centralized querying, schema enforcement, Delta table history, and downstream analytics.

Another viable option that we ultimately ended up going with is a service known as Grafana Alloy. A collector is a component in a logging pipeline that gathers data from a source system and forwards it to a destination. Grafana Alloy is used as the collector here. Grafana Alloy ran inside the EKS cluster, discovered pods, read the container log output, added K8s context, and sent logs to the next stage of the pipeline. The main point is that collectors make log collection continuous.

OTLP + Schemas

The next decision was how the logs should be represented once they leave the collector. For that, we used OpenTelemetry logs and OpenTelemetry Protocol(OTLP). The technical reason to use OTLP is because of its stability. Having a stable schema avoids a plethora of issues popping up due to your data. ETL/ELT jobs are less likely to fail due to columns being modified in some way, it makes it easier for analysts and downstream pipelines to reliably query data, and keeps your data predictable so you need less defensive logic.

A common pattern for unpredictable data and schemas is to land the raw payload into a single VARIANT type column and parse it in the silver table, which we did prior for many of our data ingestion sources. However, this was not the right fit as Zerobus OTLP ingestion expects the target table to use Databricks’ OTEL schema for logs, which is fine in this case because, again, stability already exists so there is no need. In fact, Databricks provides a CREATE TABLE statement for OTEL log ingestion to send data into Delta tables with a very specific schema for the Delta table. The schema itself even has versioning(TBLPROPERTIES ('otel.schemaVersion' = 'v2')) which makes the target schema explicit instead of leaving each source to define its own shape. In our case we set up the logs table which includes severity, body, and resource attributes. Note that while OTEL logs schema defines a common structure, some fields will be null due to different log sources providing different levels of detail. An EKS pod log may include the raw message in body and source context in attributes, while severity_number, severity_text, event_name, service_name, etc. may remain null, but can be populated and enriched in the silver and gold tables.

Check the Databricks OTel documentation page for more information on OpenTelemetry configuration and the exact schema that is needed for the logs table.

Sample Bronze table output showing EKS pod logs collected by Grafana Alloy and ingested into Databricks through Zerobus

Silver table output showing key EKS pod log fields populated and normalized from the Bronze OTEL data

Permissions

Since we’re creating a bronze table, a couple straightforward permissions need to be granted to the application_id of the service principal

%sql

GRANT USE CATALOG ON CATALOG <catalog> TO `<application-id>`;
GRANT USE SCHEMA ON SCHEMA <catalog>.<schema> TO `<application-id>`;
GRANT MODIFY, SELECT ON TABLE <catalog>.<schema>.<table> TO `<application-id>`;

Grafana Alloy Configuration: alloy-values.yaml

A Helm chart is a reusable package for deploying applications to Kubernetes. For this POC, the Grafana Alloy Helm chart created the Kubernetes resources needed to run Alloy in the EKS cluster. Grafana Alloy is configured through a file called alloy-values.yaml which is relatively simple and in that you can configure settings such as batch size, client_id, client_secret, authentication/authorization, etc. Here, we’re also able to set up which catalog, schema(not to be confused with table definition schema), and the bronze table name we want to write.

Zerobus specifications, note that this is not the full configuration. For the full configuration, please check the last section of this blog: Example alloy-values.yaml configuration

otelcol.processor.batch "default" {
  timeout         = "5s"
  send_batch_size = 100

  output {
    logs = [otelcol.exporter.otlp.zerobus.input]
  }
}

otelcol.auth.oauth2 "zerobus" {
  client_id     = "<client_id>"
  client_secret = "<client_secret>"
  token_url     = "https://<databricks_workspace_url>/oidc/v1/token"
  scopes        = ["all-apis"]

  endpoint_params = {
    "resource" = ["api://databricks/workspaces/<databricks_workspace_id>/zerobusDirectWriteApi"],
    "authorization_details" = ["[{\"type\":\"unity_catalog_privileges\",\"privileges\":[\"USE CATALOG\"],\"object_type\":\"CATALOG\",\"object_full_path\":\"<catalog>\"},{\"type\":\"unity_catalog_privileges\",\"privileges\":[\"USE SCHEMA\"],\"object_type\":\"SCHEMA\",\"object_full_path\":\"<catalog>.<schema>\"},{\"type\":\"unity_catalog_privileges\",\"privileges\":[\"SELECT\",\"MODIFY\"],\"object_type\":\"TABLE\",\"object_full_path\":\"<catalog>.<schema>.<bronze_table_name>\"}]"],
  }
}

otelcol.exporter.otlp "zerobus" {
  client {
    endpoint = "<databricks_workspace_id>.zerobus.<aws_region>.cloud.databricks.com:443"
    auth     = otelcol.auth.oauth2.zerobus.handler

    headers = {
      "x-databricks-zerobus-table-name" = "<catalog>.<schema>.<bronze_table_name>",
    }

    compression = "gzip"
  }
}

Deploying Grafana Alloy, applying configuration, and monitoring logs

Add the Grafana helm chart, install it, and apply our configurations:

helm repo add grafana https://grafana.github.io/helm-charts 
# One time setup

# Adds Grafana Helm chart repository locally so that Helm knows where to download the chart from

# Expected result: Helm should confirm that the grafana repo was added

helm install alloy grafana/alloy --namespace monitoring --create-namespace -f alloy-values.yaml 
# First install

# Installs Grafana Alloy into EKS cluster using the Grafana Alloy Helm chart. Release name is Alloy, we deploy it into monitoring namespace, and it uses the setting from the alloy-values.yaml we create

# Expected result: Helm should show that the release was deployed successfully. In EKS, you should see Alloy resources created in the monitoring namespace.

# If you want to update and apply your config changes in alloy-values.yaml later:
helm upgrade alloy grafana/alloy --namespace monitoring -f alloy-values.yaml
# Expected result: Helm should show that the release was upgraded successfully.
kubectl rollout restart daemonset/alloy -n monitoring
# Expected result: Existing Alloy pods should terminate and new Alloy pods should start.

Once deployed monitor the output:

kubectl get pods -n monitoring -w # Watches the pods in monitoring namespace in real-time

# Expected result: Alloy pods should move into a Running state. Since Alloy runs as a DaemonSet, there should typically be one Alloy pod per Kubernetes node.

kubectl logs -n monitoring <pod-name> (optional flag for more recent logs: --since=2m) # Shows logs from a specific Alloy pod.

# Expected result: You should see Alloy startup logs, component initialization logs, and ideally no authentication, permission, schema, or export errors. This command is useful for confirming that Alloy is running and for debugging issues with log collection or export to Zerobus.

Verify that the Alloy pod logs show no config, authentication, permission, schema, or connection errors. Then confirm new rows are landing in the Databricks table.

Troubleshooting Grafana Alloy

When debugging the Alloy pipeline, you can temporarily add the otelcol.exporter.debug block in the configuration file to inspect the log records that Alloy is processing. Because the debug exporter is experimental, ‘stabilityLevel’ must be set to experimental while debugging is enabled. For the production configuration, stabilityLevel should remain generally-available because the debug exporter is not included. For the production configuration, keep stabilityLevel set to the safer default generally-available.

One important detail is the authorization_details block used for Databricks OAuth. Unity Catalog privilege names must use spaces such as “USE CATALOG” and “USE SCHEMA”, not underscores. These values should match the privilege names shown in Databricks SHOW GRANTS output.

If eksctl fails with a token-related error even after aws sts get-caller-identity succeeds, you can fall back to exporting temporary AWS credentials directly from the configured profile. Run aws configure export-credentials --profile <profile_name>, then set AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN in your terminal session before retrying the eksctl command.

The namespace where the collector runs should also be excluded from log discovery. In this POC, Alloy ran in the monitoring namespace, so that namespace was dropped from discovery. This prevents Alloy from ingesting its own logs, which can create noisy feedback loops. When debug logging is enabled, this can cause log records to repeatedly contain previous log records, increasing payload size and potentially exceeding the gRPC message size limit.

For example, if Alloy runs in the monitoring namespace, the discovery relabel rule should include:

rule {
  source_labels = ["__meta_kubernetes_namespace"]
  regex         = "monitoring"
  action        = "drop"
}

Cost Control: Pausing In Between Sessions

Scaling the nodegroup to zero stops EC2 costs while preserving all cluster config (Helm release, Alloy config, Databricks table/grants) so no redeployment is needed on resume.

# Find nodegroup name:
eksctl get nodegroup --cluster=zerobus-grafana-alloy --region=us-east-1

# Pause:
eksctl scale nodegroup --cluster=zerobus-grafana-alloy --region=us-east-1 --name=<nodegroup-name> --nodes=0 --nodes-min=0 --nodes-max=3

# Confirm:
kubectl get nodes

# Wait for nodes to fully disappear — Ready,SchedulingDisabled is a normal in-progress state, not stuck.

# Resume:
eksctl scale nodegroup --cluster=zerobus-grafana-alloy --region=us-east-1 --name=<nodegroup-name> --nodes=2 --nodes-min=1 --nodes-max=3
kubectl get nodes
kubectl get pods -n monitoring

Cost note

The EKS control plane (~$0.10/hr) keeps running even at zero nodes. For pauses longer than a few days, a full teardown is more economical:

eksctl delete cluster --name zerobus-grafana-alloy --region us-east-1

Conclusion

At this point, the EKS pod logs are being collected by Grafana Alloy, shaped as OpenTelemetry log records, and ingested into a Databricks Bronze Delta table through Zerobus. From there, you’re able to write a pipeline to create silver and/or gold tables depending on the analytics use case.

For debugging purposes, you can use the kubectl library as well as setting the stabilitylevel in alloy-values.yaml to experimental and then include debugging in the output of the batch processor section.

We utilized Lakewatch, Databricks’ SIEM platform, and plugged in our existing bronze table logs to create a standardized and easily queryable silver table, using a simple configuration driven setup.

For more information on Lakewatch, info on presets + examples , and more check out: https://docs.lakewatch.com/

Appendix

Example alloy-values.yaml configuration

NOTE: For the configuration below, sensitive values such as the Databricks client ID and client secret should not be hardcoded directly in alloy-values.yaml. In production, store them in a secure secret manager such as AWS Secrets Manager and inject them into the Alloy deployment at runtime.

alloy:
  stabilityLevel: generally-available
  configMap:
    content: |
      discovery.kubernetes "pods" {
        role = "pod"
      }

      discovery.relabel "pod_logs" {
        targets = discovery.kubernetes.pods.targets

        rule {
          source_labels = ["__meta_kubernetes_namespace"]
          regex         = "monitoring"
          action        = "drop"
        }
      }

      loki.source.kubernetes "pod_logs" {
        targets    = discovery.relabel.pod_logs.output
        forward_to = [otelcol.receiver.loki.default.receiver]
      }

      otelcol.receiver.loki "default" {
        output {
          logs = [otelcol.processor.batch.default.input]
        }
      }

      otelcol.processor.batch "default" {
        timeout         = "5s"
        send_batch_size = 100

        output {
          logs = [otelcol.exporter.otlp.zerobus.input]
        }
      }

      otelcol.auth.oauth2 "zerobus" {
        client_id     = "<client_id>"
        client_secret = "<client_secret>"
        token_url     = "https://<databricks_workspace_url>/oidc/v1/token"
        scopes        = ["all-apis"]

        endpoint_params = {
          "resource" = ["api://databricks/workspaces/<databricks_workspace_id>/zerobusDirectWriteApi"],
          "authorization_details" = ["[{\"type\":\"unity_catalog_privileges\",\"privileges\":[\"USE CATALOG\"],\"object_type\":\"CATALOG\",\"object_full_path\":\"<catalog>\"},{\"type\":\"unity_catalog_privileges\",\"privileges\":[\"USE SCHEMA\"],\"object_type\":\"SCHEMA\",\"object_full_path\":\"<catalog>.<schema>\"},{\"type\":\"unity_catalog_privileges\",\"privileges\":[\"SELECT\",\"MODIFY\"],\"object_type\":\"TABLE\",\"object_full_path\":\"<catalog>.<schema>.<bronze_table_name>\"}]"],
        }
      }

      otelcol.exporter.otlp "zerobus" {
        client {
          endpoint = "<databricks_workspace_id>.zerobus.<aws_region>.cloud.databricks.com:443"
          auth     = otelcol.auth.oauth2.zerobus.handler

          headers = {
            "x-databricks-zerobus-table-name" = "<catalog>.<schema>.<bronze_table_name>",
          }

          compression = "gzip"
        }
      }