Senior DevOps Engineer - Observability

Senior DevOps Engineer - Observability

We are seeking a Senior DevOps Engineer focused on Observability to own and drive the observability strategy in our AWS Java-centric cloud environment. You will be responsible for designing and managing a code-driven Datadog observability platform that ensures full visibility into Java applications, Kubernetes workloads and AWS containerized infrastructure all while optimizing Datadog costs and eliminating unnecessary overhead.

This role requires a deep understanding of Datadog, Java logging and tracing, AWS observability best practices and cost efficiency techniques. The ideal candidate will work closely with SRE, DevOps and Software Engineers to standardize monitoring, alerting, metrics and tracing while ensuring cost-effective observability solutions.

As a Senior DevOps Engineer, you will set observability standards, lead automation efforts and mentor engineers ensuring all monitoring and Datadog configuration changes are implemented Infrastructure-as-Code (IaC). You will also drive cross-functional collaboration, working across engineering, infrastructure and product teams to deliver scalable and cost-effective observability outcomes.

Key Responsibilities

Observability Architecture & Strategy

• Own and define observability standards for Java applications, infrastructure and Kubernetes workloads
• Design and maintain Datadog observability stacks using Terraform - all configurations must be code-driven
• Define and enforce JSON structured logging, distributed tracing and metric collection best practices for Java applications
• Ensure all microservices have proper observability configurations for logs, traces and metrics
• Implement service-level objectives (SLOs), SLIs, and SLAs to measure application and system reliability
• Automate observability testing as part of CI/CD pipelines, ensuring new deployments include proper monitoring and logging

Java Application Logging, APM & Distributed Tracing

• Enforce log filtering and retention policies to prevent excessive log ingestion and reduce Datadog costs
• Work directly with Java developers to ensure proper logging, tracing and metrics instrumentation
• Integrate OpenTelemetry for Java distributed tracing, capturing request flow across microservices
• Standardize logging practices across services using Logback, Log4j or SLF4J ensuring:
• Structured JSON logs for easy parsing
• Correlation IDs for linking logs to traces
• Error reporting integrates with Datadog alerts
• Implement Datadog APM (Application Performance Monitoring) for Java-based services ensuring deep JVM observability

Datadog Cost Management & Optimization

• Own Datadog cost governance, ensuring observability costs remain within budget
• Monitor Datadog ingestion volumes (logs, traces, and metrics) to prevent overages
• Implement cost-efficient log filtering, retention and sampling policies to reduce unnecessary logging costs
• Automate Datadog usage reporting and integrate cost tracking into SRE Observability dashboards
• Establish automated alerts for unexpected increases in Datadog usage costs and cost contributors
• Advocate for efficient observability by reducing noisy alerts, redundant logs and low-value traces

Incident Response & Reliability Engineering

• Lead incident response efforts, ensuring Datadog alerts are actionable and tuned to minimize noise
• Establish RCA and use Datadog data to identify root causes of application failures
• Work with application teams to optimize their response protocol based on Datadog insights
• Implement Datadog Watchdog (AI anomaly detection) for proactive failure prediction

Kubernetes & AWS Observability

• Deploy Datadog monitoring and logging for AWS-native services: EKS, EC2, Lambda, API Gateway, RDS, DynamoDB, SQS, SNS, Step Functions, VPC Flow Logs
• Optimize Datadog Kubernetes monitoring, ensuring pod-level tracing and resource utilization tracking
• Kubernetes events and autoscaling alerts are configured in Datadog

Leadership & Mentorship

• Proactively drive observability initiatives across teams to ensure alignment, adoption and execution of observability goals
• Act as technical authority on observability, mentoring engineers on Datadog best practices
• Work closely with Java developers, helping them implement efficient logging, tracing and monitoring
• Drive observability as a key SRE function, ensuring every new service is fully instrumented from day one
• Develop internal training programs on Datadog, Terraform-based observability and AWS monitoring

Required Qualifications

• 5+ years of experience in DevOps, SRE, observability, monitoring, development or software engineering roles
• Proficient in Terraform for Infrastructure-as-Code (IaC) – Datadog must be managed as code
• Solid in Datadog including APM, Logs, Metrics, Tracing and Security Monitoring
• Coding, scripting and automation skills in Java, Python, Node, Bash or Go
• Experience integrating observability into CI/CD pipelines (GitLab CI, AWS CodePipeline, GitHub)
• Extensive experience with AWS services and their observability patterns
• Monitoring experience (ELK, Prometheus, Grafana, OpenTelemetry, New Relic, Dynatrace, Sysdig)
• Java development background with experience in:
• Spring Boot, Java microservices architecture
• Java logging frameworks (Logback, Log4j, SLF4J)
• Java APM instrumentation with OpenTelemetry or Datadog APM
• Proficient understanding of JVM performance tuning, GC monitoring, and thread profiling
• Experience implementing distributed tracing for Java applications
• Incident response and on-call experience with a proven track record of reducing MTTR