Application Engineer (Site Reliability)

Summary

The successful candidate will build and operate the AI platform that the rest of the firm consumes, i.e. the gateway to model providers, the workflow and agent runtime, the enterprise AI portal, and the experimentation environment used by the organization's investment research.

The candidate is responsible to build the automation, infrastructure, observability and release discipline that stops the same class of problem recurring.

He or she will be the release manager, i.e. owning scope, coordination and change records for what goes live. The role will lead the platform carrying more services than it started with, and less manual effort holding it up.

Skillset (Must have)

• Possess a degree in Computer Science/Information Technology or related fields.
• Hands-on experience in Production Kubernetes, ideally Amazon EKS. Able to diagnose why a pod is failing, why a rollout is stuck, or why a node is under pressure, without a runbook.
• Strong practical experience in Infrastructure as Code (IaC), ideally using Terraform. Experience in composing and applying shared modules, managing state across environments, reading a plan and knowing what it will actually do before applying it, and recovering when an apply fails part-way. The candidate does not need to have authored a reusable module library (that is owned centrally here), but need to be genuinely comfortable extending existing IaC.
• Experience in Core Hyperscaler (AWS, Azure, or GCP) in production, ideally AWS - networking/VPC, IAM, managed Kubernetes, managed relational databases, object storage, and key management with a real understanding of least-privilege access.
• Experience in release and change ownership. The candidate has personally owned releases into a controlled production environment: scope, notes, coordination with dependent teams, a change record and a rollback plan. This is a must-have requirement, and be able to build a pipeline and owns what goes through it.
• Experience in CI/CD pipeline ownership, building and maintaining pipelines, and not only consuming them.
• Experience in observability design judgement. Able to argue why one metric deserves an alert and another does not, and to describe both an alert that is fought to add and deliberately deleted.
• Working knowledge of SLOs and error budgets. The candidate does not need to have owned an error-budget policy, but should have worked somewhere that ran on one and be able to explain what it changed about how the team made decisions.
• Experience in scripting and automation. Able to write and maintain tooling others rely on, beyond shell one-liners.
• Possess genuine incident experience. Able to walk through a production incident that personally responded to: what being seen first, how to narrow it down, what get wrong on the way, and what changes afterwards.
• Experience in Linux and container debugging fundamentals, processes, networking, DNS, TLS, and resource limits.
• Possess positive learning and collaborative mindset.
• Strong analytical, problem-solving and troubleshooting skills.
• Good written and verbal communication skills.
• Agile, fast learner and able to adapt to changes.

Skillset (Good to have)

• Datadog specifically: monitor design, SLOs, APM, and controlling alert noise.
• Authoring Helm charts rather than only editing values files.
• Data migration as part of a release - planning, reversibility, and verification.
• Chaos or fault-injection experience - AWS Fault Injection Service, or comparable practice.
• Running OpenSearch, or Temporal, as operational services.
• Access and identity provisioning at enterprise scale (AD groups, entitlement workflows).
• AWS cost visibility and optimization, including token-level cost attribution.
• Prior experience in operating AI or LLM workloads, such as inference capacity, provider rate limits, unusually long request lifetimes.
• Operating an API gateway at high throughput - Kong, Envoy, AWS API Gateway or similar.