Principal Site Reliability Engineer, Infrastructure Observability

KM5

Role Summary

In this role as Principal Site Reliability Engineer, Infrastructure Observability you will help formulate, develop, and implement a team of Site Reliability Engineers (SREs) focused on the observability, sustainability, scalability, measurability and recoverability of T. Rowe Price’s innovative cloud & on-prem solutions by leveraging automation and best-of-breed tools. The successful candidate will have a strong operations & engineering background, is hands-on when needed, and has expertise in the cloud environments (public, private), infrastructure operations, DevOps practices, CI/CD toolchain and systems, code build and deployment, incident response, and 24x7 monitoring and support.

The candidate will also have extensive experience operating within a SRE function within a complex, distributed environment. They will have a demonstrated ability to work horizontally and vertically within an organization with diverse partners and sponsor groups.

Responsibilities

• Possesses extensive knowledge in own area of expertise and extensive in-depth knowledge of the broader portfolio for comprehensive understanding of up/downstream impacts across technology infrastructure
• Responsibility for the design of technology solutions to prevent or minimize service disruptions
• Prevents technology service disruptions through technology solution recommendations and automations
• Fosters a culture of deep learning through blameless post-mortems to improve the shared goal of reliability across services
• Transform operations teams by facilitating internal change to adopt SRE standard methodologies across the organization and driving strategic growth in this area within Global Technology
• Analyzes incidents impacting technology availability for high-level trends across the broad portfolio
• Drive initiatives to reduce or prevent technology failures in a complex, distributed technology environment
• Pulls together information from disconnected systems into cohesive views of the technology portfolio for identifying trends, redundancies, and risk
• Demonstrates outstanding awareness of the complexities of the tech and asset management industries
• May lead initiatives of varying degrees of complexity that span multi-functional areas and of varying degrees of complexity
• Contributes to definition of target state architecture and design of the technology environment

Qualifications

Required

• Bachelor's degree or the equivalent combination of education and relevant experience AND 10+ years of experience designing and operating cloud infrastructure with senior‑level impact.
• 5+ years building and supporting solutions in Amazon AWS
• 5+ years of experience building and running a DevOps and/or SRE function
• Experience with implementation and operation of the chaos model at scale
• Strategic and program-level implementation experience
• Demonstrable experience implementing new technology, tools, and platforms
• System administration and scripting experience
• Demonstrable experience leveraging automation to proactively prevent or quickly remediate incidents
• Fluent in multiple programming languages (e.g., Python, Java, GO, Node.js, .Net Core, etc)
• Proficiency with database development (SQL Server, PostgreSQL, MySQL, etc)
• Proficiency with defining, right-sizing, tracking, and reporting on Service Level Objectives (SLOs), Service Level Indicators (SLIs), system availability, and the progress and outcomes related to reliability
• Experience with implementing and managing Error Budgets
• Proficiency with understanding and explaining incident situations and their recovery plans to prevent recurrence
• Knowledge/experience driving dashboard standardization across the ecosystem for observability, APM and infrastructure monitoring, and application-specific logging
• Knowledge/experience with observability tools such as New Relic, SolarWinds DPA, Elastic Stack, Prometheus, Grafana, Splunk, and cloud native tools
• Knowledge/experience with cloud management tools such as Ansible, Terraform, Vault, and Vagrant
• Works independently, with guidance in only the most complex situations
• Makes sound decisions with limited facts or resources
• Balances strategic and pragmatic concerns when solving problems
• Adjusts communication style and materials to suit a given audience
• Able to clearly articulate operational principles, practices, and policies
• Stays abreast of industry trends and technologies
• Accountable for work of self and others; sets standards around which others will operate
• Maintains a broad internal professional network and knows when to engage/activate it
• Develops or mentor’s diverse talent on the team
• Ability to be on-call and/or work during off-hours

Preferred

• Cloud or SRE‑related certifications
• Working knowledge of Azure

Work Flexibility

This role is eligible for hybrid work, with up to three days per week from home.