About the job

The DCIM Lifecycle team operates one of the largest-scale monitoring systems at Google, reading telemetry from millions of devices in every Google datacenter. Our issues include managing the rapid growth and diversification of the Google fleet and hardware, new use cases for critical monitoring of third-party facilities, and retiring technical debt.

Google is bringing back tape libraries to our data centers in order to support various critical requirements including new cold storage tier, better TCO, contingency for HDD/SSD shortage due to unprecedented AI/ML capacity demand. This role is to design and delivery tape health at Google scale for reliability.

In this role, you will work with your teammates to design, code, and put into production very large-scale distributed monitoring systems and work with your team and partner teams to enable new use cases for large-scale telemetry gathering. You will also create various system monitoring dashboards, defining service level objectives (SLOs), documentation and playbooks. You will have the opportunity to take onsite trips to one or more of Google's datacenters each year to work with new systems and datacenter technical staff in person.The US base salary range for this full-time position is $166,000-$244,000 + bonus + equity + benefits. Our salary ranges are determined by role, level, and location. Within the range, individual pay is determined by work location and additional factors, including job-related skills, experience, and relevant education or training. Your recruiter can share more about the specific salary range for your preferred location during the hiring process.

Please note that the compensation details listed in US role postings reflect the base salary only, and do not include bonus, equity, or benefits. Learn more about benefits at Google.

Responsibilities

• Design, develop, and maintain software services for collecting and analyzing telemetry data from tape libraries, drives, and robotic components.

• Implement algorithms and rules to detect, diagnose, and predict hardware failures.

• Integrate tape health systems with Google's data center health monitoring infrastructure (e.g., system health, network doctor) and automated repair workflows (e.g., surgeon, silk roads).

• Collaborate with hardware engineers and vendors to understand failure modes and improve diagnostic capabilities.

• Develop dashboards and tools to provide visibility into the health and status of the tape hardware fleet. Participate in the full software development lifecycle, including requirements gathering, design, coding, testing, deployment, and operation.

Minimum qualifications

• Bachelor’s degree or equivalent practical experience.

• 5 years of experience in software development.

• 3 years of experience in one or more general-purpose programming languages (e.g., C++, Python, Go).

• 3 years of experience with distributed systems design and development.

Preferred qualifications

• Master's degree or PhD in Computer Science or a related technical field.

• 3 years of experience with hardware health monitoring, diagnostics, and repair automation systems.

• 3 years of experience working with monitoring and alerting systems (e.g., Monarch, Automon).

• 3 years of experience with large-scale data collection and analysis pipelines.

• Familiarity with data center infrastructure and operations.