Splunk's Agent Management team is looking for an experienced software engineer to join a team as Senior Staff (Principal) developer. During your work you will own the design, implementation and operation of a brand-new solution that will manage the life cycle of millions of data streaming agents throughout our customer base. The service will allow our customer's critically important security and observability operations to run uninterrupted while performing tasks such as software upgrades, security improvements and health monitoring.As a Principal Engineer, you will operate beyond a single team, setting technical direction for Agent Management and influencing architecture and engineering practices across multiple teams. You will scale your impact by enabling other senior engineers, driving alignment across the organization, and making high-impact technical trade-offs visible and actionable.

The principal engineer will and will be primarily responsible for driving product technical design and architecture in a way that should fulfill needs of Splunk customers currently operating on both cloud-base but also on premise deployment. More than this You will provide technical leadership and architectural direction, ensuring teams can deliver high-quality, scalable solutions efficiently. You will ensure that observability is built into the system, Service Level Objectives (SLO) are defined, monitored and met and that engineers are alerted when performance degrades. You will work with senior members of other teams in the Splunk eco-system to ensure their services provide the required functionality and that integration is seamless. You will also collaborate with Product Management to translate business requirements into technical tasks and provide an estimation for implementing them. Success will be measured by your ability to develop and maintain a service that is performant and serviceable while meeting execution goals. Your previous experience in leading critically important, backend-heavy projects is critical. Despite this your work will also include coordinating all aspects of the software development lifecycle and perspective. Ability to teach and raise the knowledge and skill level of junior engineers on the team is another important success measure.

Agent Management presents a set of complex technical challenges, including managing the full lifecycle of thousands of agents across highly diverse customer environments. The system must scale to support deployments with 100,000+ agents, handle large-scale configuration changes and upgrades safely, and operate reliably across hybrid cloud and on-premise architectures. It requires careful design of distributed systems, state management, backward-compatible protocols, and rollout mechanisms to ensure reliability, observability, and security at scale, while balancing performance, operability, and long-term maintainability.

• Within this role you will be also- Acts as the technical authority for Agent Management, owning the technical roadmap and architectural vision across multiple teams over a 12+ month horizon.
• Leads design discussions for systems under technical purview, ensuring that non-functional requirements (scale, security, operability, usability, etc.) are considered
• Technical ownership of the team’s work to ensure high quality
• Identify and resolve pre-production system bottlenecks and production issues
• Drive operational excellence and ensure the product is serviceable and code is maintainable
• Participate in rotating 24-7 on-call duties to diagnose and solve customer issues
• Mentor senior engineers and technical leads, helping scale technical excellence and decision-making across the organization.
• Drive cross-team architectural alignment and influence technical direction beyond the immediate team
• Represent Agent Management in architecture forums, design reviews, and cross-org technical discussions
• Be accountable for technical outcomes and long-term system health

Requirements:

• Strong fundamentals in software engineering: system design, data structures and algorithms
• Ability to lead, guide and coach strong team of other mid/senior developers
• Strong coding proficiency is required, with the ability to contribute hands-on where it has the highest impact, while primarily focusing on technical leadership and system design. Preferred techologies are Golang or C++, however we are open to your experince if you are willing to learn.
• Experience working on distributed systems in public cloud environments such as AWS (preferred), GCP, Azure and familiarity with the services available on these platforms.
• Proven ability to influence technical direction beyond a single team without direct authority

Why Cisco?

At Cisco, we’re revolutionizing how data and infrastructure connect and protect organizations in the AI era – and beyond. We’ve been innovating fearlessly for 40 years to create solutions that power how humans and technology work together across the physical and digital worlds. These solutions provide customers with unparalleled security, visibility, and insights across the entire digital footprint.

Fueled by the depth and breadth of our technology, we experiment and create meaningful solutions. Add to that our worldwide network of doers and experts, and you’ll see that the opportunities to grow and build are limitless. We work as a team, collaborating with empathy to make really big things happen on a global scale. Because our solutions are everywhere, our impact is everywhere.

We are Cisco, and our power starts with you.