AI サービスが一時的に利用できないため、復旧後に翻訳を補完します。
This article is sponsored by BigPanda and was written, edited, and published in alignment with our Emerj sponsored content guidelines. Learn more about our thought leadership and content creation services on our Emerj Media Services page. Every business-critical service failure starts the clock on lost revenue and customer trust. Enterprise systems now run at machine speed, generating alerts faster than humans can respond to. After a global IT outage in July 2024, triggered by a single faulty software update, one major U.S. airline reported to the SEC about $380 million in lost revenue and $170 million in added costs from roughly 7,000 flight cancellations over five days. A study of a large commercial bank, presented at the 2020 International Conference on Software Engineering, found alert storms of hundreds to thousands of alerts a minute occurring about weekly, each taking six engineers 55 minutes on average to handle. Of 44 engineers surveyed, 77% said there were too many alerts to identify problems manually. The researchers also observed that most engineers tended to ignore the storms until customer complaints exposed the failure. Emerj recently hosted a series on the AI in Business Podcast examining how enterprises can modernize IT operations by preventing recurring failures, correlating alert noise into actionable incidents, and automating repetitive L1 work, including guests Assaf Resnick, Founder & CEO at BigPanda; Luke Rotta, Director of Site Reliability Engineering at Charles Schwab; Dave Glick, Senior Vice President of Growth Technology at Walmart; and Dhrubojyoti Das Deb, Senior Vice President of Technology at JPMorganChase. This article examines the key insights enterprise leaders can apply as the economics of IT operations shift from scaling teams to scaling intelligence: Enterprise-owned knowledge graph for faster incident resolution: Capture incident history, live system data, and operators’ undocumented knowledge so agents can pinpoint root causes and recommend fixes in minutes. Cross-application alert correlation for faster root cause analysis: Group thousands of related alerts into single incidents and trace them across connected applications so teams stop investigating symptoms and find the originating fault sooner. Volume-first automation priorities to recover team capacity: Automate high-volume, repetitive L1 tasks before high-risk ones and reinvest the hours saved into further automation. Ticket elimination before added support headcount: Fix the causes of recurring tickets instead of adding L1 and L2 staff, freeing engineers for cost-saving and revenue-driving work. Listen to the full episodes below: Episode 1: Inside the Shift to Agentic IT Ops – with Assaf Resnick of BigPanda Guest: Assaf Resnick, Founder & CEO at BigPanda Expertise: AI in IT Operations, Enterprise Automation, Venture Capital, Enterprise Software Brief Recognition: Assaf Resnick is Founder & CEO of BigPanda, a company he has led since 2012. Before founding BigPanda, he was a Principal at Sequoia Capital, where he invested in early- and growth-stage software and internet companies. He holds a Bachelor of Science in Business Administration from the University of California, Berkeley’s Haas School of Business. Episode 2: How to Reset Your IT Operations and Build for Intelligent Response – with Luke Rotta of Charles Schwab Guest: Luke Rotta, Director of Site Reliability Engineering at Charles Schwab Expertise: Enterprise AI Strategy, IT Strategy, Site Reliability Engineering, Digital Transformation Brief Recognition: Luke Rotta is Director of Site Reliability Engineering at Charles Schwab, with 25 years of financial services technology leadership. He leads reliability engineering, automation, and operational improvements across Schwab’s wealth technology platform. He also taught DevOps through the University of Chicago’s professional education program and served on an advisory board at Seton Hall University. Rotta holds an M.S. in Business Information Technology and a B.S. in Computer Science from DePaul University. Episode 3: Intelligent Operations at Enterprise Scale – with Dave Glick of Walmart Guest: Dave Glick, SVP of Enterprise Business Services at Walmart Expertise: Generative AI, Distributed Systems, E-commerce Technology, Technology Leadership Brief Recognition: Dave Glick is SVP of Enterprise Business Services at Walmart. Previously, he served as CTO of Flexe, a logistics technology company, and spent 20 years at Amazon building core retail and operations systems, including the company’s original automated pricing system, warehouse management capabilities, and Amazon Flex. He also served on the boards of Crowd Cow and Businesses Ending Slavery and Trafficking (BEST). Glick holds a Ph.D. in Physics from the University of North Carolina at Chapel Hill and a B.S. in Physics from the University of Michigan. Episode 4: Reducing Alert Noise with AI in Enterprise IT – with Dhrubojyoti Das Deb of JPMorganChase Guest: Dhrubojyoti Das Deb, Senior Vice President of Technology at JPMorganChase Expertise: Enterprise Architecture, GenAI, Cloud Modernization, Banking and Payments Technology Brief Recognition: Dhrubojyoti Das Deb is Senior Vice President of Technology at JPMorganChase, where he leads engineering for a major e-commerce credit and debit card processing platform handling nearly 70% of U.S. merchant traffic. He leads more than 100 software engineers across the U.S., India, and Singapore, overseeing architecture and delivery for systems designed for high availability and fault tolerance. Previously, he spent more than 12 years at Cognizant in technical architecture, engineering, and delivery leadership roles. He holds a B.Tech in Applied Electronics and Instrumentation Engineering from Haldia Institute of Technology. Enterprise-Owned Knowledge Graph for Faster Incident Resolution Assaf Resnick opens the series by arguing that enterprises are treating IT operations as a staffing problem when it has become a data problem. Hybrid cloud, microservices, and CI/CD (Continuous Integration and Continuous Delivery) mean systems change and break faster than ever, generating, by his estimate, a thousand times more data at a thousand times the speed. Outsourcing to global systems integrators, the cost-cutting trend of the past three decades, has not solved the problem. With an average tenure of about seven months, engineers often leave soon after they learn the environment. Round-the-clock coverage adds three shift handoffs a day, each a point where context can be lost. Dhrubojyoti Das Deb raises the same risk later in the series: knowledge of legacy applications leaves with the subject matter experts who hold it. Resnick’s alternative is to onboard agents the way an enterprise would onboard a new employee, through an IT knowledge graph that holds three kinds of context: Live awareness: What is happening across the environment right now, connected at a scale no human operator can track. Operational history: Everything that has happened in the environment, including transcripts from previous bridge calls. Tribal knowledge: The undocumented expertise of long-tenured operators, plus what agents learn by observing how the strongest operators resolve problems. He illustrates the impact with a hypothetical network issue affecting a large bank’s ATMs. Today, thousands of alerts reach an L1 team that may be new to the environment. A 15-minute triage target stretches to two hours, and a bridge call runs six more hours before someone traces the cause to a security update made three weeks earlier. With the knowledge graph in place, an agent spots the problem in about 30 seconds, matches it to a similar incident from seven months earlier, and recommends rolling back the change. Luke Rotta notes that while incidents share patterns, in more than two decades supporting systems he has found each one has some piece that is slightly different. Where an agent’s confidence falls below its threshold, Resnick says that it hands the issue to a human. It routes the issue to an L1 operator or directly to the specialist who owns that domain, rather than convening a full bridge call. In Rotta’s experience, getting the right people involved is where much of the slowness in resolving incidents comes from. Resnick illustrates the shift in cost structure as a move from roughly 300 outsourced operators to an agentic layer managed by two or three site reliability engineers. He also points to a Fortune 500 healthcare customer that reduced incidents and self-inflicted outages by an amount he valued at half a billion dollars. He advises against changing tooling, processes, outsourcing relationships, and org charts all at once, comparing the shift to how CIOs moved to public cloud, one workflow at a time: Add an agentic coworker: Bring the agent into bridge calls and give it access to operational data so it learns how teams resolve incidents. Start with low-barrier workflows: Point it at real, recurring problems where a quick win is likely. Promote early wins internally: Use visible results to build confidence for broader rollout. Expand toward full coverage: Once the agent handles around 30% of the work, set a roadmap toward the rest. Resnick is equally clear about who should own that knowledge, regardless of which vendors or consultants help build it: “I think the important thing is that you build an IT knowledge graph that encapsulates all of the knowledge and data and history within your enterprise. And that should be yours. It shouldn’t sit inside of the walled garden of a specific vendor.” — Assaf Resnick, Founder & CEO at BigPanda Cross-Application Alert Correlation for Faster Root Cause Analysis The bottleneck in incident response, according to Dhrubojyoti Das Deb, is no longer a lack of data but knowing where a problem started. Logging, tracing, and telemetry have grown with the business, but each application’s data still sits in its own silo. In a chain of five connected applications, he explains, the fourth may show the error while the problem started in the first, which raised no alerts and passed bad data through the second and third. Teams lose time investigating the application that shows the error, and the issue persists until they trace the fault back to its source. Luke Rotta reframes the cost of that delay as opportunity cost rather than downtime alone. While a service is unavailable, the business cannot capture new sales or transactions, and the impact varies with how busy that time of day is. Dave Glick sees the same drag in engineering. When a ticket arrives, engineers act as “digital detectives,” working out what happened and trying to reproduce it, toil he wants removed from engineers and operators alike. Alongside the loss of expert knowledge, Das Deb identifies three frictions that make it hard to separate signal from noise: Alert overload: He estimates that 99% of incoming data is noise, such as debug and informational messages. Complex dependency stacks: Applications built on different languages and platforms, from Java and Python to cloud and AI services, create dependencies that are harder to trace. Constant change: Multiple CI/CD releases a week change an application’s state, and new data may not reach traceability and telemetry. Rotta argues that unintegrated to [truncated for AI cost control]