Exploring the Role of SRE in the AI Era Through Data and Incident Management

In Part 1, we discussed the background behind SIGQ’s migration of the Incident Lake data platform to ClickHouse and introduced the technical features of ClickHouse that support the infrastructure of AI companies.In an era where petabyte-scale data is updated daily, how will business decision-making and operational management change? In this article, based on the second half of a conversation between Alexey Milovidov (hereinafter Alexey), co-founder and CTO of ClickHouse, and Mr. Kanetsuki, CEO of SIGQ, we explore “SRE in the AI Era,” with a focus on data-driven decision-making.

This is the second part of the interview series. For the second part, please see " Why SIGQ Migrated from BigQuery to ClickHouse."

Beyond the Boundaries of Incident Management and BI: The Future Pioneered by Agentic Analytics

Q. SIGQ refers to Incident Lake as the “Incident Intelligence Layer.” How does it differ from traditional incident management tools?

Kanetsuki: Traditional incident management tools focus on individual members of the development team. However, technical tasks such as collecting error logs and server metrics represent only a small part of the overall incident management process.

I believe the biggest challenge is how to manage the entire incident response process. Incident responders must assess the scope of the impact on users and services while considering the response strategy, including the business impact. This goes beyond mere log analysis and involves decisions that are very close to business judgment.

Alexey: So, it sounds like Incident Lake is targeting an area that combines observability and business analytics. I suppose that means it needs to handle data from both business metrics and technical metrics.

Kanetsuki: You’re absolutely right. For example, you could combine data from CRM and contract management tools to calculate the impact of SLA violations.

Alexey: Until now, people have had to painstakingly write SQL queries to analyze data across various sources, but in the future, we’ll be able to simply ask questions in natural language and get answers. To make this possible, it will become increasingly important for AI agents to have access to all relevant data.

However, this raises a new issue. While humans can write queries at a rate of about one per minute at best, AI agents can easily execute dozens of queries per minute. The emergence of AI agents has further heightened the importance of real-time databases.

Q. ClickHouse is indeed the perfect foundation for such AI-driven applications. AI agents are likely to bring about significant changes in the world of data analysis as well.

Alexey: This approach, in which AI issues queries to an analytical database to answer questions, is known as “Agentic Analytics.” This method can also be applied to Business Intelligence (BI), and we have reached a point where traditional BI tools risk becoming obsolete unless they incorporate AI agents.

This same approach is now extending into the realm of observability, including incident response, root cause analysis (RCA), security investigations, and network monitoring. Think back to incident response a few years ago.You were likely poring over thousands of charts, searching for anomalies. After painstakingly configuring alert rules by hand, you’d be inundated with a flood of alerts. And when you tried to investigate the logs, you had to start from scratch, trying to figure out what to search for. It was a never-ending cycle of such tasks.

AI agents are now taking over many of these tasks. This is what is known as AI-SRE. Simply enter a single prompt, and you’ll receive preliminary analysis and proposed solutions. In some cases, prompts may not even be necessary, as the AI autonomously handles the entire cycle—from anomaly detection to investigation, explanation, and resolution.

Engineers Will Always Be Needed: Data Supporting "Human Decision-Making" in the Age of AI 

Q. Traditional data analysis and intelligence have faced the challenge of being unable to effectively utilize the vast amounts of collected data for decision-making. AI agents seem poised to be a major help in solving this problem. But if that’s the case, what role will be left for humans?

Alexey: The problems we’re trying to solve are becoming increasingly complex. Thanks to AI, we no longer have to worry as much about minor issues or infrastructure tasks, which means we can focus on more important things. As a result, there’s a growing demand to accomplish even more. Consequently, our work is shifting toward higher-level tasks.

However, no matter how much data we collect, it is ultimately humans who decide how to act. AI agents may present many hypotheses, but the responsibility for determining the right course of action remains with humans—such as employees and managers.

Sometimes, we have to draw on all our experience to rule out incorrect hypotheses or meaningless responses.AI agents provide prompts for action, but those prompts are often misguided. For example, in the context of incident response, AI works very well when an experienced SRE provides the prompt. On the other hand, when someone with limited hands-on experience in production environments attempts to resolve an incident, they tend to accept the AI’s incorrect answers at face value. That is precisely why engineers must continue to draw on their past experience while continuing to learn.

Kanetsuki: I basically agree with Alexey. Once a decision is made, someone has to take responsibility for it. For example, when making a critical decision—such as replacing a database because it’s corrupted—you have to shoulder a commensurate level of responsibility. Therefore, as things stand, I believe that difficult decisions—especially those inherent in incident management—should be made by humans.

However, AI can make a significant contribution when it comes to supporting the decision-making process. The challenge today lies in filtering out the most important information from vast amounts of accumulated data to accelerate decision-making. That is why Incident Lake leverages LLMs through various features, including real-time RAG, decision support, and automated reporting. By helping to understand the context and generating executive summaries necessary for making judgments, AI plays a vital role in supporting human decision-making.

On the other hand, hallucinations are an inherent part of LLMs. For accurate analysis—particularly in incident response—it is essential to reliably mitigate these hallucinations. Otherwise, we risk losing our customers’ trust. For this reason, while Incident Lake uses LLMs for report generation and situation summarization, it also employs traditional machine learning for tasks such as calculating incident severity.

The key to growing a business lies in earning the trust of users

Q. Since the word “trust” has come up, I’d like to shift the topic here. SIGQ is steadily laying the groundwork for rapid growth, including launching a full-scale fundraising campaign. So, just for reference, I’d like to ask: ClickHouse has already grown into a product that can be called a global standard. If you had to name one thing that was most important during the community’s growth, what would it be?

Alexey: The key was to remain open to the community and keep everyone involved motivated. For example, we receive pull requests from all over the world for our codebase, but when we review them, we sometimes find that they don’t meet our quality standards. In a typical OSS project, we might simply reject them or ask for revisions. However, we see potential for growth in every contributor.We motivate them, actively lend a hand, and help them feel like they’re truly part of the project. That way, they become our fans and spread the word about the project to others.

It was by no means an easy journey. There was a time when the earliest community consisted of just a handful of dedicated individuals. During that period, we had to keep providing them with positive feedback and encouragement.

Kanetsuki: That kind of enthusiastic engagement is what builds trust in ClickHouse and its community. Of course, the product’s excellence goes without saying, but another reason we chose ClickHouse is the community. For example, the Japanese members of the ClickHouse community actively supported us, provided us with free credits, and gave us opportunities to share our insights at ClickHouse community events.If we have any questions, they usually respond quickly within an hour. That’s why we trust ClickHouse. This reliability is a crucial factor for any company, and it’s a matter of survival, especially for startups.

At SIGQ, we also place a high priority on building trust with our customers, albeit in different ways. For example, obtaining SOC 2 Type 1 certification and presenting at international conferences as researchers are just a few examples of this.

Everyone knows about big companies like Toyota, but no one knows about small startups. That’s why building strong relationships with customers is the most important factor in growing a business. I firmly believe that it’s not just the technology itself, but the people and communities behind it that shape trust in a startup.

To create the best businesses and products

Q. Finally, could you both share a message for our readers?

Alexey: Current AI-assisted coding still requires frequent decision-making, so we remain at the stage where engineers use AI as a tool. But what will remain in a few years, when models have evolved further and can automate much of the code writing and decision-making? I believe it will be product vision and product strategy.Until now, a company’s competitive advantage has lain in its “ability to work faster.” Consequently, there is a pessimistic view that when the era of outsourcing work to AI arrives, the amount of capital available to cover the cost of using AI will directly determine competitive advantage. However, taking a more optimistic view, humans should be able to identify areas for independent exploration and define unique product strategies and visions.

There is no end to technological innovation. Opportunities to tackle new challenges can always be found. The key is to explore a wide range of options. Even in established fields or highly competitive sectors, there is still room for optimization and quality improvement. It is still possible to develop the best products, and doing so will bring value to future customers.

Kanetsuki: Now that combining and leveraging AI and data has become the norm, the importance of data freshness is growing more and more critical. Especially when using RAG, data freshness is a matter of life and death. That is precisely why I believe ClickHouse is the best database for building RAG systems.

Let me say this again. Based on my experience over the past decade of thoroughly utilizing numerous databases—whether OLAP or OLTP—I can say without hesitation that ClickHouse is the best database available today. That is precisely why I want to make even greater use of ClickHouse and share its value not only with Japanese companies but with businesses around the world, with the aim of encouraging its adoption.

Alexey: I’m really glad to hear that. That’s exactly why we do this work.

Q. Thank you so much for taking the time to speak with us today!

In the AI era, SREs are evolving from "data interpreters" to "decision designers"

With the emergence of Agentic Analytics and AI-SRE, tasks such as root cause analysis, summarization, and hypothesis evaluation will become significantly more efficient in the near future. At the same time, areas involving decision-making with accountability, customer support, and product strategy will remain the domain of humans. Even if AI presents hypotheses, these are areas where humans must ultimately take responsibility.

Through their core products and services, SIGQ and ClickHouse will continue to contribute to leveraging the power of AI to support human decision-making.