In Part 1, we covered why SIGQ migrated Incident Lake's data platform to ClickHouse, and the technical characteristics of ClickHouse that power the infrastructure of AI companies. In an era when petabyte-scale data is updated every day, how will business decision-making and operations management change? Drawing on the second half of a conversation between Alexey Milovidov, co-founder and CTO of ClickHouse ("Alexey"), and Kanetsuki, CEO of SIGQ, this article explores "SRE in the AI era," built on the premise of data-driven decision-making.
This is Part 2 of the conversation. For Part 1, see "Why SIGQ Migrated from BigQuery to ClickHouse."
Beyond the Boundary Between Incident Management and BI: The Future Opened Up by Agentic Analytics
Q. SIGQ calls Incident Lake an "incident intelligence layer." How is it different from traditional incident management tools?
Kanetsuki: Traditional incident management tools focus on individual members of developer teams. But technical tasks like collecting error logs and server metrics are only a small part of incident management as a whole.
I think the biggest challenge is how to handle the entire incident flow. Incident responders need to assess the scope of impact on users and services and decide on a response that takes business impact into account. That isn't just log analysis; it's a domain very close to business decision-making.
Alexey: So you could say that what Incident Lake is trying to address is something like a combination of observability and business analytics. You'd need to handle data from both business metrics and technical metrics, wouldn't you?
Kanetsuki: Exactly. For example, it might combine data from CRM and contract management tools to calculate the impact of SLA violations.
Alexey: Until now, humans have had to painstakingly write SQL queries to analyze data across every data source. Going forward, you'll be able to get answers just by asking questions in natural language. To make that possible, it becomes increasingly important for AI agents to have access to all the relevant data.
But that creates a new problem. A human writes queries at most about once a minute, while an AI agent can easily run dozens of queries per minute. The arrival of AI agents makes real-time databases even more important.
Q. ClickHouse seems like the ideal foundation for those kinds of AI-driven applications. AI agents look set to bring major changes to the world of data analytics, too.
Alexey: This approach, where AI issues queries to an analytical database to answer questions, is called "Agentic Analytics." It can also be applied to Business Intelligence (BI), and we've reached the point where traditional BI tools risk becoming obsolete unless they incorporate AI agents.
The same idea is spreading to observability domains such as incident response, RCA (root cause analysis), security investigations, and network monitoring. Think back to incident response a few years ago. You were probably staring at thousands of charts looking for anomalies. You'd painstakingly set up alert rules by hand, only to be overwhelmed by a flood of alerts. When you wanted to investigate logs, you had to start by figuring out what to even search for. It must have been one task like that after another.
AI agents are now taking over much of this work. That's what's called AI-SRE. Send a single prompt and you get back a preliminary analysis and proposed fixes. Or, without even needing a prompt, AI may autonomously run the entire cycle from anomaly detection through investigation, explanation, and resolution.
Engineers Will Still Be Needed: Data That Supports "Human Decision-Making" in the AI Era
Q. Data analytics and intelligence have long struggled with being unable to put the huge volumes of collected data to proper use in decision-making. AI agents look like they could be a big help in solving this. But then, what role is left for humans?
Alexey: The problems we're trying to solve are becoming more sophisticated. Thanks to AI, we're less bogged down by trivial problems and infrastructure work, so we can take on more. That, in turn, creates demand to do even more. Work is shifting to a higher level accordingly.
But no matter how much data you collect, in the end it's humans who decide how to act. AI agents may present many hypotheses, but the responsibility for deciding which direction to take remains with humans—employees and managers.
Sometimes you have to draw on all your experience to rule out wrong hypotheses and meaningless answers. AI agents give you a starting point for action, but that direction is often wrong. In incident response, for example, AI works extremely well when an experienced SRE is giving it prompts. On the other hand, when someone with little hands-on production experience tries to resolve an incident, they tend to accept the AI's wrong answers at face value. That's why engineers still need to draw on their experience and keep learning.
Kanetsuki: I basically agree with Alexey. When a decision is made, someone has to take responsibility for it. For example, when you make a major call like "the database is broken and has to be replaced," you have to bear a correspondingly large responsibility. So for now, I think humans should make the difficult decisions, especially those that come with incident management.
That said, AI can contribute a great deal to supporting those decisions. Today's challenge is filtering what matters out of the huge volumes of accumulated data and speeding up decision-making. That's why Incident Lake uses LLMs in a range of features, including real-time RAG, decision support, and automated reporting. By helping people understand the situation and generating the executive summaries they need to make decisions, it can play a role in supporting human decision-making.
On the other hand, hallucinations are inherent to LLMs. For accurate analysis—especially in incident response—hallucinations must be reliably suppressed. Otherwise, you'll lose customers' trust. So while Incident Lake uses LLMs for report generation and situation summaries, it also uses traditional machine learning for things like calculating incident severity.
The Key to Growing a Business Is Earning Users' Trust
Q. Since the word "trust" came up, I'd like to change topics. SIGQ is steadily preparing for rapid growth, including launching a major fundraising effort. For reference, ClickHouse has already grown into what could be called a global standard product. If you had to name the single most important thing in growing the community, what would it be?
Alexey: The key was staying open to the community and supporting the motivation of the people involved. For example, our codebase receives pull requests from all over the world, and when we review them, they sometimes don't meet our quality standards. A typical OSS project might simply reject them or ask for changes. But we see growth potential in every contributor. We motivate them, actively lend a hand, and let them feel that they're part of the project. Then they become our fans and spread the word about the project.
This was by no means an easy road. In the very early days, there were times when the community consisted of only a handful of dedicated people. During that period, we had to keep giving them positive feedback and motivation.
Kanetsuki: That kind of committed engagement is what builds trust in ClickHouse and its community. The product is excellent, of course, but the community is another reason we chose ClickHouse. For example, the ClickHouse team in Japan actively supported us, provided free credits, and gave us opportunities to share our insights at ClickHouse community events. Whenever we have a question, they usually respond within an hour. That's why we trust ClickHouse. This kind of reliability is an important factor for any company, and for startups in particular, it's a matter of survival.
At SIGQ, we also place a strong emphasis on earning customers' trust, though in a different way. Obtaining SOC 2 Type 1 certification and presenting as researchers at international conferences are examples.
Everyone knows a big company like Toyota, but nobody knows a small startup. That's why strong relationships with customers are the most important thing in growing a business. I'm convinced that it's not just the technology itself, but also the people and community behind it, that shape trust in a startup.
Building the Best Business and Product
Q. Finally, could each of you share a message for our readers?
Alexey: AI coding today still requires frequent decision-making, and we're still at the stage where engineers use AI as a tool. But a few years from now, when models have advanced further and can automate much of code writing and decision-making, what will be left? I think it's product vision and product strategy. Until now, a company's competitive advantage has lain in "the ability to work faster." So there's a pessimistic view that once we enter an era of outsourcing work to AI, the amount of capital available to pay for AI will translate directly into competitive advantage. But looking at it more optimistically, humans should be able to find things worth exploring on their own and define unique product strategies and visions.
Technological innovation never ends. You can always find opportunities to work on something new. What matters is exploring a wide range of options. Even in established fields or highly competitive areas, there's still plenty of room for optimization and quality improvement. It's still possible to build the best product, and that will deliver value to future customers as well.
Kanetsuki: Now that combining AI and data has become the norm, data freshness is more important than ever. Especially when using RAG, data freshness is a matter of life and death. That's why I believe ClickHouse is the best database for building RAG.
Let me say it again. Based on my experience over the past ten years of putting countless databases—both OLAP and OLTP—through their paces, today I can answer without hesitation: "ClickHouse is the best database." That's why I want to make even more use of ClickHouse, and to share its value not only with Japanese companies but with companies around the world, helping drive ClickHouse adoption.
Alexey: I'm really happy to hear that. Hearing words like that is what we work for.
Q. Thank you both so much for your time today!
SRE in the AI Era: From "People Who Read Data" to "People Who Design Decisions"
With the arrival of Agentic Analytics and AI-SRE, work such as root-cause investigation, summarization, and weighing hypotheses will rapidly become more efficient. Meanwhile, areas such as decisions that carry responsibility, customer communication, and product strategy will remain with humans. Even when AI presents hypotheses, these are areas where humans must ultimately take responsibility.
Through the products and services that form that foundation, SIGQ and ClickHouse will continue to help put the power of AI to work for human decision-making.