SIGQ Inc. provides "Incident Lake," an AI agent specialized in incident management. The company originally used BigQuery as its data platform, but migrated to ClickHouse about six months ago and now runs multiple customers' production workloads on ClickHouse.
In late April, at a location in Tokyo, Alexey Milovidov, co-founder and CTO of ClickHouse ("Alexey"), sat down for a conversation with Takaaki Kanetsuki, CEO of SIGQ Inc. ("Kanetsuki"). In this article, we explore the background of the migration and how ClickHouse addressed the challenges SIGQ was facing.
This is Part 1 of the conversation. For Part 2, see "What Should SRE Look Like in the AI Era? Insights from Data and Incident Management."
How a C++ Engineer Created a Database for Analyzing Some of the World's Largest Traffic
Q. Alexey, you've been developing ClickHouse since 2009. What got you involved with databases in the first place? Please tell us, including the challenges you faced at the start.
Alexey: I wasn't originally a database developer; I was a C++ engineer at a large company. I worked on a service that provided website operators with analytics reports on their visitors. That's where I ran into a big challenge. On a platform generating hundreds of billions of events every day, we had to offer reports that were almost infinitely customizable, letting users get any aggregation they wanted. On top of that, it had to be real-time enough that results appeared the instant a user defined a report.
Of course, MySQL couldn't handle this use case, so I started trying all kinds of databases. Along the way, I learned about column-oriented databases and big data platforms, and compared existing products in terms of performance, compression ratio, and cost. As a result, I began to think that I could build a column-oriented storage system specialized for aggregation, sorting, and filtering myself. What I built was a prototype called "OLAPServer," and it turned out to work better than the existing systems.
*OLAP (OnLine Analytical Processing): Technology for instant multidimensional analysis of massive amounts of data stored in data warehouses and similar systems—for example, analyzing sales data along multiple axes such as region and time.
Q. And that led to today's ClickHouse.
Alexey: At that point, I didn't have a clear vision that would lead to ClickHouse, but I vaguely wanted to generalize the system so it could handle a wide variety of queries. I worked on it in my spare time at first, and eventually during working hours too, and the first version of ClickHouse was released in 2012. It was used not only by my team but by other departments in the company, such as e-commerce and business analytics. Later, as I attended technical conferences, I realized there was enormous demand for this kind of database. Engineers at many companies were each building prototypes similar to mine. I became convinced it should be open source, persuaded the company, and released it in 2016. This June marks exactly ten years since ClickHouse was open-sourced.
The BigQuery Constraints Incident Lake Was Facing
Q. SIGQ recently migrated Incident Lake's data platform from BigQuery to ClickHouse Cloud. What challenges did the migration solve?
Kanetsuki: First, a quick introduction to Incident Lake. Incident Lake is an incident management platform with integrated AI. While many existing tools are aimed at developers, ours focuses on the managers who oversee incident response.
In incident response, requests for failure handling and report writing fly back and forth, and this work is very burdensome. Incident Lake's AI agents help with tasks such as timeline generation and report generation. So when a failure occurs, Incident Lake collects not only error logs but also communication logs from tools like Slack and Teams.
Alexey: So you use the collected data to generate summaries automatically.
Kanetsuki: Exactly. And the most important thing in making this work is data freshness. If you can't tell whether the current data is correct or what's most up to date, the support provided by AI agents can't be trusted in the first place.
We store two kinds of data: raw data, such as user messages, and vectorized data. We vectorize all generated data and reports, enabling something close to semantic search.
Data freshness is critical for tracking an incident response situation that changes moment by moment. That means we need to update the collected data in line with the latest timeline.
But at the time, BigQuery had a constraint that prevented updating data immediately after insertion, which made it hard to maintain data freshness. So we took the opportunity to migrate to ClickHouse Cloud.
Alexey: Many ClickHouse users are companies that had used BigQuery for years. Like SIGQ, many migrate because of latency. Many also migrate for the richer feature set, since ClickHouse can do more. And cost is another important issue. BigQuery uses a pay-as-you-go pricing model that charges per query and per volume of data processed. That's fine when a human runs a query now and then, but it isn't well suited to users who run large numbers of queries—especially AI agents. AI agents often run 10 or 20 queries for a single request, and with BigQuery, costs skyrocket.
Q. How do you use ClickHouse's features in Incident Lake? In particular, are there any features whose value you've really felt?
Kanetsuki: One example is ClickHouse's approximate vector search. Incident Lake uses this feature to retrieve similar incidents.
In incident response, it's important to know whether similar cases have happened before. But finding similar incidents with simple vector search is actually very difficult. So Incident Lake uses ClickHouse's approximate vector search to express similarity as a number. Customers have responded very positively to this similarity score and the similar-incident search feature.
Before migrating to ClickHouse, we had implemented similarity calculation ourselves, and it was a very heavy workload. By using ClickHouse's approximate vector search, I feel we've greatly reduced the load while increasing the value of Incident Lake.
Q. I heard SIGQ was the first company in Japan to use ClickHouse's approximate vector search.
Kanetsuki: Yes. We started using this new feature the very day it became available on ClickHouse Cloud. In fact, it was a feature we'd been waiting on for a long time—we even asked the ClickHouse team, "When will it be available?"
Alexey: Customers like that make us really happy. We're always looking for technical partners who will validate new features with real business workloads. Perhaps Kanetsuki's enthusiasm helped push us to release this feature on ClickHouse Cloud.
Why the World's Largest AI Companies Choose ClickHouse
Q. ClickHouse is known as an analytical database fast enough to return sub-second responses to queries over petabyte-scale data and complex telemetry. How can it combine data at that scale with real-time performance?
Alexey: First, as a basic premise, ClickHouse doesn't require you to separate transactional (OLTP) data from analytical (OLAP) data. One typical misconception about big data is the idea that transactional data and the fresh data used for analysis should be managed separately.
*OLTP (Online Transaction Processing): Technology aimed at accurately reading and writing small amounts of data (transactions), such as bank transfers or e-commerce orders.
For example, the idea of "keeping all historical data in S3 or an RDBMS and putting only the small amount of fresh data needed for real-time analytics into an analytical database" is a common misconception. In reality, by combining several technologies, you can achieve both near-real-time data ingestion and sub-second responses to analytical queries. Specifically, these include column-oriented storage for fast query processing and the MergeTree data structure, which enables fast data ingestion.
In addition, ClickHouse stores all data in shared storage combined with tiered caching. By placing caches on local machines' SSDs and memory, you get in-memory-level performance for frequently accessed data, NVMe SSD performance for most data, and the cost efficiency of object storage for the system as a whole.
ClickHouse's strength is that it can deliver all of this as a single integrated system.
Q. Many of the world's largest AI companies, such as OpenAI and Anthropic, run their observability on ClickHouse. Why did they choose ClickHouse?
Alexey: The answer is simple. No other technology can handle the enormous scale of data that AI companies need. In fact, they tried many technologies, and all of them broke. But ClickHouse worked surprisingly well, so they've kept using it.
ClickHouse's strength lies in its scalability. AI companies have to handle tens of petabytes of data flowing in every day. With ClickHouse, you can build clusters of thousands of nodes and run distributed queries across them. You can also efficiently build text indexes over massive log data. ClickHouse handles these things far better than existing systems.
Another major feature is its high data compression ratio. Companies like Comcast and eBay have achieved compression ratios of 20x and 30x. Other systems don't offer storage efficiency this high, so in environments storing tens of petabytes of data, ClickHouse stands alone. And combined with object storage, it becomes a near-perfect solution.
In fact, the largest AI companies—OpenAI, Anthropic, Tesla, xAI—use ClickHouse at massive scale for observability, analyzing fresh data at full scale.
Kanetsuki: That really clicks for me. Once you understand how nodes scale up and how data is distributed, you can intuitively see why ClickHouse can keep handling petabyte-scale data.
The Future of Databases: Unifying OLTP and OLAP
Q. Alexey mentioned earlier that "ClickHouse doesn't require you to separate transactional data from analytical data." Will the two converge in large-scale data platforms in the AI era? Please share your views on the outlook for next-generation databases.
Kanetsuki: Actually, alongside running the business, I also do database research, focusing on an approach called HTAP. At my previous job, I worked on operating a large-scale data analytics system, and I struggled with managing latency when sending data from OLTP to OLAP. That gave me a strong interest in the HTAP approach, which eliminates the data integration work itself, and I started researching it. This March, I presented a paper at DOLAP, co-located with EDBT/ICDT (one of the most prominent database conferences in Europe).
*HTAP: Technology for running transaction processing (OLTP) and analytical processing (OLAP) on a single platform.
The current challenge with HTAP is data synchronization. Internally, data written to OLTP is transferred to OLAP, but this is an expensive process, and maintaining the data flow is difficult. If you want to analyze data right after writing it but query latency takes an hour, you still can't grasp the latest situation.
Alexey: The challenge in building an HTAP database is that the internal data structures each workload requires are fundamentally different. OLAP databases need column-oriented storage and sparse indexes over compressed blocks to achieve fast processing. But with these data structures, it's hard to efficiently read or update a single record. OLTP databases, on the other hand, need fine-grained indexes on each record and data structures that can quickly replace records.
So there are two approaches. One is to find a compromise within a single data structure. In-memory databases are the classic example, but in practice they haven't been very successful. When handling large-scale data, a unified data structure isn't well suited to analytical processing, and it breaks down as soon as the data no longer fits in memory.
The other is to quickly synchronize two different data structures. This is currently the more common approach, and it's the one ClickHouse Cloud takes. Specifically, we offer ClickHouse and ClickHouse Postgres on fast NVMe storage and keep them in sync. This makes data in Postgres available from ClickHouse with reasonable latency.
Kanetsuki: Incident Lake also uses Postgres and ClickHouse together. We don't currently sync the two, but we'd like to consolidate as much data as possible into ClickHouse. Unique indexes on IDs and SQL compatibility are the bottlenecks—are there plans to develop those?
Alexey: The answer is yes; it's already in development. As for SQL compatibility, testing with SQL Logic Test already shows over 97% compatibility, so standard SQL features are well supported.
What a Data Platform Needs in the AI Era
What emerged from this conversation is that SIGQ's reason for migrating to ClickHouse wasn't simply cost reduction. The more fundamental motivation was meeting the data platform requirements of incident management in the AI era—and those requirements aligned closely with ClickHouse's history and technical characteristics.
In Part 2, we dig into how incident management and SRE will change in the AI era on top of this data platform.