Improving Time Series Data Compression in Apache IoTDB

Download PDF

Abstract

Time series data are generated on an unprecedented scale across various domains. Although traditional compression techniques reduce storage costs, they typically require full decompression before querying, leading to increased latency and higher resource consumption. Homomorphic compression (HC), which enables direct computation on the compressed data without decompression, shows the potential for both reduced storage and improved query performance. However, the unique complexities of time series data pose challenges that current HC methods have yet to adequately address. In this paper, we introduce HC theory in the time series domain, transformatively enabling HC to time series database queries. Building on our theory, we develop CompressIoTDB – a novel homomorphic compression framework integrated into Apache IoTDB. By leveraging our proposed CompColumn structure, our framework supports a wide range of query operators, including filtering, aggregation, and window-based functions, all while maintaining data in its compressed form. Furthermore, we incorporate system-level optimizations such as late decompression and dynamic auxiliary management to further boost query efficiency. Extensive experiments show that CompressIoTDB significantly enhances query processing for time series data, achieving an average throughput improvement of 53.4% and memory usage reduction of 20%.