VIDEX: A Disaggregated and Extensible Virtual Index for Cloud-Native and AI-Driven Databases

Download PDF

Abstract

Virtual indexes play a crucial role in database query optimization. However, with the rapid advancement of cloud computing and AIdriven models for database optimization, traditional virtual index approaches face significant challenges. Cloud-native environments often prohibit direct conducting query optimization process on production databases due to stability requirements and data privacy concerns. Moreover, while AI models show promising progress, their integration with database systems poses challenges in system complexity, inference acceleration, and model hot updates. In this paper, we present VIDEX, a three-layer disaggregated architecture that decouples database instances, the virtual index optimizer, and algorithm services, providing standardized interfaces for AI model integration. Users can configure VIDEX by either collecting production statistics or loading from a prepared file, enabling highaccuracy what-if analysis using virtual indexes that yield query plans identical to production instances. Additionally, users can freely integrate new AI-driven algorithms into VIDEX. VIDEX has been deployed at ByteDance, serving thousands of MySQL instances daily and over millions of SQL queries for index optimization tasks.