Noise Matters: Cross Contrastive Learning for Flink Anomaly Detection

Download PDF

Abstract

Flink clusters often suffer from hotspot issues where the monitored job delay and CPU usage keep rising and remain high. This necessitates the detection of anomalous time series to pinpoint the hotspot machines. However, the state-of-the-art unsupervised time series anomaly detection (UTAD) methods are ineffective in this scenario. We identify two main reasons for this. First, the hotspot scenario requires us to pay particular attention to Flink-specific anomalies, e.g., slow-rising and high-level anomalies, which the existing methods struggle to address. Second, the state-of-the-art anomaly detection methods often assume that training datasets do not contain anomalies, but the data collected from the running Flink clusters contains noise, which causes these methods to learn anomalous patterns as normal patterns. In this paper, we first conduct experiments to analyze why existing methods fail in the Flink scenario. To tackle these challenges, we propose a cross-contrastive approach to learn the context information for each timestamp to enable Flink-specific anomaly detection. Then, to address noisy anomalies, we incorporate prior knowledge to set an anomaly boundary to prevent the model from learning anomalous patterns. Extensive experiments show that our method not only outperforms existing methods in the Flink scenario but also achieves state-of-the-art results on public benchmark datasets.