Measurement data reduction through variation rate metering

We present an efficient network measurement primitive that measures the rate of variations, or unique values for a given characteristic of a traffic flow. The primitive is widely applicable to a variety of data reduction and pre-analysis tasks at the measurement interface, and we show it to be particularly useful for building data-reducing preanalysis stages for scan detection within a multistage network analysis architecture. The presented approach is based upon data structures derived from Bloom filters, and as such yields high performance with probabilistic accuracy and controllable worst-case time and memory complexity. This predictability makes it suitable for hardware implementation in dedicated network measurement devices. One key innovation of the present work is that it is self-tuning, adapting to the characteristics of the measured traffic.