Optimal operator deployment and replication for elastic distributed data stream processing

By processing data in a real-time fashion, Data Stream Processing (DSP) applications are receiving an increasing interest for building new pervasive services. Due to the unpredictable data source rate, DSP applications demand high dynamism and elasticity, so to acquire and release computing resources in response to workload fluctuations. In this paper, we deal with a key problem for the effective runtime management of a DSP application in geo-distributed environments: we investigate the placement and replication decisions while considering the application and resource heterogeneity and the migration overhead, so to select the optimal adaptation strategy that can minimize migration costs while satisfying the application Quality of Service (QoS) requirements. We present Elastic DSP Replication and Placement (EDRP), a unified framework for the QoS aware initial deployment and runtime elasticity management of DSP applications. In EDRP, the deployment and runtime decisions are driven by the solution of a suitable integer linear programming problem, whose objective function captures the relative importance between QoS goals and reconguration costs. We also present the implementation of EDRP and the related mechanisms on Apache Storm. We conduct a thorough experimental evaluation, both numerical and prototype-based, that shows the benefits achieved by EDRP on the application performance.