On the Cost of Acking in Data Stream Processing Systems

Alessio Pagliari, Fabrice Huet, Guillaume Urvoy-Keller. On the Cost of Acking in Data Stream Processing Systems. 2019 19th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing (CCGRID)

Data Stream Processing, Message Reliability, Apache Storm, Acking Framework, Scheduling

The widespread use of social networks and applications such as IoT networks generates a continuous stream of data that companies and researchers want to process, ideally in real-time. Data stream processing systems (DSP) enable such continuous data analysis by implementing the set of operations to be performed on the stream as directed acyclic graph (DAG) of tasks. While these DSP systems embed mechanisms to ensure fault tolerance and message reliability, only few studies focus on the impact of these mechanisms on the performance of applications at runtime. In this paper, the authors demonstrate the impact of the message reliability mechanism on the performance of the application. They use an experimental approach, using the Storm middleware, to study an acknowledgment-based framework. They compare the two standard schedulers available in Storm with applications of various degrees of parallelism, over single and multi cluster scenarios. The authors show that the acking layer may create an unforeseen bottleneck due to the acking tasks placement; a problem which, to the best of their knowledge, has been overlooked in the scientific and technical literature. The authors propose two strategies for improving the acking tasks placement and demonstrate their benefit in terms of throughput and latency.