Scalable Planning for Distributed Stream Processing Systems

Anton Riabov, Zhen Liu

Recently the problem of automatic composition of workflows has been receiving increasing interest. Initial investigation has shown that designing a practical and scalable composition algorithm for this problem is hard. A very general computational model of a workflow (e.g., BPEL) can be Turing-complete, which precludes fully automatic analysis of compositions. However, in many applications, workflow model can be simplified. We consider a model known as the Stream Processing Planning Language (SPPL), applicable in stream processing and other related domains. SPPL replaces the notion of concurrency by timeless functional computation. In addition, SPPL defines workflow metrics of resource consumption and quality of service. Experiments have shown earlier that even a naive SPPL planning algorithm significantly outperforms existing metric PDDL planners on stream processing workflow composition problems. In this paper we describe an efficient and scalable algorithm for finding high-quality approximate solutions for large instances of SPPL problems. We demonstrate the scalability of the algorithm on synthetic benchmarks that are derived from practical problems. We also give an example of SPPL model for practical problems.

This page is copyrighted by AAAI. All rights reserved. Your use of this site constitutes acceptance of all of AAAI's terms and conditions and privacy policy.