Selective Coflow Completion for Time-sensitive Distributed Applications with Poco

DOI number:10.1145/3404397.3404449

Journal:49th International Conference on Parallel Processing（ICPP）

Abstract:Recently, the abstraction of coflow is introduced to capture the collective data transmission patterns among modern distributed data-parallel application. During processing, coflows generally act as barriers; accordingly, time-sensitive applications prefer their coflows to complete within deadlines and deadline-aware coflow scheduling becomes very crucial. Regarding these data-parallel applications, we notice that many of them, including large-scale query system, distributed iterative training, and erasure codes enabled storage, are able to tolerate loss-bounded incomplete inputs by design. This tolerance indeed brings a flexible design space for the schedule of their coflows: when getting overloaded, the network can trade coflow completeness for timeliness, and balance the completenesses of different coflows on demand. Unfortunately, existing coflow schedulers neglect this tolerance, resulting in inflexible and inefficient bandwidth allocations. In this paper, we explore this fundamental trade-off and design Poco, a POlicy-based COflow scheduler, to achieve customizable selective coflow completions for these emerging time-sensitive distributed applications. Internally, Poco employs a suite of novel designs along with admission controls to make flexible, work-conserving, and performance-guaranteed rate allocation to online coflow requests very efficiently. Extensive trace-based simulations indicate that Poco is highly flexible and achieves optimal coflow schedules respecting the requirements specified by applications.

Co-author:Shouxi Luo,Pingzhi Fan,Huanlai Xing,Hongfang Yu

Document Code:10.1145/3404397.3404449

Translation or Not:no

Date of Publication:2020-08-31