Meeting Coflow Deadlines in Data Center Networks With Policy-Based Selective Completion

DOI number:10.1109/TNET.2022.3187821

Affiliation of Author(s):chool of Computing and Artificial Intelligence, Southwest Jiaotong University

Journal:IEEE/ACM Transactions on Networking

Key Words:Coflow,data center networks,flow scheduling

Abstract:Recently, the abstraction of coflow is introduced to capture the collective data transmission patterns among modern distributed data-parallel applications. During processing, coflows generally act as barriers; accordingly, time-sensitive applications prefer their coflows to complete within deadlines, and deadline-aware coflow scheduling becomes very crucial. Regarding these data-parallel applications, we notice that many of them, including large-scale query systems, distributed iterative training, and erasure codes enabled storage, are able to tolerate loss-bounded incomplete inputs by design. This tolerance indeed brings a flexible design space for the schedule of their coflows: when getting overloaded, the network can trade coflow completeness for the timeliness, and balance the completeness of different coflows on demand. Unfortunately, existing coflow schedulers neglect this tolerance, resulting in inflexible and inefficient bandwidth allocations. In this paper, we explore this fundamental trade-off and design, a POlicy-based COflow scheduler, along with a transport layer enhancement scheme, to achieve customizable selective coflow completion for emerging time-sensitive distributed applications. Internally, employs a suite of novel designs along with admission controls to make flexible, work-conserving, and performance-guaranteed rate allocation to online coflow requests very efficiently. Extensive trace-based simulations indicate that is highly flexible and achieves optimal coflow schedules respecting the requirements specified by applications.

Co-author:Shouxi Luo,Pingzhi Fan,Huanlai Xing,Hongfang Yu

Document Code:10.1109/TNET.2022.3187821

Page Number:1-14

ISSN No.:1063-6692

Translation or Not:no

Date of Publication:2022-07-12

Included Journals:SCI