Evaluating Task-Level CPU Efficiency for Distributed Stream Processing Systems
Journal
Big Data and Cognitive Computing
ISSN
2504-2289
Type
journal article
Date Issued
2023-03-10
Author(s)
Rank, Johannes
;
Pereira, Ella
;
Pereira, Rubem
;
Min, Geyong
;
Herget, Jonas
;
;
Krcmar, Helmut
Abstract
Big Data and primarily distributed stream processing systems (DSPSs) are growing in complexity and scale. As a result, effective performance management to ensure that these systems meet the required service level objectives (SLOs) is becoming increasingly difficult. A key factor to consider when evaluating the performance of a DSPS is CPU efficiency, which is the ratio of the workload processed by the system to the CPU resources invested. In this paper, we argue that developing new performance tools for creating DSPSs that can fulfill SLOs while using minimal resources is crucial. This is especially significant in edge computing situations where resources are limited and in large cloud deployments where conserving power and reducing computing expenses are essential. To address this challenge, we present a novel task-level approach for measuring CPU efficiency in DSPSs. Our approach supports various streaming frameworks, is adaptable, and comes with minimal overheads. This enables developers to understand the efficiency of different DSPSs at a granular level and provides insights that were not previously possible.
Language
English (United States)
Keywords
CPU efficiency
big data
distributed stream processing
performance
task-level measurement
profiling
flink
spark
HSG Classification
contribution to scientific community
Refereed
Yes
Volume
7
Number
1
Subject(s)
Division(s)
File(s)![Thumbnail Image]()
Name
BDCC-07-00049-v2.pdf
Size
7.25 MB
Format
Adobe PDF
Checksum (MD5)
1cf46d4b8be853fd603f27a2863b77e7