Presentation

· Contributors · Organizations · Search Program · My Schedule · Happening Now · Maps

This content is available for: Workshop Reg Pass. Upgrade Registration

A data science pipeline synchronisation method for edge-fog-cloud continuum

SessionThe 18th Workshop on Workflows in Support of Large-Scale Science (WORKS23) - Part 2 of 2

DescriptionThis paper presents an adaptive continuum synchronisation method for data science pipelines deployed on edge-fog-cloud infrastructures. In a diagnostic phase, a model, based on the Bernoulli principle, is used as an analogy to create a global representation of bottlenecks in a pipeline. In a supervision phase, a watchman/sentinel cooperative system monitors and captures the throughput of the pipeline stages to create a bottleneck-stage scheme. In a rectification phase, this system produces replicas of stages identified as bottlenecks to mitigate the workload congestion using implicit parallelism and load balancing algorithms. This method is automatically and transparently invoked to produce in runtime a steady continuum dataflow. To test our proposal, we conducted a case study about the processing of medical and satellite data on fog-cloud infrastructures. The evaluation revealed that this method creates, without characterising workloads nor knowing infrastructure details, continuum dataflows, which yield a competitive performance with solutions in the state-of-the-art.

Author/Presenters