Micro-Workflows Data Stream Processing Model for Industrial Internet of Things

Ameer B. A. Alaasam; Gleb I. Radchenko; Andrey N. Tchernykh

doi:10.14529/jsfi210106

Authors

Ameer B. A. Alaasam South Ural State University
Gleb I. Radchenko South Ural State University
Andrey N. Tchernykh South Ural State University CICESE Research Center Ivannikov Institute for System Programming of the RAS

DOI:

https://doi.org/10.14529/jsfi210106

Abstract

The fog computing paradigm has become prominent in stream processing for IoT systems where cloud computing struggles from high latency challenges. It enables the deployment of computational resources between the edge and cloud layers and helps to resolve constraints, primarily due to the need to react in real-time to state changes, improve the locality of data storage, and overcome external communication channels’ limitations. There is an urgent need for tools and platforms to model, implement, manage, and monitor complex fog computing workflows. Traditional scientific workflow management systems (SWMSs) provide modularity and flexibility to design, execute, and monitor complex computational workflows used in smart industry applications. However, they are mainly focused on batch execution of jobs consisting of tightly coupled tasks. Integrating data streams into SWMSs of IoT systems is challenging. We proposed a microworkflow model to redesign the monolith architecture of workflow systems into a set of smaller and independent workflows that support stream processing. Micro-workflow is an independent data stream processing service that can be deployed on different layers of the fog computing environment. To validate the feasibility and practicability of the micro-workflow refactoring, we provide intensive experimental analysis evaluating the interval between sensor messages, the time interval required to create a message, between sending sensor message and receiving the message in SWMS, including data serialization, network latency, etc. We show that the proposed decoupling support of the independence of implementation, execution, development, maintenance, and cross-platform deployment, where each micro-workflow becomes a standalone computational unit, is a suitable mechanism for IoT stream processing.

References

Aazam, M., Zeadally, S., Harras, K.A.: Deploying Fog Computing in Industrial Internet of Things and Industry 4.0. IEEE Transactions on Industrial Informatics 14(10), 4674–4682 (2018), DOI: 10.1109/TII.2018.2855198

Alaasam, A.B.A., Radchenko, G., Tchernykh, A., Borodulin, K., Podkorytov, A.: Scientific Micro-Workflows: Where Event-Driven Approach Meets Workflows to Support Digital Twins. In: Proceedings of the International Conference Russian Supercomputing Days (RuSCDays’18), 24-25 Sept. 2018, Moscow, Russia. vol. 1, pp. 489–495 (2018)

Alaasam, A.B.A., Radchenko, G., Tchernykh, A.: Stateful Stream Processing for Digital Twins: Microservice-Based Kafka Stream DSL. In: 2019 International Multi-Conference on Engineering, Computer and Information Sciences (SIBIRCON), 21-27 Oct. 2019, Novosibirsk, Russia. pp. 0804–0809. IEEE (2019), DOI: 10.1109/SIBIRCON48586.2019.8958367

Alaasam, A.B.A., Radchenko, G., Tchernykh, A., González Compeán, J.L.: Analytic Study of Containerizing Stateful Stream Processing as Microservice to Support Digital Twins in Fog Computing. Programming and Computer Software 46(8), 511–525 (2020), DOI: 10.1134/S0361768820080083

Badia, R.M., Ayguade, E., Labarta, J.: Workflows for science: a challenge when facing the convergence of HPC and Big Data. Supercomputing Frontiers and Innovations 4(1), 27–47 (2017), DOI: 10.14529/jsfi170102

Bonomi, F., Milito, R., Zhu, J., Addepalli, S.: Fog computing and its role in the internet of things. In: Proceedings of the First Edition of the MCCWorkshop on Mobile Cloud Computing, 13-17 Aug. 2012, Helsinki, Finland. pp. 13–15. Association for Computing Machinery, New York, NY, USA (2012), DOI: 10.1145/2342509.2342513

Boyes, H., Hallaq, B., Cunningham, J., Watson, T.: The industrial internet of things (IIoT): An analysis framework. Computers in Industry 101(December 2017), 1–12 (2018), DOI: 10.1016/j.compind.2018.04.015

Cao, J., Zhang, Q., Shi, W.: Challenges and Opportunities in Edge Computing, pp. 59–70. Springer, Cham (2018), DOI: 10.1007/978-3-030-02083-5_5

Carvalho, O., Roloff, E., Navaux, P.O.: A Distributed Stream Processing based Architecture for IoT Smart Grids Monitoring. In: Companion Proceedings of the 10th International Conference on Utility and Cloud Computing, 5-8 Dec. 2017, Austin, Texas, USA. pp. 9–14. ACM, New York, NY, USA (2017), DOI: 10.1145/3147234.3148105

Chandy, K.M.: Event Driven Architecture. In: Encyclopedia of Database Systems, pp. 1040–1044. Springer US, Boston, MA (2009), DOI: 10.1007/978-0-387-39940-9_570

Deelman, E., Singh, G., Su, M.H., et al.: Pegasus: A framework for mapping complex scientific workflows onto distributed systems. Scientific Programming 13(3), 219–237 (2005), DOI: 10.1155/2005/128026

Goyal, P., Mikkilineni, R.: Policy-Based Event-Driven Services-Oriented Architecture for Cloud Services Operation & Management. In: 2009 IEEE International Conference on Cloud Computing, 21-25 Sept. 2009, Bangalore, India. pp. 135–138. IEEE (2009), DOI: 10.1109/CLOUD.2009.76

Haag, S., Anderl, R.: Digital twin – Proof of concept. Manufacturing Letters 15, 64–66 (2018), DOI: 10.1016/j.mfglet.2018.02.006

Hao, Z., Novak, E., Yi, S., Li, Q.: Challenges and Software Architecture for Fog Computing. IEEE Internet Computing 21(2), 44–53 (2017), DOI: 10.1109/MIC.2017.26

Hirales-Carbajal, A., Tchernykh, A., Roblitz, T., Yahyapour, R.: A Grid simulation framework to study advance scheduling strategies for complex workflow applications. In: 2010 IEEE International Symposium on Parallel & Distributed Processing, Workshops and Phd Forum (IPDPSW), 19-23 April 2010, Atlanta, GA, USA. pp. 1–8. IEEE (2010), DOI: 10.1109/IPDPSW.2010.5470918

Hirales-Carbajal, A., Tchernykh, A., Yahyapour, R., et al.: Multiple Workflow Scheduling Strategies with User Run Time Estimates on a Grid. Journal of Grid Computing 10(2), 325–346 (2012), DOI: 10.1007/s10723-012-9215-6

Iturriaga, S., Nesmachnow, S., Tchernykh, A., Dorronsoro, B.: Multiobjective Workflow Scheduling in a Federation of Heterogeneous Green-Powered Data Centers. In: 2016 16th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid), 16-19 May 2016, Cartagena, Colombia. pp. 596–599. IEEE (2016), DOI: 10.1109/CCGrid.2016.34

Kalske, M., Mäkitalo, N., Mikkonen, T.: Challenges When Moving from Monolith to Microservice Architecture. In: Current Trends inWeb Engineering, 5-8 June 2017, Rome, Italy, pp. 32–47. Springer, Cham (2018), DOI: 10.1007/978-3-319-74433-9_3

Korambath, P., Wang, J., Kumar, A., Davis, J., Graybill, R., Schott, B., Baldea, M.: A Smart Manufacturing Use Case: Furnace Temperature Balancing in Steam Methane Reforming Process via Kepler Workflows. Procedia Computer Science 80, 680–689 (2016), DOI: 10.1016/j.procs.2016.05.357

Li, X., Song, J., Huang, B.: A scientific workflow management system architecture and its scheduling based on cloud service platform for manufacturing big data analytics. The International Journal of Advanced Manufacturing Technology 84(1-4), 119–131 (2016), DOI: 10.1007/s00170-015-7804-9

Luo, J., Yin, L., Hu, J., Wang, C., Liu, X., Fan, X., Luo, H.: Container-based fog computing architecture and energy-balancing scheduling algorithm for energy IoT. Future Generation Computer Systems 97, 50–60 (2019), DOI: 10.1016/j.future.2018.12.063

Miranda, V., Tchernykh, A., Kliazovich, D.: Dynamic Communication-Aware Scheduling with Uncertainty of Workflow Applications in Clouds. In: High Performance Computer Applications, 9-13 March 2015, Mexico City, Mexico, Communications in Computer and Information Science, vol. 595, pp. 169–187. Springer, Cham (2016), DOI: 10.1007/978-3-319-32243-8_12

Naseri, M., Towhidi, A.: Stateful Web Services: A Missing Point in Web Service Standards. In: Proceedings of the International MultiConference of Engineers and Computer Scientists 2007 (IMECS 2007). pp. 993–997. Hong Kong, China (2007)

Peiffer, C., L’Heureux, I.: System and method for maintaining statefulness during client-server interactions. (12) United States Patent (US8346848B2) (2013), https://patents.google.com/patent/US8346848B2/en

Plociennik, M., Zok, T., Altintas, I., et al.: Approaches to Distributed Execution of Scientific Workflows in Kepler. Fundamenta Informaticae 128(3), 281–302 (2013), DOI: 10.3233/FI-2013-947

Qamsane, Y., Chen, C.Y., Balta, E.C., et al.: A unified digital twin framework for real-time monitoring and evaluation of smart manufacturing systems. In: 2019 IEEE 15th International Conference on Automation Science and Engineering (CASE), 22-26 Aug. 2019, Vancouver, BC, Canada. pp. 1394–1401 (2019), DOI: 10.1109/COASE.2019.8843269

Radchenko, G., Alaasam, A.B., Tchernykh, A.: Micro-Workflows: Kafka and Kepler Fusion to Support Digital Twins of Industrial Processes. In: 2018 IEEE/ACM International Conference on Utility and Cloud Computing Companion (UCC Companion), 17-20 Dec. 2018, Zurich, Switzerland. pp. 83–88. IEEE (2018), DOI: 10.1109/UCC-Companion.2018.00039

Richards, M.: Software Architecture Patterns. O’Reilly Media, 1005 Gravenstein Highway North, Sebastopol, CA 95472 (2015)

Savchenko, D., Radchenko, G., Taipale, O.: Microservices validation: Mjolnirr platform case study. In: 2015 38th International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO), 25-29 May 2015, Opatija, Croatia. pp. 235–240. IEEE (2015), DOI: 10.1109/MIPRO.2015.7160271

da Silva, R.F., Pottier, L., Coleman, T., Deelman, E., Casanova, H.: WorkflowHub: Community Framework for Enabling Scientific Workflow Research and Development. In: 2020 IEEE/ACM Workflows in Support of Large-Scale Science (WORKS), 12 Nov. 2020, GA, USA. pp. 49–56. IEEE (2020), DOI: 10.1109/WORKS51914.2020.00012

Simpkin, C., Taylor, I., Harborne, D., Bent, G., Preece, A., Ganti, R.K.: Efficient orchestration of Node-RED IoT workflows using a Vector Symbolic Architecture. Future Generation Computer Systems 111, 117–131 (2020), DOI: 10.1016/j.future.2020.04.005

Tao, F., Sui, F., Liu, A., et al.: Digital twin-driven product design framework. International Journal of Production Research 57(12), 3935–3953 (2019), DOI: 10.1080/00207543.2018.1443229

Yang, P.C., Purawat, S., U. Ieong, P., et al.: A demonstration of modularity, reuse, reproducibility, portability and scalability for modeling and simulation of cardiac electrophysiology using Kepler Workflows. PLOS Computational Biology 15(3), e1006856 (2019), DOI: 10.1371/journal.pcbi.1006856

Zhang, Q., Cheng, L., Boutaba, R.: Cloud computing: state-of-the-art and research challenges. Journal of Internet Services and Applications 1(1), 7–18 (2010), DOI: 10.1007/s13174-010-0007-6

Zheng, C., Tovar, B., Thain, D.: Deploying high throughput scientific workflows on container schedulers with makeflow and mesos. Proceedings of the 17th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID 2017), 14-17 May 2017, Madrid, Spain pp. 130–139 (2017), DOI: 10.1109/CCGRID.2017.9