Dramatic changes in the technology landscape marked by increasing scales and pervasiveness of compute and data have resulted in the proliferation of edge applications aimed at effectively processing data in a timely manner. As the levels and fidelity of instrumentation increases and the types and volumes of available data grow, new classes of applications are being explored that seamlessly combine real-time data with complex models and data analytics to monitor and manage systems of interest. However, these applications require a fluid integration of resources at the edge, the core, and along the data path to support dynamic and data-driven application workflows, that is, they need to leverage a computing continuum. In this article, we present our vision for enabling such a computing continuum and specifically focus on enabling edge-to-cloud integration to support data-driven workflows. The research is driven by an online data-driven tsunami warning use case that is supported by the deployment of large-scale national environment observation systems. This article presents our overall approach as well as current status and next steps.
AbadiDJCarneyDÇetintemelU, et al. (2003) Aurora: a new model and architecture for data stream management. The VLDB Journal12(2): 120–139.
2.
AbadiDJAhmadYBalazinskaM, et al. (2005) The design of the borealis stream processing engine. Cidr5(2005): 277–289.
3.
AbdelBakyMDiaz-MontesJUnuvarM, et al. (2017a) Enabling distributed software-defined environments using dynamic infrastructure service composition. In: 2017 17th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID), Madrid, Spain, 14–17 May 2017, pp. 274–283. IEEE Press. DOI: 10.1109/CCGRID.2017.104.
4.
AbdelBakyMZouMZamaniAR, et al. (2017b) Computing in the continuum: combining pervasive devices and services to support data-driven applications. In: 2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS), Atlanta, GA, USA, 5–8 June 2017, pp. 1815–1824.IEEE.
5.
AltintasIBerkleyCJaegerE, et al. (2004) Kepler: an extensible system for design and execution of scientific workflows. In: Proceedings of the 16th International Conference on Scientific and Statistical Database Management, Santorini Island, Greece, 23–23 June 2004, pp. 423–424.IEEE.
6.
AmadeoMCampoloCMolinaroA (2016) Information-centric networking for connected vehicles: a survey and future perspectives. IEEE Communications Magazine54(2): 98–104.
BennettJCAbbasiHBremerPT, et al. (2012) Combining in-situ and in-transit processing to enable extreme-scale scientific analysis. In: Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis, Salt Lake City, Utah, 10–16 November 2012, p. 49.IEEE Computer Society Press.
9.
BhatVParasharMKlaskyS (2007) Experiments with in-transit processing for data intensive grid workflows. In: Proceedings of the Eighth IEEE/ACM International Conference on Grid Computing, Austin, Texas, USA, 19–21 September 2007, pp. 193–200. IEEE Computer Society.
10.
BlytheJJainSDeelmanE, et al. (2005) Task scheduling strategies for workflow-based applications in grids. In: IEEE International Symposium on Cluster Computing and the Grid, 2005. CCGrid 2005, Vol. 2, Cardiff, Wales, UK, 9–12 May 2005, pp. 759–767. IEEE.
11.
BormannPDi GiacomoD (2011) The moment magnitude mw and the energy magnitude me: common roots and differences. Journal of Seismology15(2): 411–427.
12.
CarbonePKatsifodimosAEwenS, et al. (2015) Apache flink: Stream and batch processing in a single engine. Bulletin of the IEEE Computer Society Technical Committee on Data Engineering36(4).
13.
CherniackMBalakrishnanHBalazinskaM, et al. (2003) Scalable distributed stream processing. CIDR3: 257–268.
14.
Da XuLHeWLiS (2014) Internet of things in industries: a survey. IEEE Transactions on Industrial Informatics10(4): 2233–2243.
15.
DalalNTriggsB (2005) Histograms of oriented gradients for human detection. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Vol. 1. San Diego, CA, USA, 20–25 June 2005, pp. 886–893. IEEE.
16.
DeelmanESinghGSuMH, et al. (2005) Pegasus: a framework for mapping complex scientific workflows onto distributed systems. Scientific Programming13(3): 219–237.
17.
DeelmanEPeterkaTAltintasI, et al. (2017) The future of scientific workflows. The International Journal of High Performance Computing Applications32(1): 159–175.
18.
DíazMMartínCRubioB (2016) State-of-the-art, challenges, and open issues in the integration of internet of things and cloud computing. Journal of Network and Computer applications67: 99–117.
19.
Diaz-MontesJAbdelBakyMZouM, et al. (2015) Cometcloud: enabling software-defined federations for end-to-end application workflows. IEEE Internet Computing19(1): 69–73.
20.
FabianNMorelandKThompsonD, et al. (2011) The Paraview Coprocessing Library: a scalable, general purpose in situ visualization library. In: 2011 IEEE Symposium on Large Data Analysis and Visualization (LDAV), Providence, Rl, USA, 23–24 October 2011, pp. 89–96. IEEE.
21.
FreyJTannenbaumTLivnyM, et al. (2002) Condor-G: a computation management agent for multi-institutional grids. Cluster Computing5(3): 237–246.
22.
Gibert RenartEBalouek-ThomertDHuX, et al. (2017) Online decision-making using edge resources for content-driven stream processing. In: IEEE International Conference on e-Science, Auckland, New Zealand, 24–27 October 2017, IEEE.
23.
HeintzBChandraASitaramanRK (2016) Trading timeliness and accuracy in geo-distributed streaming analytics. In: Proceedings of the Seventh ACM Symposium on Cloud Computing, Santa Clara, CA, USA, 5–7 October 2016, pp. 361–373. ACM
24.
JonesN (2018) Top strategic iot trends and technologies through 2023. Gartner Research Barcelona Spain.
25.
KarimRDingCMiriA (2013) An end-to-end QoS mapping approach for cloud service selection. In: 2013 IEEE Ninth World Congress on Services (SERVICES), Santa Clara, CA, USA, 28 June–3 July 2013, pp. 341–348. IEEE.
26.
KlaskySALudaescherBParasharM (2006) The center for plasma edge simulation workflow requirements. In: Proceedings of the 22nd International Conference on Data Engineering Workshops, 2006., Atlanta, GA, USA, 3–7 April 2006, pp. 73–73. IEEE.
27.
KloudasKMamedeMPreguiçaN, et al. (2015) Pixida: optimizing data parallel jobs in wide-area data analytics. Proceedings of the VLDB Endowment9(2): 72–83.
28.
KrepsJCorpLNarkhedeN, et al. (2011) Kafka: a distributed messaging system for log processing. In:NetDB’11 12 June 2011, Athens, Greece.
29.
KuhlenTPajarolaRZhouK (2011) Parallel in situ coupling of simulation with a fully featured visualization system. In: Proceedings of the 11th Eurographics Conference on Parallel Graphics and Visualization (EGPGV), Llandudno, UK, 10–11 April 2011, pp. 101–109. Aire-la-Ville, Switzerland: Eurographics Association.
30.
KulkarniSBhagatNFuM, et al. (2015) Twitter heron: Stream processing at scale. In: ACM SIGMOD Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data, Melbourne, Victoria, Australia, 31 May –4 June 2015, pp. 239–250. New York, NY, USA: ACM.
31.
LuNChengNZhangN, et al. (2014) Connected vehicles: solutions and challenges. IEEE Internet of Things Journal1(4): 289–299.
32.
MaswoodMMSNasimRKasslerAJ, et al. (2018) Cost-efficient resource scheduling under QoS constraints for geo-distributed data centers. In: NOMS 2018-2018 IEEE/IFIP Network Operations and Management Symposium, Taipei, Taiwan, 23-27 April 2018, pp. 1–9. IEEE.
33.
McKeownNAndersonTBalakrishnanH, et al. (2008) Openflow: enabling innovation in campus networks. ACM SIGCOMM Computer Communication Review38(2): 69–74.
34.
MorelandKOldfieldRMarionP, et al. (2011) Examples of in transit visualization. In: Proceedings of the Second International Workshop on Petascal Data Analytics: Challenges and Opportunities, Seattle, Washington, USA, 14 November 2011, pp. 1–6. New York, NY, USA: ACM.
35.
NasiriHNasehiSGoudarziM (2018) A survey of distributed stream processing systems for smart city data analytics. In: Proceedings of the International Conference on Smart Cities and Internet of Things, SCIOT'18, New York, NY, USA, pp. 12:1–12:7. ACM.
36.
PandeySWuLGuruSM, et al. (2010) A particle swarm optimization-based heuristic for scheduling workflow applications in cloud computing environments. In: 2010 24th IEEE International Conference on Advanced Information Networking and Applications (AINA), Perth, WA, Australia, 20–23 April 2010, IEEE, pp. 400–407.
37.
ParasharMAndersonSDeelmanE, et al. (2016) Large facilities cyberinfrastructure: report from the NSF large facilities cyberinfrastructure workshop. DOI: 10.7278/S5SN074P.
PuQAnanthanarayananGBodikP, et al. (2015) Low latency geo-distributed data analytics. ACM SIGCOMM Computer Communication Review45(4): 421–434.
40.
RabkinAAryeMSenS, et al. (2014) Aggregation and degradation in JetStream: streaming analytics in the wide area. In: NSDI, Vol. 14, Seattle, WA, USA, 2–4 April 2014, pp. 275–288.
41.
RosenbergFAntonCMichlmayrA, et al. (2009) An end-to-end approach for QoS-aware service composition. In: IEEE InternationalEnterprise Distributed Object Computing Conference, 2009. EDOC’09, Auckland, New Zealand, 1–4 September 2009, pp. 151–160. IEEE.
42.
SantosITillyMChandramouliB, et al. (2013) Dial: distributed streaming analytics anywhere, anytime. Proceedings of the VLDB Endowment6(12): 1386–1389.
43.
SchroederWParasharMShoshaniA, et al. (2011) In situ data processing for extreme scale computing. Proceedings of SciDAC.
44.
ShiWCaoJZhangQ, et al. (2016) Edge computing: vision and challenges. IEEE Internet of Things Journal3(5): 637–646.
45.
SmithLMBarthJAKelleyDS, et al. (2018) The ocean observatories initiative. Oceanography31:16–35.
46.
SrinivasMPatnaikLM (1994) Genetic algorithms: a survey. Computer27(6): 17–26.
47.
ToshniwalA (2014) Storm@twitter. In: Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data, SIGMOD ‘14, Snowbird, Utah, USA, 22–27 June 2014, pp. 147–156. New York, NY, USA: ACM
48.
TudoranRNanoOSantosI, et al. (2014) JetStream: enabling high performance event streaming across cloud data-centers. In: Proceedings of the Eighth ACM International Conference on Distributed Event-Based Systems, Mumbai, India, 26–29 May 2014, pp. 23–34. New York, NY,USA: ACM.
49.
VermaAKaushalS (2012) Deadline and budget distribution based cost-time optimization workflow scheduling algorithm for cloud. In: IJCA Proceedings on International Conference on Recent Advances and Future Trends in Information Technology (iRAFIT 2012), Vol. 4. iRAFIT (7), pp. 1–4.
50.
VermaAKaushalS (2014) Deadline constraint heuristic-based genetic algorithm for workflow scheduling in cloud. International Journal of Grid and Utility Computing5(2): 96–106.
51.
VulimiriACurinoCGodfreyB, et al. (2015) WANalytics: analytics for a geo-distributed data-intensive world. In: CIDR, Asilomar, California, USA, 4–7 January 2015.
52.
YangS (2017) IoT stream processing and analytics in the Fog. arXiv preprint arXiv:1705.05988.
53.
YuJBuyyaR (2006) Scheduling scientific workflow applications with deadline and budget constraints using genetic algorithms. Scientific Programming14(3–4): 217–230.
54.
ZahariaMChowdhuryMFranklinMJ, et al. (2010) Spark: cluster computing with working sets. In: Proceedings of the Second USENIX Conference on Hot Topics in Cloud Computing, HotCloud’10, Boston, MA, 22–25 June 2010, pp. 10-10. Berkeley, CA, USA: USENIX Association. Available at: http://dl.acm.org/citation.cfm?id=1863103.1863113 (accessed September 2019).
55.
ZamaniARAbdelBakyMBalouek-ThomertD, et al. (2017) Supporting data-driven workflows enabled by large scale observatories. In: 2017 IEEE 13th International Conference on e-Science (e-Science), Auckland, New Zealand, 24–27 October 2017, pp. 592–595. IEEE.
56.
ZamaniARAbdelBakyMBalouek-ThomertD, et al. (2019) Submarine: a subscription-based data streaming framework for integrating large facilities and advanced cyberinfrastructure. Concurrency and Computation: Practice and Experience, p.5256. Epub ahead of print 11 April 2019. DOI: 10.1002/cpe.5256.
57.
ZamaniARZouMDiaz-MontesJ, et al. (2017c) Deadline constrained video analysis via in-transit computational environments. IEEE Transactions on Services Computing, p.5256.
58.
ZamaniARBalouek-ThomertDVillalobosJ, et al. (2018) Runtime management of data quality for scientific observatories using edge and in-transit resources. In: 2018 30th international symposium on computer architecture and high performance computing, Lyon, France, 24 September 2018, pp. 274–281.
59.
ZhengY (2011) Location-based social networks: users. In: ZhengY.ZhouX. (eds) Computing with Spatial Trajectories. Berlin: Springer, pp. 243–276.
60.
ZhengYCapraLWolfsonO, et al. (2014) Urban computing: concepts, methodologies, and applications. ACM Transactions on Intelligent Systems and Technology (TIST)5(3): 38.
