Building the Infrastructure Memex: VU on Operational Data Analytics in the 21st Century

Abstract

Our society has turned digital: From science to business, from online shopping to online gaming, from education to government, digital applications depend every moment on Information and Communication Technology (ICT) infrastructure. To pilot this infrastructure and navigate through it, we need detailed information and decision-making support, provided on-time, cheaply, and reliably. (We briefly argue non-data-driven approaches are currently unable to cope with real conditions.) In this talk, we argue that Operational Data Analytics (ODA) can provide these capabilities, but requires advances not only in monitoring and observability, but also in data sourcing and ontology mapping, data cleaning and filtering, data characterization and modeling, and process management. Furthermore, we argue a digital twin, capable to simulate both the current conditions and to make long-term predictions, should be integrated with the ODA system - providing an advanced form of A - to help with resource management and scheduling, pipeline optimization, energy awareness, general system tuning, capacity planning, etc. We present a reference architecture for ODA, a partial analysis of the state of the art, and experience with data collection and its use in digital twinning of datacenters. This work aligns with and benefits from work in collaboration with the SPEC RG Cloud Group and, among others, the EU Graph-Massivizer project and the OffSense and 6G Future Network Services projects in the Netherlands