Keynote, HotTopiCS @ ICPE 2013
Over the past five years, Infrastructure-as-a-Service clouds have grown into the branch of ICT that offers services related to on-demand lease of storage, computation, and network. One of the major impediments in the selection and even use of (commercial) IaaS clouds is the lack of benchmarking results, that is, the lack of trustworthy quantitative information that allows (potential) cloud users to compare and reason about IaaS clouds.
In this talk we discuss empirical approaches to quantitative evaluation, which we find to be a necessary bumpy road toward cloud benchmarking. Both industry and academia have used empirical approaches for years, but the limited success achieved so far for IaaS clouds and similar systems (e.g., grids) is perhaps indicative of the complexity and size of challenges. We present initial results of our research into cloud workload characterization, including Big Data applications. We present the lessons we have learned in developing the SkyMark framework for cloud performance evaluation and the results of our SkyMark-based investigation of three research questions: What is the performance of production IaaS cloud services? How variable is the performance of widely used production cloud services? and What is the impact on performance of the user-level middleware, such as the provisioning and allocation policies that interact with IaaS clouds? We discuss the impact of our findings on large-scale, many-task, and many-user applications; notably, we discuss not only cloud performance, but also operation and behavior.
In contrast to previous attempts, our research combines empirical and other approaches, for example modeling and simulation, for gaining more insight; is based on a combination of short-term and multi-year measurements, for ensuring the longevity of our results; and uses large, comprehensive studies of several real clouds, for reducing the threats to validity related to the experimental environment. This presentation can also provide useful insights for fields related to benchmarking, for example experimental evaluation conducted in other large-scale distributed systems.
Last but not least, we present a roadmap toward cloud benchmarking and the way we plan to progress on it with other members of the RG Cloud Group of the Standard Performance Evaluation Corporation (SPEC).
URL: http://research.spec.org/working-groups/rg-cloud-working-group.html