A performance Brokerage for Heterogeneous Clouds

A Workload-Specific Performance Brokerage for Infrastructure Clouds
John O’Loughlin
Submitted in fulfillment of the requirements for the award of the degree Doctorate of Philosophy within the University of Surrey.

Department of Computer Science, University of Surrey

Guildford, GU2 7XH, United Kingdom

Supervised by Dr Lee Gillam

1.Declaration and Copyright

I confirm that the submitted work is my own work and that I have clearly identified and fully acknowledged all material that is entitled to be attributed to others (whether published or unpublished) using the referencing system set out in the programme handbook. I agree that the University may submit my work to means of checking this, such as the plagiarism detection service Turnitin® UK. I confirm that I understand that assessed work that has been shown to have been plagiarised will be penalised.

All rights reserved © 2017 John O’Loughlin.

The copyright of this thesis rests with the author. Copies (by any means) either in full, or of extracts, may not be made without prior consent from the author.

2.Table of Contents

1.Declaration and Copyright 2

2.Table of Contents 3

4.Dedications 4

6.Acknowledgements 5

7.Abstract 6

9.Table of Tables 8

10.Table of Figures 8

12.1 Introduction 9

1.1 The Cloud Performance Problem 9

1.2 A Performance Broker 11

1.3 Research Problem and Questions 13

1.4 Methodology and Scope 15

1.5 Contributions 17

1.6 Thesis Structure and Summary 22

14.2 Clouds, Utility and Commodity 25

2.1 Definition and Characteristics of Cloud Services 26

2.2 Elasticity 30

2.3 Service Models 32

2.4 Deployment Models 36

2.5 Regions and Availability Zones 38

2.6 VM Sizing: Instance Types 39

2.7 Utility and Commodity 43

2.8 Chapter Summary 48

16.3 Cloud Performance 51

3.1 Motivation: The Performance Problem 52

3.2 Provider Performance Commitments 56

3.3 Virtualisation 59

3.4 Performance Metrics 63

3.5 Compute Benchmarks 69

3.6 Chapter Summary 73

18.4 Cloud Performance, Brokers, Markets and Pricing 75

4.1 Cloud Performance 77

4.2 Performance Improvement Strategies 89

4.3 Brokers 94

4.4 Compute as a Commodity Market 107

4.5 Pricing 112

4.6 Cloud Simulators 115

4.7 Chapter Summary and Key Findings 117

19.5 Performance Measurement Results 126

5.1 Methodology 129

5.2 Performance Variation on EC2, Rackspace and GoGrid 136

5.3 Performance Consistency 145

5.4 Workload Specific Variation 150

5.5 Location Dependence 155

5.6 Performance Refresh: The C4 family 159

5.7 Longitudinal Variation 163

5.8 Performance Correlations 169

5.9 Performance Interference from co-location 173

5.10 Summary and Discussion 176

21.6 An On-demand Performance Brokerage 181

6.1 High Level Overview of Broker Operation 183

6.2 The Cloud Exchange (CeX) 188

6.3 Sellers on the CeX 190

6.4 Instance Performance 192

6.5 Performance Tranches and Pricing 198

6.6 The Broker’s Clients 200

6.7 Demand and Pool Management 207

6.8 Model Parameters 209

6.9 Simulation Events and Collected Statistics 209

6.10 Is the Broker Profitable? 213

6.11 Summary 216

22.7 Critique 219

7.1 Critique of Problem Choice 220

7.2 Critique of Chapter 5 224

7.3 Critique of Chapter 6 227

7.4 Chapter Summary 242

24.8 Conclusions and Future Work 245

8.1 Thesis Structure and Summary 246

8.2 Research Problem and Questions 248

8.3 Conclusions 256

8.4 Future Work 258

25.Bibliography 263

26.Appendix 280

A1: CPU Architecture and Performance 280

A2: Additional Performance Results 291

A3: SLA Pricing and Correlated Risks 293

A4: Risk Diversification and Instance Seeking 300

3.

4.Dedications

This thesis is dedicated to Natasha, without whose support, commitment and love it would not have been possible. And to Dylan, a constant source of love, joy and inspiration, especially as he claims to have completed two PhDs before the age of four.

This is also dedicated with love to Mum, Dad and to Nana. Also to Siobhan, Caroline, Harry and Felisa.
Finally, love to Daniel, Jordon, Finley, Chloe, Alice, Amaia, Saira and Naila, and in the hope that this will prove an inspiration to you all, showing what hard work can bring.

5.

6.Acknowledgements

I would like to thank Dr Lee Gillam for giving me the opportunity to work with him on this PhD, in doing so he has helped me to fulfill a long held ambition. In particular, I thank him for his support and encouragement in matters both academic and beyond, enabling me to achieve more than I thought possible at the start. He promised me fame, fortune and hard work, and without doubt has delivered on the latter. I’m sure the former will be forthcoming in time. Finally, I fully absolve him should a spherical cow be found in this document.

I would like to acknowledge and thank the University of Surrey for payment of fees.
Finally, I would like to thank a number of colleagues past and present. In particular, Gary Dear for being a constant source of encouragement and support, even if it did prove somewhat expensive in coffee, and Bin Li for many Cloud related discussions.

7.Abstract

Major Cloud providers offer a seemingly unlimited supply of compute resources for rent on-demand, with instances (virtual machines) being particularly popular amongst a range of service offerings. However, variation in performance across supposedly identical instances, supported by heterogeneous hardware, sold at the same price leads to variation in workload execution costs. Indeed, users pay more to have work delivered slower whilst missing out on the concomitant benefits of completing work faster and receive no performance assurances.

To address price, performance, and assurance, we propose and evaluate a broker that re-prices cloud provider instances according to measured performance levels to offer performance-assured instances. Such a broker does not impose changes on Cloud provider business models, but to be viable the broker must be profitable and yet profitable Cloud brokers seem not to exist in the literature.
We investigate broker profitability through simulations that model a commodity exchange analogously to extant financial exchanges, with sellers characterised in accordance with extant major Cloud providers and the broker modelled as a so-called Zero Intelligence (ZI) trading agent. Instance performance data are drawn from extensive benchmarking of Amazon’s Elastic Compute Cloud (EC2), and a Google workload trace comprising some 650,000 jobs provides for buyer willingness.
The maximum profit margin that could be achieved by the proposed broker across multiple demand profiles is 4%, and achieving this requires 4 different types of hardware that exhibit an average performance degrade of 52% from best to worst. A loss is made under a variety of other conditions. At best, such a broker would support a low margin high volume business, leaving it sensitive to market competition, vagaries in demand, exchange transaction fees and gaming strategies that clients may be able to employ. As such, we question the viability of brokers proposed elsewhere which claim to offer performance services of various kinds, despite profitability not having been evaluated and, worse, without operational costs addressed. 
Original contributions from this research include: (1) quantification and characterisation of performance variation amongst Cloud instances; (2) a model of instance performance that is qualitatively similar to results found empirically in both cross-sectional and longitudinal studies; (3) an exposition of the underestimation of risk in extant performance improvement strategies; (4) a Cloud broker offering performance-assured instances and the set of assumptions to be met for profitability; and (5) a strategy for minimising risks to performance due to correlated instance usage.