A performance Brokerage for Heterogeneous Clouds

1.3 Research Problem and Questions

Performance variation across supposedly identical instances is commonplace on Infrastructure Clouds and leads to a number of problems for users. Although performance variation is widely reported on, extant work on brokers fails to adequately address it; comparisons tend to focus on differences in performance between different providers at different prices, but not on differences on the same provider at the same price. The only work (that we are aware of) that directly addresses performance variation at the same price is so-called instance placement gaming that implements various strategies designed to improve performance.

Given that the problem is long standing and well known to providers, technical solutions are either not available or expensive to implement. Providers could supply performance-assured instances by taking run-time measurements before delivering instances. However this requires a change in their business model of mass production based on a design time specification which will likely drive up costs. Similarly, brokers that operate by negotiating between users and providers for specific needs require a compute marketplace where providers offer bespoke solutions. Notably, suggestions for future marketplaces based on the lines of extant commodity exchanges where instances (and other Cloud resources) can be traded also fail to acknowledge and address performance variation of supposedly identical instances.
A performance broker that assumes a commodity type marketplace and offers performance-assured instances, and in doing so creates a secondary marketplace, is a novel solution to the performance variation problem as there are no brokers (that we are aware of) that operate in a similar manner. However, this is only a viable solution if the broker is sustainable i.e. profitable. As such, the research problem this thesis addresses is:
To what extent can a performance broker profitably address performance variation in commodity Infrastructure Cloud marketplaces through offering performance-assured instances?
To structure and guide the investigation of our research problem we develop the following research questions:
RQ0: How should we specify and measure instance performance?
RQ1: How do brokers offering performance based services address variation at the same price?
RQ2: What are the risks for users of Cloud systems when seeking to take advantage of performance variation?
RQ3: How should we design performance based experiments?
RQ4: What are the major characteristics of performance variation?
RQ5: Can we find real-world examples of Cloud like systems whose users express performance needs and pay for different levels of performance?
RQ6: How should we model the broker?
RQ7: What are the risks the broker is exposed to when offering an on-demand performance service?
In the next section we describe the methodology we employ to address these questions, and the scope we consider.

1.4 Methodology and Scope

Peck (2004) defines a simulation as ‘Capture and mimic of real-world systems that, unlike real-world systems, can be acted upon’. A common use case for simulation is business planning, as it allows for investigation of different scenarios which otherwise would incur prohibitive costs, as would be the case if we were to investigate broker profitability by constructing and operating a real-world performance brokerage. As such, we intend to investigate our research problem through the use of simulation.

We must construct a model which is then translated into code. The model will need to specify the operation and interaction of a number of elements, which, a priori, must include clients who have various performance requirements, a marketplace where instances can be obtained, and providers who sell on the marketplace. Arguably the most important element will be instances with measurable performance properties. We must ensure the model we create is an accurate representation of the various elements it is mimicking. As such, important considerations when constructing and implementing a model in code are validation and verification, which Easterbrook (2010) succinctly describes as: Validation: Are we building the right system? Verification: Are we building the system right?
With regards to validation, Naylor and Finger (1967) suggest particular attention should be paid to the validly of the assumptions made when constructing a model. They categorise assumptions into structural assumptions and data assumptions. The former are with regards to how ‘things’ operate and interact. The latter are in regard to sources of data used, either directly as input or to inform how elements are constructed. We intend to ensure validity of assumptions we make by:

1. 
   Conducting a thorough review of relevant extant work on Cloud marketplaces, brokers, pricing and performance of instances.
   
2. 
   Where possible, verifying key findings. For example, we can verify data findings by conducting performance experiments.
   

Arguably, however, the most important component of the model is instance performance. We can either implement an appropriate existing performance model, if indeed any exist, or develop our own. Implementing an existing model requires accepting assumptions made in its development, and at a minimum we would need to verify the validity of these. Implementing our own model requires a source of performance data, and whilst we could make use of performance data reported in extant work, arguably we would need to empirically verify the data. Producing our own data allows for verification of extant results, but also allows for focus on the particular problem at hand: sufficiently characterising performance to allow for acceptance/rejection of extant performance models or development of a new one. Further, we would expect that observations from initial measurements will lead to formulation of new hypotheses for testing.

However, we are now faced with the issue of scope: what to measure and where. Undoubtedly, issues such as boot times and service response times will be of value to some users, however as we envisage a broker who has already acquired instances and measured their performance we do not consider these further. Our focus is on the performance of instances with respect to how well they run workloads. Different workloads have different access patterns in respect to underlying (hardware) resources, and the performance of a particular workload depends upon the resources it makes use of. A CPU bound workload is one whose performance is predominately, but not necessarily entirely, determined by the CPU, whilst for an I/O bound workload its performance is determined by the underlying storage system. CPU and I/O bound workloads are commonly considered to be most prevalent on Cloud systems. Indeed, recent developments have seen the Standard Performance Evaluation Corporation (SPEC) release a new Cloud benchmarking suite, the SPEC Cloud IaaS 2016, which makes use of 2 workloads: one I/O bound and one CPU bound.
We choose to focus primarily (but not exclusively) on compute and memory bandwidth performance of instances. We justify this choice by noting that variable storage performance on Clouds is beginning to be addressed by providers, most notably by AWS whose block level storage services offer so-called provisioned IOPS. This service allows for storage performance to be requested and is priced according to that performance level. This is notable as these are runtime assurances; however, for CPU or memory bandwidth bound workloads no such runtime performance assurances exist; indeed, the broker will provide these.
Further, it is not feasible to measure all possible instance types on all Clouds as it would be prohibitively expensive. As such, we limit the scope of empirical investigation to ‘common’ instance types on major IaaS Clouds. By common instance types we mean those without specialist hardware such as GPU cards, but include families such as general purpose, high-CPU and high memory. Further, empirical work is mainly, but not exclusively, conducted on EC2, a choice justified due to the scale of EC2, and the fact that it is commonly considered to be the de-facto standard for Cloud Computing. We also note that the majority of published work on Cloud performance is conducted on EC2 and so this readily offers for comparisons with such work.