A performance Brokerage for Heterogeneous Clouds

19.5 Performance Measurement Results

In the previous chapter we reviewed, amongst other things, extant work relating to Cloud performance, the purpose of which was to identify opportunities for a performance broker, as well as to determine if a model of instance performance could be constructed from it. With regards to the former, we find that performance variation is reported across a range of instance types and providers, and in respect to a variety of different types of workloads. Indeed, performance variation is reported on for latency sensitive MPI codes, I/O bound workloads and CPU bound workloads. As such, there appears to be a number of potential opportunities for brokering performance.

However, brokering of MPI codes is arguably a niche market, whilst brokering of I/O bound workloads appears difficult as IOPS performance is too unpredictable. The performance of CPU bound workloads, and in particular the variation found, is tightly coupled to the underlying hardware. It is possible to identify differences in hardware by detecting differences in CPU model, and whilst further differences may exist we have found no way to easily detect them. As such, once an instance has been obtained, its performance measured and CPU model identified, we can estimate a future performance range with a degree of confidence. Further, this estimated range will be less wide than the possible range of performance before the instance was acquired. Arguably, being able to offer assurances over future performance of CPU bound workloads makes brokering of them possible.
However, extant work typically characterises performance in terms of mean and standard deviation or speed ups with respect to mean performance, with little, if any, consideration given to the characteristics of the performance range as a whole. We consider extant work to be insufficient to construct a realistic model of instance performance. Further, we find no available performance data sets which we could use for this purpose. As such, we have a requirement to generate new performance data.

In this chapter we describe the methodology employed when measuring performance; propose a number of hypotheses for testing; describe the experiments we conduct and analyse results. The objective is to provide a sufficiently detailed characterisation of performance so as to allow for the constructing of a model of it, which we do in the next chapter. To guide our experiments, and based on discussions of Cloud performance work, we proposed the followed hypotheses for testing in section 4.7.1:

H1: Performance variation is commonplace across different instance types and providers, and is found on homogeneous and heterogeneous instance types. Heterogeneity of instance types leads to variation and increased heterogeneity typically leads to increased variation.

H2: Heterogeneity typically produces consistent ranges of variation.

H3: Different workloads have different levels of variation due to differences in how they utilise underlying hardware.

H4: The allocation of CPU models to instances made within the same request is more irregular than extant assumptions allow for.

H5: Instances running on supposedly identical hardware, as identified by having the same CPU model, do not necessarily have the same performance levels over a given period of time.

H6: The actions of an instance can degrade the performance of its co-locating neighbour, the effect of which varies by workload.
In section 5.1 we discuss our empirical and statistical methodology. With regards to the former we discuss the need for both cross-sectional and longitudinal experiments. Further, we also note a requirement for use of commonly accepted benchmarks which are designed to avoid various benchmarking pitfalls as discussed in sections 3.5 and 3.6. With regards to the latter, we note Google Site Reliability Engineering recommend use of percentiles for system performance reporting as the data is often irregular and multi-modal, citing the mean as being misleading, and that the extremities are of interest as this is where SLA violations occur (Beyer et al., 2016). When considering the longitudinal data the most important property of the generated time series is stationarity, which means that the mean and variance of the data does not vary with time.
In section 5.2 through to 5.4 we experimentally verify H1-H3, which serves to provide additional supporting evidence for extant results. However, in addition to verification, we also provide a new characterisation of performance: The distribution of results obtained from a cross-section is multi-modal for heterogeneous instance types, with clustering based on CPU model close to the best possible per CPU value. This produces a highly peaked distribution of per CPU values with a long tail.
In section 5.5 we verify H4, demonstrating that that the allocation of CPU models to instances is more irregular than the assumptions made in the performance improvement strategies discussed in 4.2 would allow for. As Clouds periodically introduce new instance types, in section 5.6 we measure performance of the C4 and M4 family on EC2 to ensure our conclusions remain valid. In doing so we take the opportunity to increase the benchmark set so as to further our understanding of workload specific characteristics. As these are homogeneous instance types, variation found here raises questions as to both the temporal nature of performance amongst homogeneous instance types, and the causes of observed variation.
In section 5.7 we address H5. In particular, we conduct a longitudinal study of 50 instances from an instance type known, at the time of writing, to be homogeneous²¹. We find that different instances can have different means and variation over time, something not catered for in extant instance performance models. From this study we find that of the 48 instances for which we could collect results, 38 of them have stationary behaviour. Notably, this also establishes how future performance can be estimated from past performance for sufficiently well behaved instances. In section 5.9 we take advantage of dedicated instance types on EC2 in order to investigate resource contention and demonstrate how resource contention can severely degrade the performance of co-locating instances. This verifies H6.
In addition, in section 5.8 we also find correlations in performance of homogeneous instances; however the correlation differs by workload and also appears asymmetric. Finally, in 5.10 we summarise our findings.