A performance Brokerage for Heterogeneous Clouds

5.3 Performance Consistency

We now address hypotheses H2 and H3 by conducting a new cross-sectional study. We recall that:

H2: Heterogeneity typically produces consistent ranges of variation.

H3: Different workloads have different levels of variation due to differences in how they utilise underlying hardware.
To investigate both performance consistency and workload specific performance variation, we increased the size of our benchmarking set to include NAMD, POV-Ray and GNUGO, allowing us to further investigate workload specific performance characteristics of CPU bound workloads. As each instance will run more benchmarks than those in section 5.2, we have an increase in instance run times, and so a potential increase in per instance benchmark costs. To manage our costs we restrict ourselves to m1.small instances and m2.large only.
We attempt to benchmark 550 m1.small spot instances, of which 511 completed the benchmarks and the rest were reclaimed by EC2 before completion. Notably, we identified 2 CPU models not found earlier: the AMD HE 2281 and Xeon E5-2651. In terms of release dates, these are the oldest and newest M1 CPU, respectively. In our previous sample we did not identify any AMD models due their scarcity, whilst the E5-2651 had not yet been introduced. For m2.large, we measured 80 instances and detected the same 2 CPU models as previously, namely the E5-2665 and X5550. [Note: we refer to the AMD HE 2281 simply as AMD].
In this section we discuss consistency i.e. H2, leaving the discussion of H3 to the next section. If performance is consistent we would expect the bzip2 results on our new sample of m1.small and M2 instances to be quantitatively similar to the results presented in 4.2. We present histograms for bzip2 for both instance types for our second sample below, and we also include the histogram of previous results to ease comparison.

Figure : Histograms for sample 1 (left hand column) and sample 2 (right hand column) bzip2 results on M1 and M2 respectively. We note consistency in terms of per CPU variation overall variation. However, we find differences in proportions of different CPU models encountered. For example, for M1, sample 1 has less E5430 and more E6545 than sample 2.

We find a good degree of consistency in performance between the two samples, and we report various percentiles below.

Table : Minimum, 25^th percentile, median, 75^th percentile, 95^th percentile and maximum values of bzip2 for samples 1 and 2 respectively.

Sample	Min(s)	25^th Perc(s)	Median(s)	75^th Perc(s)	95^th Perc(s)	Max(s)
1	425	460	502	534	642	716
2	417	456	475	521	684	746

The observed differences are accounted for by differences in proportions of instances on various CPU models. Indeed, as noted, the second sample contains CPU models not encountered in the first sample. In Table 9 below we present per CPU results for both samples.

Table : The minimum, median and maximum of bzip2 per CPU in both sample 1 and 2 respectively. We note per CPU consistency of results between sample 1 and 2, and we highlight results of the E5-2651 and E5645 for exemplification.

CPU	Sample	Min(s)	Median(s)	Max(s)
E5430	1	425	444	482
	2	417	447	487
E5-2650	1	443	469	519
	2	442	465	515
E5-2651	1	N/A	N/A	N/A
	2	441	482	543
E5645	1	488	507	544
	2	491	510	530
E5507	1	578	612	716
	2	584	641	746
AMD	1	N/A	N/A	N/A
	2	650	686	709

Whilst we have a large range of performance overall, once an instance has been obtained and its CPU model determined, its performance is relatively predictable. Indeed, we would expect it to fit with previously observed ranges for the CPU, which is typically much smaller than the overall range. As such, performance risk primarily exists at point of sale i.e. before an instance is obtained.

However, there is also the possibility of an instance whose performance is degraded beyond that of any previously observed. Further, and as happened in our second sample, there is also the possibility of obtaining an instance on a previously unseen CPU model, for which there is no past history and so we cannot estimate future performance. Looking ahead to section 6.2, we consider a commodity marketplace for instances where multiple providers sell instances. In this scenario we may expect the supply of various CPU models (for the same types of instances) to be quite variable as sellers come in and out of the market, and new data to be required as new CPU models emerge.

Interestingly, despite being the newest CPU model available at the time, the mean bzip2 performance of the E5-2651 is lower than both the E5-2650 and the E5430. It is often assumed that as Clouds introduce new hardware platforms they will outperform older generations, and such performance improvements will accrue to users without any increases in price. This is clearly not always the case and newer CPU models may not always provide better performance for certain workloads.

For STREAM, however, we find in the first sample that there are certain M2 instances on E5-2665 with increased performance compared to the second. One potential explanation for this is that the second sample ran on ‘busier’ hosts and so had increased contention for memory bandwidth. But we cannot rule out the possibility of an underlying hardware change which we were unable to detect. For the X5550 we again have a wide range, and so we can say that the degree of inconsistency is consistent. We present histograms below.

Figure : Histogram for STREAM on M2 in sample 1 (left hand column) and (right hand column) sample 2 respectively. As STREAM is measured in MB/s we recall that higher is better. We observe the unusual performance characteristics of instances on X5550 in both samples. Further, instances on E5-2665 had better performance than those in sample 2.
The results presented and discussed in this section confirm H2. We address H3, which concerns workload specific variation further, next.