A performance Brokerage for Heterogeneous Clouds

5.4 Workload Specific Variation

Our focus in this section is H3 i.e. the workload specific nature of performance variation. As noted, we have extended our benchmark set to include GNU GO, POV-Ray and NAMD, but restrict instance classes to m1.small and M2 family as compared to results discussed in section 5.2. We present the histograms followed by summary statistics of both raw results and normalized results.

Figure : Histogram for GNUGO, POV-Ray, NAMD and bzip2 on M1 (sample 2) respectively. For each workload we have a different ordering of CPUs from best to worst performing, demonstrating that performance is workload specific.

Figure : Histogram for GNUGO, POV-Ray, NAMD and bzip2 on M2 (sample 2) respectively. For GNUGO and POV-Ray X5550 is typically the best CPU model, whilst for NAMD and bzip2 E5-2665 typically perform best.

Table : Minimum, 25^th percentile, median, 75^th percentile, 95^th percentile and maximum values of bzip2, GNUGO, POV-Ray and NAMD for M1 and M2 respectively.

Instance Class	Benchmark	Min(s)	25^th Perc(s)	Median(s)	75^th Perc(s)	95^th Perc(s)	Max(s)
m1.small	bzip2	418	457	475	522	684	745
	GNUGO	180	196	200	206	212	230
	POV-Ray	474	543	554	576	581	593
	NAMD	1865	2010	2089	2171	2207	2430
M2	bzip2	163	165	168	179	185	227
	GNUGO	65	73	75	77	81	91
	POV-Ray	190	205	206	209	218	232
	NAMD	733	743	755	767	788	887

Table : Minimum, 25^th percentile, median, 75^th percentile, 95^th percentile and maximum values of bzip2, GNUGO, POV-Ray and NAMD for M1 and M2 respectively expressed as a degrade relative to the minimum. We highlight the minimum, median and maximum

Instance Class	Benchmark	Min(s)	25^th Perc(s)	Median(s)	75^th Perc(s)	95^th Perc(s)	Max(s)
m1.small	bzip2	1.0	1.09	1.14	1.25	1.64	1.78
	GNUGO	1.0	1.08	1.11	1.14	1.17	1.27
	POV-Ray	1.0	1.15	1.17	1.2	1.23	1.25
	NAMD	1.0	1.07	1.12	1.16	1.18	1.3
M2	bzip2	1.0	1.01	1.03	1.1	1.13	1.39
	GNUGO	1.0	1.12	1.15	1.18	1.25	1.4
	POV-Ray	1.0	1.08	1.09	1.1	1.15	1.22
	NAMD	1.0	1.01	1.03	1.05	1.08	1.2

From the results in Table 11 we can observe that, for the same instance type, different workloads have different degrees of variation. On the m1.small, for GNUGO, POV-Ray and NAMD, we find degradation from best to worst is in the range 1.25 to 1.3. Potentially, then, we may expect most workloads to be of this order, whilst perhaps considering bzip2, where we find a degradation of 78%, as an outlier. Indeed, we can plausibly argue that bzip2 has wider variation simply due to its poor performance on the E5507 and AMD CPUs. If we exclude these from our data, the bzip2 normalised range is now 1.29.

It is far from clear that bzip2 on an E5507 is an unusually bad combination. Indeed, if we consider GNUGO on M2, we find a degradation of 1.4, higher than the 1.27 found on m1.small despite the former having only 2 different CPU models and the latter 6. This is due to the performance of GNUGO on the E5-2665. The degree of variation for a workload on an instance type is dependent upon the workload/CPU combination, and we cannot necessarily use variation found on one instance type to predict variation on another.
It is clear that different CPU models have different performance ranges for the same workload, although we can find examples of significant overlap such as the AMD bzip2 range being entirely contained within the E5507 range. Further, as the histograms make clear, for various workload/CPU pairs we cannot say that a particular CPU model is best/worst for all workloads. From best to worst, we find the bzip2 ranking on m1.small to be E5430, E5-2650, E5-2651, E5645, E5507 and AMD. However, for POV-Ray we find AMD, E5645, E5507, E5-2651, E5-2650 and E5430. Note that the ranking is almost entirely reversed. The observation that different CPU models are better/worse depending upon the workload, is in accordance with observations made, for example, by Lenk et al. (2011) and Phillips et al. (2011), and in each of these cases different sets of benchmarks are used. As a consequence, it is not a feature of the particular benchmarks we have used, but rather a commonplace feature of heterogeneous Clouds.
To further understand workload/CPU performance we break out the results by CPU model, presenting both raw and normalised results, which are presented in tabled 12 and 13 in the appendix, section titled Additional Performance Results

From the above results we can observe that, for a typical workload/CPU combination, the width from median to maximum is wider than from minimum to median, and often significantly so. On our histograms this gives rise to high peaks close to the minimum i.e. clustering of instance performance close to best possible, together with a long tail. Notably, for a given workload, it is uncommon to find different CPUs with essentially identical performance, and so heterogeneity introduces permanent differences. The latter feature shows that even on homogeneous instances, which are apparently identical, large performance degradation (from the min) can occur. In the absence of being able to directly observe all activity on a host, our best explanation for such degradation is resource contention.

The results presented in this section confirm H3. Indeed, with regards to workload specific variation, we find not only does variation vary by workloads, but that different CPUs are better/worse for different workloads. Further, we are also able to provide a general characterisation of performance in terms of per CPU shape i.e. highly peaked close to best possible with a long tail, and multi-modal when heterogeneity is present.
We next consider how performance may vary by location due to variations in hardware by location.