A performance Brokerage for Heterogeneous Clouds

5.7 Longitudinal Variation

In sections 5.2 – 5.5, the results presented are from a cross-section giving us a snapshot in time of variation in performance across instances. However, given a particular instance, what does this variation imply for its performance over a given (future) time period? For example, if we know from previous cross-section results that performance of instances on a given CPU model may vary by up to 25%, does this imply that the performance of a particular instance on the CPU will vary by 25% over a future time period? We use the term intra variation to mean the variation of a particular instance over a period of time whilst inter variation is the variation found in a cross-section. So the question of interest is the degree of magnitude of intra variation as compared to inter variation. A further question arises: can different instances running on supposedly identical hardware have different mean performance over some period of time? We suspect this is the case, and indeed we formulated this as the following hypothesis:

H5: Instances running on supposedly identical hardware, as identified by having the same CPU model, do not necessarily have the same performance levels over a given period of time.
We investigate the longitudinal performance of 50 m3.medium spot instances over a 40 hour period, of which 2 were reclaimed before we could collect any results. For every instance we took a bzip2 measurement every 12 minutes giving a total of 200 measurements per instance. When computing the Augmented Dickey-Fuller (ADF) test statistic for each instance, we reject the null hypothesis of the existence of a unit root for 38 instances at the 5% significance level. We recall that the existence of unit root implies the time series is non-stationary, and so has either time varying mean or variance (Chatfield, 2003). Informally, we can say that the performance of most instances is mainly consistent (constant mean and variance) over the 40 hours. In the table below we present a number of time-plots of stationary performance.

Figure : Example time-plots of instances with stationary performance.
We can observe that individual stationary instances have small, arguably negligible, variation. We also note that small differences in performance persisted across the time period, for example, in bottom left hand figure we have an instance with mean performance of 285s and a minimum of 283s, whilst the instance in the bottom right hand corner of the figure has a mean of 280s and a maximum of 282s. When we consider the performance measurements of all instances combined we find a mean of 285s and a standard deviation of 7s. For each stationary instance we find that intra deviation is less than the inter deviation. However, even amongst stationary instances we find some notable performance features. In particular, a group of 9 instances all experienced a large deviation in performance at approximately the same time. In table 24 below we present time-plots for 4 of this group.

Figure : Example time-plots of instances with stationary performance, but all exhibit a large deviation at approximately the same time.
The most likely cause of this is some external event to which all these instances are responding. Potentially they could all be on the same host and so may have been affected by the same physical degradation in the underlying server, or the arrival of one or more particularly resource consuming co-locating instances. Alternatively, they may have some other shared component in common such as network storage. It does however reveal the possibility that the performance of all instances may be impacted by the same stimulus i.e. performance may be correlated for some instances.
Not all instances are stationary, and in Figure 9 below we present a number of time-plots for non-stationary series. Instances in row 1 both have a mean that varies over time and are slightly noisier than the stationary instances presented above. We also note that the instance in the top left hand corner is consistently above 303s, whereas the instance in row 1, column 2 has a maximum of 291s. This further demonstrates how differences may persist over time.
In the middle row we observe instances that have experienced a jump in their performance. Following this jump they stay at this new level for a period before returning to their previous performance. We presently attribute these jumps to co-locating instances starting and terminating on the host. In the middle right we have an example of a locally-stationary instance, where the instance was stationary, then experienced a jump to a new stationary performance level, before returning to its previous level, for which it is again stationary. In the bottom row we also have locally-stationary instance, albeit each with one small jump to a new level, which in both cases is a slight increase in mean performance.


Figure : Example time-plots of instances with non-stationary performance. The observed jumps in performance are likely due to the arrival and leaving of new instances.
Finally, we find one example, in our set of 48, of somewhat pathological behaviour, and we present a time-plot of this below. Looking ahead to section 5.9, we demonstrate that memory bandwidth starvation due to the resource consuming actions of co-locating instances can cause large deviations in performance. A possible explanation for the observed performance is that the instance is co-locating with one or more instances with high memory bandwidth needs.

Figure : An instance with erratic performance, where future performance is not predictable. The broker could not supply such instances to its clients.

The results in this section confirm H5, and our main finding is that instances are typically stationary, and if not then they are locally stationary. This is result is important for the broker as it means instances can be offered with assurances over their future performance.