软件性能测试中的变点检测（CS SE）---Elva

我们描述了在存在噪声的情况下自动检测软件产品性能变化的过程。随着对软件产品的更改提交到源存储库，大量测试将会定期运行，我们希望能够识别出那些对性能回归有影响的更改。以前，我们依靠人工检查时间序列图来识别重大变化，后来，这种方法被一个基于阈值的检测系统所取代，但这两个系统都无法及时发现性能的变化。本文描述了我们最近实现的一个基于E-Divisive算法的变点检测系统。该算法会生成一个变点列表来表示性能结果历史中的重大变化。只需要一个人来检查变点列表，以便进行可操作的变更，然后对这些变更进行筛选以供进一步检查。使用变点检测对我们检测性能变化的能力产生了巨大的影响，在数量上，它显著降低了我们错误检测出一个未发生的性能变化的概率，而在质量上，它使整个性能评估过程更容易、更有成效（例如，获得较小的回归）和更及时。

原文题目：Change Point Detection in Software Performance Testing

原文：We describe our process for automatic detection of performance changes for a software product in the presence of noise. A large collection of tests run periodically as changes to our software product are committed to our source repository, and we would like to identify the commits responsible for performance regressions. Previously, we relied on manual inspection of time series graphs to identify significant changes. That was later replaced with a threshold-based detection system, but neither system was sufficient for finding changes in performance in a timely manner. This work describes our recent implementation of a change point detection system built upon the E-Divisive means algorithm. The algorithm produces a list of change points representing significant changes from a given history of performance results. A human reviews the list of change points for actionable changes, which are then triaged for further inspection. Using change point detection has had a dramatic impact on our ability to detect performance changes. Quantitatively, it has dramatically dropped our false positive rate for performance changes, while qualitatively it has made the entire performance evaluation process easier, more productive (ex. catching smaller regressions), and more timely.

原文作者：David Daly, William Brown, Henrik Ingo, Jim O'Leary, David Bradford

原文链接：https://arxiv.org/abs/2003.00584