The Best Revenue Significance Calculator for A/B Testing
用于 A/B 测试的最佳收入显着性计算器

May 11, 2017Roopa Carpenter
May 11， 2017 鲁帕·卡彭特

If you’re conducting A/B tests on your ecommerce website and are not tracking revenue, then you are missing out on a crucial component for successful testing: having the right KPI.
如果您在电子商务网站上进行 A/B 测试并且没有跟踪收入，那么您就错过了成功测试的关键组成部分：拥有正确的 KPI。

Tracking revenue allows your team to make effective business decisions, because you’re measuring performance in a way that actually impacts the bottom line.
跟踪收入可以让您的团队做出有效的业务决策，因为您正在以实际影响底线的方式衡量绩效。

So which revenue metrics should you choose?
那么您应该选择哪些收入指标呢？

Some common revenue metrics don’t tell the whole story, which is why we recommend using revenue per visitor (RPV). RPV measures the amount of revenue generated each time a user visits your site:
一些常见的收入量度并不能说明全部情况，因此我们建议使用每位访客收入 （RPV）。RPV 衡量用户每次访问您的网站时产生的收入金额：

RPV = Total Revenue RPV = 总收入
Total Users 用户总数

We’re about to explain: 我们即将解释：

• why revenue per visitor is such a crucial (composite) metric
  为什么每位访客的收入是一个如此关键的（复合）指标
• the need to rewrite the RPV’s formula to include transaction rate and AOV
  需要重写 RPV 的公式以包括交易速率和 AOV
• the right way to measure its statistical significance
  衡量其统计显著性的正确方法
• how to use our free online revenue significance calculator
  如何使用我们的免费在线收入重要性计算器
• how to hack your way around sampled data to get the most accurate results
  如何破解采样数据以获得最准确的结果

Why Use Revenue Per Visitor in A/B Testing?
为什么在 A/B 测试中使用每个访客的收入？

If your team tracks only transaction rate (the percentage of visitors that purchased) or average order value (AOV) as your primary metric for testing, your results are at risk of having blind spots.
如果您的团队仅跟踪交易率（购买访客的百分比）或平均订单价值 （AOV） 作为测试的主要指标，则您的结果存在盲点风险。

Some people assume that AOV is relatively constant and they only need to focus their efforts on increasing transaction rate in order to increase revenue. However, this logic doesn’t always apply.
有些人认为 AOV 是相对恒定的，他们只需要将精力集中在提高交易率上，就可以增加收入。但是，此逻辑并不总是适用。

In some circumstances, increasing conversion rate can negatively affect your overall revenue.
在某些情况下，提高转化率可能会对您的整体收入产生负面影响。

For example, if you have a test variation that increases the conversion rate, but users choose to purchase the lower-priced product instead of the more expensive product, this can decrease AOV and overall revenue.
例如，如果您有一个提高转化率的测试变体，但用户选择购买价格较低的产品而不是更昂贵的产品，这可能会降低 AOV 和整体收入。

Alternatively, people may focus their efforts on improving only AOV to increase revenue which can lead to a decrease in transaction rate, ultimately hurting revenue.
或者，人们可能只将精力集中在改进 AOV 以增加收入上，这可能导致交易率下降，最终损害收入。

For example, consider an ecommerce website test where the variation increases the spend threshold to qualify for free shipping. This can lead to a higher AOV, but can also decrease transaction rate because there may be visitors who want free shipping but don’t want to spend the extra money to qualify. As a result, they may choose not to purchase.
例如，考虑一个电子商务网站测试，其中变体提高了符合免费配送资格的支出阈值。这可能会导致更高的 AOV，但也可以降低交易率，因为可能有访客想要免费送货但不想花额外的钱来获得资格。因此，他们可能会选择不购买。

The examples above illustrate the need to have a solid conversion strategy for revenue that incorporates both metrics.
上面的示例说明了需要制定包含这两个指标的可靠收入转化策略。Revenue per visitor is that composite metric, which accounts for both transaction rate and AOV Click & Tweet!
每个访客的收入是那个复合指标，它同时考虑了交易率和AOVClick & Tweet！.

In fact, we can rewrite the RPV’s formula to include these two elements:
事实上，我们可以重写 RPV 的公式以包括以下两个要素：

Total Revenue = AOV x Transactions
总收入 = AOV x 交易
Transaction Rate = Transactions/Total Users
事务率 = 事务数 / 总用户数
RPV= AOV x Transaction Rate
RPV = AOV x 交易率

So if your business had 1,000 transactions for every 15,000 users with an AOV of $50, the RPV would be:
因此，如果您的企业每 15,000 个用户有 1,000 笔交易，AOV 为 50 美元，则 RPV 将为：

Total Revenue = $50 * 1,000 = $50,000
总收入 = 50 USD * 1000 = 50000 USD
Transaction Rate = 1000/15,000 = 0.067
交易率 = 1000/15000 = 0.067
RPV = $50 * 0.067 = $3.35
RPV = 50 美元 * 0.067 = 3.35 美元

Monitoring trends in RPV can help your team analyze sales performance. It’s useful for evaluating your new visitor acquisition and paid user acquisition efforts.
监控 RPV 的趋势可以帮助您的团队分析销售业绩。它可用于评估您的新访客获取和付费用户获取工作。

Generally, a positive trend in RPV shows that your company’s sales efforts are working well.
一般来说，RPV 的积极趋势表明贵公司的销售工作运作良好。

However, if your revenue per visitor is trending downward, this could be the result of an increase in unqualified users to the site or potential site problems (e.g. broken shopping cart), which negatively affects your transaction rate.
但是，如果您的每位访客收入呈下降趋势，这可能是由于网站不合格用户增加或潜在的网站问题（例如购物车损坏）造成的，这会对您的交易率产生负面影响。

Or your visitors may be converting at the same rate but are spending money on lower value items (e.g. higher priced product is out of stock), which negatively impacts your AOV.
或者您的访问者可能以相同的转化率进行转化，但将钱花在价值较低的商品上（例如，价格较高的产品缺货），这会对您的 AOV 产生负面影响。

Taking the example above, let’s say the number of users increased to 20,000 due to a social campaign that recently launched. Assuming the AOV stayed the same, your team would find that RPV is trending negatively:
以上面的示例为例，假设由于最近启动的社交活动，用户数量增加到 20,000 人。假设 AOV 保持不变，您的团队会发现 RPV 呈负趋势：

Transaction Rate = 1,000/20,000 = 0.05
交易率 = 1,000/20,000 = 0.05
RPV = $50 * 0.05 = $2.50
RPV = 50 美元 * 0.05 = 2.50 美元

Now let’s assume that the traffic stayed the same but your most expensive product was out of stock, causing the AOV to decrease to $37.30:
现在，我们假设流量保持不变，但您最昂贵的产品缺货，导致 AOV 降至 37.30 美元：

Transaction Rate = 1,000/15,000 = 0.067
交易速率 = 1000/15000 = 0.067
RPV = $37.30 * 0.067 = $2.50
RPV = 37.30 美元 * 0.067 = 2.50 美元

RPV does not replace the need to keep an eye other metrics like AOV and transaction rate. It removes potential blind spots that can occur if you choose to track only those metrics. In essence, it gives your team a better sense of the bigger picture.
RPV 并不能取代密切关注 AOV 和交易率等其他指标的需要。它消除了您选择仅跟踪这些指标时可能出现的潜在盲点。从本质上讲，它让您的团队更好地了解大局。

How NOT to Calculate Statistical Significance
如何不计算统计显著性

If your team is already using revenue per visitor as the main KPI for your tests, you may have figured out why you shouldn’t use the standard online revenue significance calculators to determine whether your test variation is having an actual impact on RPV. These standard “tools” perform calculations using a T-test, which operates on one critical assumption: that the metric you’re tracking follows a normal distribution.
如果您的团队已经将每位访客的收入用作测试的主要 KPI，您可能已经弄清楚了为什么不应该使用标准的在线收入显著性计算器来确定您的测试变体是否对 RPV 产生实际影响。这些标准“工具”使用 T 检验执行计算，该检验基于一个关键假设运行：您正在跟踪的指标服从正态分布。

_{Source: Statistics Cheat Sheet
资料来源：统计作弊表}

Revenue per visitor doesn’t follow a normal distribution and therefore violates this assumption, because the majority of visitors to your site will not convert or make a purchase. As a result, you’ll discover that RPV’s distribution contains a greater concentration of $0 values and there is no limit on how much a visitor can spend, which may result in your RPV data containing some extreme values.
每位访客的收入不遵循正态分布，因此违反了这一假设，因为您网站的大多数访客不会转化或进行购买。因此，您会发现 RPV 的分布包含更集中的 0 美元值，并且访客可以消费的金额没有限制，这可能会导致您的 RPV 数据包含一些极端值。

For these reasons, RPV’s distribution tends to be right-skewed, making the standard T-test less reliable for measuring statistical significance.
由于这些原因，RPV 的分布往往是右偏态的，这使得标准 T 检验在测量统计显著性方面不太可靠。

The Right RPV Confidence Calculator for the Job
适合这项工作的 RPV 置信度计算器

To solve this problem, we launched a free online Revenue Per Visitor confidence calculator designed specifically for calculating RPV’s statistical significance. Our RPV calculator utilizes the Wilcoxon Rank Sum Test, which is not based on the assumption that the data follows a normal distribution.
为了解决这个问题，我们推出了一个免费的在线 Revenue Per Visitor 置信度计算器，专为计算 RPV 的统计显著性而设计。我们的 RPV 计算器使用 Wilcoxon 秩和检验，该检验不是基于数据服从正态分布的假设。

In fact, the Wilcoxon Rank Sum Test employs a non-parametric technique — a technique that does not rely on any specific distributional assumption — in order to test whether there is a difference.
事实上，Wilcoxon 秩和检验采用了一种非参数技术（一种不依赖于任何特定分布假设的技术）来检验是否存在差异。

This calculation is far more reliable in determining whether there is an actual impact on RPV. It includes a two-tailed calculation, so you can use it to determine whether the variation had a positive impact or a negative impact when compared to the control.
在确定是否对 RPV 有实际影响时，这种计算要可靠得多。它包括一个双尾计算，因此您可以使用它来确定与对照相比，变异是具有积极影响还是负面影响。

How to Use the RPV Calculator
如何使用 RPV 计算器

If you take a sneak peek at our testing confidence calculator, you’ll notice it looks different from the standard statistical significance calculators.
如果您先睹为快我们的测试置信度计算器，您会注意到它看起来与标准的统计显着性计算器不同。

Standard Online Calculators
标准在线计算器

Blast’s Revenue Per Visitor (RPV) Calculator
Blast 的每次访客收入 （RPV） 计算器

As mentioned above, you cannot simply enter total visitors and total revenue per variation to determine statistical significance.
如上所述，您不能简单地输入访客总数和每个变体的总收入来确定统计意义。

To accurately measure whether there is an impact on RPV, you need to have user-level data.
要准确衡量是否对 RPV 有影响，您需要拥有用户级数据。

Most businesses choose to integrate their A/B tests with their preferred analytics platform and analyze test performance there. This allows teams to make an apples-to-apples comparison when looking at performance across different channels, such as testing and marketing efforts.
大多数企业选择将他们的 A/B 测试与他们首选的分析平台集成，并在其中分析测试性能。这使团队在查看不同渠道（例如测试和营销工作）的绩效时可以进行同类比较。

The problem is that while you can see overall revenue for test variations within analytics, it is much more difficult to get access to user-level data.
问题在于，虽然您可以在 Analytics 中看到测试变体的总体收入，但访问用户级数据要困难得多。

Unsampled Google Analytics Data Hack
非抽样 Google Analytics 数据黑客

The Blast team has a solution for obtaining user-level data so you can make use of the revenue significance calculator.
Blast 团队有一个用于获取用户级数据的解决方案，因此您可以使用收入重要性计算器。

It may take a little leg work in the beginning, but your team will reap the benefits for the long term. To get user-level data within Google Analytics, follow the steps below and you’ll be on your way to A/B testing success.
一开始可能需要一些跑腿工作，但从长远来看，您的团队将获得好处。要在 Google Analytics 中获取用户级数据，请按照以下步骤操作，您将踏上 A/B 测试成功的路。

1. Create a Custom Dimension for Client ID
1. 为客户 ID 创建自定义维度

Google Analytics (GA) has recently started offering a new User Explorer report. The best part of this report is that it has a Client ID dimension that tracks user-level behavior, which is specific to browser and device.
Google Analytics （GA） 最近开始提供新的用户分层图表报告。此报表的最佳部分是它具有一个 Client ID 维度，用于跟踪特定于浏览器和设备的用户级行为。

Now the downside! 现在是缺点！

In its current state, you can’t access this dimension outside of this report, so your team can’t pull this data into a custom report.
在当前状态下，您无法在此报表之外访问此维度，因此您的团队无法将此数据提取到自定义报表中。

To get around this problem, your team will need to create a custom dimension for the Client ID. This step should take roughly 1-2 hours for your analytics team to create, QA, and implement. Once this is implemented you’ll be able to use the Client ID dimension for your test reports as well other Google Analytics reports.
要解决此问题，您的团队需要为客户 ID 创建自定义维度。您的分析团队大约需要 1-2 小时来创建、QA 和实施此步骤。实施此操作后，您将能够将“客户端 ID”维度用于测试报告以及其他 Google Analytics（分析）报告。

You may think this step isn’t worth the effort and that you can just export the data from the User Explorer report, but that will only work if you have minimal traffic to the site. The User Explorer report caps the data to 10,001 rows.
您可能认为此步骤不值得付出努力，您可以只从 User Explorer 报告中导出数据，但这仅在您网站的流量最少时才有效。User Explorer 报表将数据限制为 10,001 行。

If your site receives more than 10,000 visitors within the time frame you select, then you won’t be able to see all user-level data and instead will get a sampling of the data. By creating the Client ID custom dimension, you can create a custom report for your test, containing the Client ID, where you’ll be able to capture all the rows of data.
如果您的网站在您选择的时间范围内接收的访客超过 10,000 人，那么您将无法看到所有用户级别的数据，而是获得数据样本。通过创建“客户端 ID”自定义维度，您可以为测试创建包含“客户端 ID”的自定义报告，以便捕获所有数据行。

User Explorer Report: Limits Client ID and accompanying revenue data to 10,001 rows.
用户分层图表报告：将 Client ID 和随附的收入数据限制为 10001 行。

Custom Report: Provides Client ID (via a custom dimension) and accompanying revenue data greater than 10,0001 rows.
自定义报告： 提供 Client ID（通过自定义维度）和超过 10,0001 行的随附收入数据。

2. Utilize unSampler to Export All Data
2. 使用 unSampler 导出所有数据

As your team uses the Client ID custom dimension within other Google Analytics reports, there is another challenge that lies ahead. Google Analytics caps the number of rows you can export at one time to 5,000 rows.
由于您的团队在其他 Google Analytics（分析）报告中使用“客户 ID”自定义维度，因此还面临着另一个挑战。Google Analytics 将您一次可以导出的行数限制为 5,000 行。

If you really have the time, you can attempt to export your report data 5,000 rows at a time, but for most people this is completely inefficient. Previous hacks like altering the number in the url to show more rows no longer work.
如果您真的有时间，可以尝试一次导出 5,000 行报告数据，但对于大多数人来说，这是完全无效的。以前的技巧（例如更改 url 中的数字以显示更多行）不再有效。

If your business has Google Analytics 360, then your team has the feature to export all data by utilizing the Unsampled report.
如果您的企业拥有 Google Analytics 360，那么您的团队可以使用非抽样报告导出所有数据。

Resolve the sampling issues from the standard version of Google Analytics is as simple as creating an unSampler account and linking your Google Analytics account to it. Doing so will enable your team to easily create a test report (where you will have access to your custom dimensions) and export all of your data to CSV.
解决标准版 Google Analytics 的采样问题就像创建一个 unSampler 帐户并将您的 Google Analytics 帐户链接到它一样简单。这样，您的团队就可以轻松创建测试报告（您可以在其中访问自定义维度）并将所有数据导出为 CSV。

3. Format & Upload CSV
3. 格式化和上传CSV

Once you’ve exported data from your unSampler Report, you’ll need to take a few quick steps to format it so it will be ready to use with the revenue significance calculator. First, you’ll need to filter your data for the control:

Then copy the revenue data and paste it in a new tab (optional: you can rename the header to Control Revenue).
然后复制收入数据并将其粘贴到新选项卡中（可选：您可以将标题重命名为 Control Revenue）。

Repeat this step with your test variation. After doing so, in the new tab you should have two columns for revenue (Control Revenue and Variation Revenue). Please note, if you have more than one test variation, you’ll need to create separate tabs for each one (e.g. Control vs Variation 1, Control vs Variation 2, Control vs Variation 3).
对测试变体重复此步骤。执行此操作后，在新选项卡中，您应该有两列收入（Control Revenue 和 Variation Revenue）。请注意，如果您有多个测试变体，则需要为每个变体创建单独的选项卡（例如，Control vs Variation 1、Control vs Variation 2、Control vs Variation 3）。

Save this new tab as a CSV file (or multiple CSV files if you have more than one test variation) and then it’s ready for the RPV Calculator.
将此新选项卡另存为 CSV 文件（如果您有多个测试变体，则另存为多个 CSV 文件），然后它就可以用于 RPV 计算器了。

Before uploading your file to the calculator, you can adjust the threshold for determining statistical significance — the default is set at 95%. The last step is simply uploading your file.
在将文件上传到计算器之前，您可以调整用于确定统计显著性的阈值 — 默认值设置为 95%。最后一步是简单地上传您的文件。

The results you get are fast, reliable and easy to understand.
您得到的结果是快速、可靠且易于理解的。

A/B Test Results You Can Trust
您可以信赖的 A/B 测试结果

While it takes a little bit of effort in the beginning to properly measure revenue per visitor, once it’s set you can easily analyze this KPI for future tests. Further, by using the free online revenue significance calculator, you can trust that the correct method of analysis was applied.
虽然一开始需要付出一些努力才能正确衡量每个访客的收入，但一旦设置好，您就可以轻松分析此 KPI 以供将来测试。此外，通过使用免费的在线收入显着性计算器，您可以相信应用了正确的分析方法。

Your team can rely on test performance results to make those important business decisions.
您的团队可以依靠测试结果来做出这些重要的业务决策。

Please share your comments or let us know if you have questions regarding this process or the calculator.
请分享您的评论，或者如果您对此过程或计算器有任何疑问，请告诉我们。

About the Author 关于作者

Roopa Carpenter 鲁帕·卡彭特

Roopa Carpenter is Vice President of Digital Experience (DX) at Blast. She leads a team of talented, DX consultants responsible for helping brands to better understand, optimize and measurably impact their digital experience. With many years of experience, Roopa offers a high level of knowledge and guidance to clients across a variety of industries regarding testing and personalization strategy and execution, user experience research and closing the empathy gap through Voice of Customer. Her data-driven approach focuses on impacting customer conversion and driving desired business outcomes.
Roopa Carpenter 是 Blast 的数字体验 （DX） 副总裁。她领导着一支由才华横溢的 DX 顾问组成的团队，负责帮助品牌更好地了解、优化和衡量其数字体验。凭借多年的经验，Roopa 为各行各业的客户提供高水平的知识和指导，包括测试和个性化策略和执行、用户体验研究以及通过客户之声缩小同理心差距。她的数据驱动方法专注于影响客户转化率并推动预期的业务成果。

Connect with Roopa on LinkedIn. Roopa Carpenter has written 19 posts on the Blast Digital Customer Experience and Analytics Blog.
在 LinkedIn 上与 Roopa 联系。Roopa Carpenter 在 Blast Digital 客户体验和分析博客上撰写了 19 篇博文。