Did you find this helpful?

A/B Testing with Stores and Test Buckets

In our Stores guide, we demonstrate how to set up a store and make a few items available to a player at special or alternate prices. In our A/B Tests guide, we demonstrate how to create A/B tests and form player buckets for random, percentage-based user distribution.

This guide lets you combine these features, and produce several versions of the store available to different A/B testing groups (buckets).

We first define an A/B test. In this case we define a test called "Store A/B Testing" which splits users into 3 groups. First group (Control) contains 34% of all the players. The other 2 test groups each contain 33% of all players:

Next we create one store to be the original version, and then we create 3 other stores, which are A/B tested variations of the original. Each store is created independently, has it's own unique identifier and should have content that is customized for that segment:

Return to the original store page and locate "Segment Overrides" section. It will not only contain usual player segments, but it will also expose all the buckets for all A/B tests you currently have. Make sure your A/B test buckets are on top of the list and assign store override settings for each bucket, as shown on the picture below:

Do not forget to save the settings. In short: you have A/B test where players are distributed into 3 groups of 50%, 30% and 20%. For each group of players you assign different version of the store. You may then use A/B testing reports to track user conversion for each version of the store.

Best Practices for Store Segmentation

  • Store segmentation is public information
    • There are many ways for players to gain information about alternate stores
    • Players naturally transition between segments, and they'll see those changes when they transition
    • Players discuss content on forums, community sites, wikis, etc
    • In the PlayFab API, store information is public unless you disable those APIs with our API Access Policy
    • Thus, you should assume that players will be aware of other stores and their details
  • Provide varying content, not varying pricing:
    • Players will feel cheated if:
      • If prices go up or down when they transition segments
      • If prices described on wikis are different than their own
    • The only exception to the pricing rule would be first time purchases
      • You must secure the client API methods with our API Access Policy to prevent multiple purchases, or make repurchasing less meaningful through game design
  • Content should be relevant to the segment
    • Whales are willing to spend large amounts of real money, and thus are likely to buy more expensive bundles, if the value is worthwhile
    • First time buyers may be given an option which is a particularly good deal, but can only be [usefully] obtained once
    • End game players may only be interested in a specific subset of items

Did you find this helpful?