Skip to main content

A/B tests in Cauldron

An A/B test, or experiment, shows different players different versions of one thing in your deck and compares how they play. You declare the experiment in castle.json, read each player's version (its variant) with the SDK, and Castle splits the deck's numbers by variant.

Declare it​

{
"experiments": {
"tutorial_length": { "variants": ["short", "long"] },
"shop_price": { "variants": ["low", "mid", "high"], "weights": [50, 25, 25] }
}
}
  • Each key is one experiment. A deck can run up to 8 at once.
  • variants lists its versions, 2 to 8 of them.
  • weights is the percentage of players each variant gets, as whole numbers adding up to 100. Leave it out for an equal split.
  • Experiment and variant names are 1 to 32 characters of a-z, 0-9 and _.

The experiments take effect when you publish. Publishing refuses an experiments block that breaks one of these rules, and says which experiment and which rule.

Castle decides which variant each player gets. A player keeps the same variant every time they play, and being in one variant of an experiment says nothing about their variant in another. Neither the deck nor the player can choose.

Read it in the deck​

import { experiment, experimentsReady } from "castle-web-sdk";

await experimentsReady();

if (experiment("tutorial_length") === "long") {
startLongTutorial();
} else {
startShortTutorial();
}

Await experimentsReady() once when the deck starts, before the first read. After that, experiment(name) returns the player's variant right away.

Always handle null. experiment(name) returns null when the player isn't in the experiment: the deck doesn't declare it, it was removed, the player is outside its audience, or the player is on an older version of the Castle app. Give null the behavior the deck had before the test, as the else above does.

Read each experiment where it changes what the player sees, not all at once at the start. The first read records that the player was shown their variant, and the results page counts those players separately (see Results). experiments() returns all of the player's variants at once and records nothing.

A multiplayer deck's server can read the same variants for each player with session.experiment. Use the server's answer for anything other players depend on, like prices or lives. Players in one session can be in different variants, so a test of something they all share, like the map, needs a rule for whose variant applies.

See Experiments in the SDK reference for the details of each call.

Choose who is in it​

An audience limits an experiment to some players. A player has to match every key you give:

"tutorial_length": {
"variants": ["short", "long"],
"audience": { "countries": ["US", "CA"], "platforms": ["web"], "players": "new" }
}
  • countries: two-letter country codes in capitals, for where the player is connecting from.
  • platforms: any of web, ios and android.
  • players: new (hasn't played this deck before) or returning.

Castle decides once per player whether they're in, and keeps that decision: a new player stays in after their first play, and a player who's in on the web is in on their phone too. A player outside the audience reads null. Changing the audience decides every player again on their next visit.

Pick a winner​

When one variant has won, name it:

"shop_price": { "variants": ["low", "mid", "high"], "weights": [50, 25, 25], "winner": "mid" }

After you publish, every player in the experiment's audience gets mid, so you can ship the winner right away and clean up the code later. A player outside the audience still reads null, so remove the audience too if the winner should reach everyone. Removing an experiment from castle.json ends it, and every read of it returns null.

Try each variant yourself​

On your computer: castle-web serve gives you the variant your account will get once the deck is published, and puts you in every audience. To see another one, add ?experiment.<name>=<variant> to the workspace's URL, for example http://localhost:5757/?experiment.tutorial_length=long. It applies to the whole workspace, including the local multiplayer server, until you clear it with an empty value (?experiment.tutorial_length=) or stop the workspace. Reload an open deck to see the change.

On the published deck: open the deck's page on castle.xyz, signed in as the deck's owner, with the same parameter: https://castle.xyz/d/<deckId>?experiment.tutorial_length=long. The variant holds for a day, for your account on that deck, and an empty value ends it sooner. It does nothing for anyone but the owner. Your own plays are never counted in the results, so trying variants doesn't change them.

Results​

When a deck you made has experiments, its page on castle.xyz shows you an Experiments button that only you can see. It opens the results, one card per variant, over the last 7, 30 or 90 days:

  • Exposed players: players the deck read this variant for.
  • Players and Plays: players who finished a play in this variant, and how many plays.
  • Play time.
  • Day 1 return and Day 7 return: the share of players who played again, in the same variant, that many days after their first play. Each comes with the range its true value likely falls in, and the page names a leader only when every variant has enough players and the difference is unlikely to be chance.

Players who aren't in the experiment get a card of their own.

Pick how long to run the test before you look. Stopping the first day a variant looks ahead makes false wins more likely.

Your own events​

To measure something specific to your game, like how many players finish level 3, record your own events with Analytics.event:

import { Analytics } from "castle-web-sdk";

Analytics.event("level_complete", { level: 3, seconds: 41 });

Each event carries the player's variants, so it can be compared by variant. An event name has to be registered before players' events are kept: after publishing, play the deck yourself, signed in as its owner, through every path that sends an event. The Experiments page doesn't show your own events yet.

Before you read the results​

  • Changing variants or weights moves some players. Going from 50/50 to 60/40 moves 10% of players from the second variant to the first. Those players have seen both versions, so numbers from before and after the change don't mix cleanly. Decide the split before the test starts.
  • Renaming an experiment starts a new one, and every player is assigned again.
  • A guest who logs in can switch variants once. A player who isn't signed in plays as a guest. If they then log in to an account they already had, they become that account, which may be in a different variant.
  • Players count in their variant for the whole session, whether or not the deck showed it to them yet. A player who quits before reaching the shop still counts in a shop_price variant, so a test of something late in the game moves the numbers less than it moves the players who got there.
  • A player who reads null on an older app is still counted in the variant Castle assigned them. That pulls the variants' numbers toward each other; it doesn't favor one.

Next: SDK reference · Back to Multiplayer