Flags and experiments

These commands create and roll out feature flags, fix them for one person, and run A/B tests from draft to a concluded winner. Studio's Feature flags and A/B tests pages do the same.

Feature flags

Production and test

A flag decides separately in production and in test, which apps using a test key read. Test follows production until you change something in it: --env test on enable, disable, rollout, override or rule gives test a configuration of its own, started from production's. Without --env, a command changes production.

voidhash flag enable new-onboarding --rollout 100% --env test   # on for test builds only
voidhash flag override new-onboarding ada@example.com on --env test
voidhash flag promote new-onboarding                            # the same in production

flag promote copies test's switch, overrides, rules and share to production, and test follows production again. flag follow drops test's configuration without changing production.

Rules

An environment is read top to bottom: its switch, then overrides, then rules in order, then the share everyone else gets. A rule says what share of the people it matches get the flag, and for a flag with variants which one. It matches the people it names one by one:

voidhash flag rule add new-checkout ada@example.com user_8231 --rollout 100% --name "Beta testers"
voidhash flag rule add new-checkout user_4411 --rollout 0% --first   # never gets it
voidhash flag rollout new-checkout 0%                                # only the named people do

and, with --when, everyone a condition holds for. Repeat --when and all of them must hold:

# Half of the people in Germany and Austria whose app is 2.4 or newer
voidhash flag rule add new-checkout --when 'country in DE,AT' --when 'app_version >= 2.4' --rollout 50%
# Everyone on the pro plan who opted into the beta, except on the web
voidhash flag rule add new-checkout --when 'trait:plan in pro' --when 'trait:beta is set' \
  --when 'platform not in web' --rollout 100%

A condition is <what> <how> [<values>], with values separated by commas:

A rule can name people and have conditions; it then matches either way. A new rule is read after the ones before it; --first puts it first. flag view lists them in order, in words.

flag evaluate knows a person's traits but not their app, so a condition on the app counts as not set unless you say what the app would report:

voidhash flag evaluate ada@example.com --platform ios --app-version 2.4.1 --country DE

Experiments

Always pass --metric, the goal event. Without it, results count purchase_completed, an event Voidhash never sends, so every arm shows 0 conversions. See Conversions for which built-in event to use.

--arm takes NAME=PAYWALL with --location, or NAME=LOCATION:PAYWALL[,LOCATION:PAYWALL]. Repeat it for each arm; the first is the control unless --control names another, and --weights 60,40 splits traffic unevenly.

--flag gives each arm a value for one of your feature flags, as KEY=VALUE,VALUE with one value per --arm, in order. A value is on, off, or one of the flag's variant keys, which turns it on with that variant. Repeat it for each flag. An arm needs no paywall then, so --arm NAME alone tests a flag by itself:

voidhash experiment create --name "New checkout" --metric '$subscription.created' \
  --arm Control --arm "New checkout" --flag new-checkout=off,on

While the experiment runs, each flag takes the value of the person's arm, after the flag's switch and overrides and before its rules. flag evaluate then says a running experiment set it. It only looks: it doesn't put the person in an experiment.

Example

voidhash flag create --key new-onboarding --rollout 10%
voidhash experiment create --name "Annual first" --metric '$subscription.created' \
  --location onboarding --arm Control=pro --arm Annual=annual-first

Next steps

• How a flag decides: environments, overrides, rules and rollouts.
• A/B test results: conversions and verdicts.
• Analytics and ads: measure the results.