Manage Your Experiments lets you run A/B tests (also known as split tests) on your brand’s listing content. Experiments help you compare two versions of content against each other so you can see which performs better. At the end of an experiment, you can review which version of content performed the best and then publish the winning content. By running experiments, you can learn how to build better content that appeals to your customers and helps to drive more sales.
What are good experiments?
The best experiments share these characteristics:
Version A and Version B are very different from each other. For example, with A+ content, use different modules or different orders of modules. For product titles, try significantly shortening the title length to reduce noise and encourage more customers to visit your detail page. For image experiments, try alternatives that make your product easier to understand and more information-rich.
Important: The more that your content and treatments are different, the more likely it will be that any performance differences detected are meaningful and not caused by random chance.
The duration of the experiment is 10 weeks. This allows us time to collect as much data as possible.
Set up an experiment
Step 1: Start an Experiment: From the Manage Your Experiments main screen, click on the Create a New Experiment drop-down. Then select your desired experiment type.
Step 2a: Select an eligible ASIN to experiment on: You will be prompted to select a Reference ASIN. A “Reference ASIN” is the primary product that will be included in the experiment.
Step 2b: For A+ Content experiments, we will first redirect you to the “Create an A/B test page”. From there, you will be able to select a Reference ASIN. A “Reference ASIN” is the primary product that will be included in the experiment.
You may choose a variational ASIN for experimentation. If you choose a variational ASIN, the system will detail which child ASINs can be included in the experiment. The system will display ASIN eligibility status, along with details about why some ASINs may be ineligible.
Note: ASINs with low traffic may not show up in the available ASINs list and are not eligible for experimentation.
Step 3: Add Experiment Details: To create your experiment, enter the following details:
Experiment name: This name will only be visible to you. This name is important as you will use it to identify your experiment when it is running to review the results.
Hypothesis: A hypothesis is one of the most important parts of your experiment. Your hypothesis asks a question that you expect to evaluate with your experimental content. An example hypothesis could be “Changing my product title from being vague to being more descriptive will drive more sales”. By stating and validating the hypothesis, you can work to create learnings that you can apply to products beyond those under experimentation.
Experiment duration and start dates: The recommended experiment duration is 8-10 weeks. You can always change your duration or end your experiment early. However, the longer you run your experiment, the more confident you will be in the results.
Note: Depending on the validation time of the experimental content type, the earliest start day may be several days into the future. This gives Amazon time to validate that all submitted content meets our guidelines.
Step 4: Select experimental content: Based on the experimental content type, select content this way:
Product Titles: Enter your proposed titles for experimental content into each associated box.
A+ Content: Use the selection drop-downs for Version A, to select content that has been previously approved, already has ASINs applied and is not part of a current experiment. We will only show you content that is associated with the Reference ASIN. For Version B content, it is possible to create new content, duplicate the existing Version A content to then modify or select from existing content variations that are different from Version A. In all cases, ASINs for Version B are automatically inherited from Version A to prevent a mismatch between the two versions of content.
Product images: Click Upload image and then use the file picker to select a compliant product image.
For variational ASINS, you can submit product title or product image content for some or more of the child ASINs, but you always have to submit content for the parent. This is because the parent ASIN’s content is used more broadly across the customer experience.
Typically, you’ll have to create new content to use as Version B. The easiest way to do this is to click the link on our page that says “Start by duplicating Version A”. That will create a copy of your Version A content with the same set of ASINs (both versions of content must have the same ASINs applied to submit a valid experiment).
Step 5: Submit your experiment: At this point, your experiment will be scheduled pending content validation.
Important: Make sure that you return to Manage Your Experiments in the days after you have submitted your experiment to validate that content validation passed. If it failed (for example, submitting an image with a non-white background in a category where this is required), modify the content to be compliant with the failed validation and submit your experiment again.
Any content submitted during an experiment must meet the same guidelines as any regular content not part of an experiment.
Specific guidelines for experimental content types is as follows:
Product Titles: Product titles must not have more than 200 characters, including spaces. This upper limit applies to all categories. Some categories might have a limit of even fewer characters.
Product images: Images are very important to customers, so quality matters. Choose images that are clear, easy to understand, information-rich and attractively presented.
A+ Content: Amazon has specific terms and policies regarding types of A+ content that may not be allowed. Version A must already be approved. Version B can be submitted at the time that the experiment is set up, but both versions of content need to be approved before the experiment can begin. Product title and image experiments will have their content validated as part of the experiment submission.
Edit an experiment
To edit an experiment, start by viewing the experiment details. There are multiple states that your experiment can potentially be in, and these states will affect what detail can be edited. Your experiment can be edited in these states:
Scheduled, waiting content validation: In this state, you can edit any of your experiment details including its content.
Failed content validation: In this state, content that you submitted failed validation and you need to revise your experimental content for your experiment to run.
Scheduled, successful content validation: In this state, your content has passed validation and your experiment schedule is locked. If you choose to change the experiment contents, you must select a new experiment start date to allow time to repeat the content validation process.
Experiment in progress (partial editing available): While an experiment is in progress, the content can no longer be revised. The hypothesis and duration of the experiment can still, however, be revised as desired. If you want to change the experimental content once the experiment is in progress, cancel the existing experiment and create a new one.
Experiments in these states cannot be edited:
Cancelled: Your experiment was terminated before the end date.
Completed: You experiment reached the end date. At this point, all customers will see your original (control) content again until you make a decision about what content to publish.
Important: Regardless of the experiment results, when your experiment ends, your experimental content will not publish automatically. Make sure that you publish your experimental content if you want customers to see it post-experiment.
Cancelling an experiment
To cancel an experiment, start by viewing the experiment details. Then, select Cancel experiment and provide a reason for cancellation. For example, you might indicate that cancellation was due to an experimental content error or that the experiment realised its desired result before the end date. After you have cancelled the experiment, the test will be over and results will no longer be collected. Customers viewing your product page will only view the original content. You’ll still be able to see all cancelled experiments in your experiment dashboard, along with any results that were collected up until the time of cancellation.