QA Experiment Checklist

Background

Taplytics makes it as easy as possible to running QA on experiments with a number of different options to choose from to fit within your team’s QA workflow. For the purposes of this checklist we will make a suggestion based off of our recommended workflow, but feel free to adjust based off of your own needs.

If you need help building a custom QA process, our Customer Success team would be happy to help!

Helpful Tips & Tools

There are some tools and initial SDK setup tips that can make your life easier in understanding what’s happening “under the hood” so to speak. If you can be armed with these ahead of running through QA, you can save yourself a lot of time.

This guide will not require the use of anything more than the core Taplytics platform and your QA devices, but if you would like help with how to use other tools in your QA process, our Customer Success team would be happy to help get you set up.

Access to the Taplytics Interface
I know this should be a no-brainer, but sometimes QA users don’t get access to the tool that is generating the push notifications. Making sure you or other QA users have access to your app’s Taplytics project should be the first thing you do.
Setting up Advanced Pairing
With Advanced Pairing set up, you can send yourself a custom link to identify your devices to Taplytics. Making them easily discoverable in the Taplytics interface. The docs for Advanced pairing can be found here for iOS and Android.
Setting a User ID
If you can have your team set a custom user ID to Taplytics it can be used as another method of finding your device in Taplytics, by searching User Insights. The docs for setting a User ID can be found here for iOS and Android.
Charles (or some other proxying tool)
Charles is a network monitoring that shows you the network requests that your device is making. This tool is amazingly useful for letting you know what your device is getting from Taplytics.
Taplytics BigQuery (if you aren’t using this yet, ask your Account Manager)
Taplytics BigQuery gives you and your team access to the real-time feed of all of your raw data within Taplytics. If you know SQL, you can query the data on the fly to see what events and data Taplytics is logging.

Process Rules:

Require each QA user to have their own user account for QA.
Ensure that there is only one build of the app running on a QA device at a time.
Log in to your app at the beginning of each session.
Log out of your app at the end of each session.
Always log out of your account before deleting a build of the app.

QA of On-Device Experience:

Ensure your experiment and variations are properly set up and your experiment is in Draft
Install the most recent build of your app onto your test device
Find your specific device in Taplytics
If you’re using Advanced Pairing send yourself a link and find your device on the Device Settings page
Or search for your device in Taplytics User Insights by using your user ID
Navigate to your user profile on User Insights
Select your specific QA device in the dropdown of the Variation Switcher
Add your device to the draft experiment you are running QA for
Apply the variation for the experiment
If your app was open force close it
Open your app
Navigate to the area of the app where the experiment should be and observe the change
Go back to the Variation Switcher on Taplytics User Insights
Apply the baseline variation
Force close your app
Open your app
Navigate to the area of the app where the experiment should be and observe the experiment in baseline
Repeat the application and observation stages as many times as necessary and try other ways of navigating to the area of your app where the change should be applied
If you're blocked on QA because you have multiple devices in the QA device dropdown menu (shown below), here are some steps you can take to be able to ensure that you're choosing the right device to be able to force it into an experiment and variation that you're QA'ing.

Fullscreen_2018-01-31__12_24_PM.jpg

Clear any app data
Reset your device's advertising ID
delete the build of the app that you're testing
Reinstall the app
From there, you'll want to find your device to be able to force it in the right variation and see if the proper responses are coming down through Charles.

You can do that in two ways:

Pairing your app and finding it in your Device Settings
Finding your device in the User Insights in the lefthand navigation panel. There, you can use a combination of filters to pinpoint your device.

Once you find your device, you can use the variation switcher to manually force a device into an experiment and the desired variation.

Let us know if you have any issues with QA by emailing [email protected] or if you have a private slack channel that was already set up for you, message us there! :)