Road to Release: Your step-by-step guide to beta-testing in 2023

How to avoid crashing on take-off by testing your product with real users

You want to ensure your product doesn't suck

Have you ever visited a website and you were left wondering whether it was built by a chaotic bat; or have you downloaded an app you were excited about but soon after aborted mission because, “who built this thing and what were they thinking?”

No?

Okay, at the very least you have encountered a situation where an app you like or need was inconvenient in specific ways. Either it takes forever to send an OTP, or a button hides a key component you want to use, or for the life of you, you cannot remember where you need to click to do a specific key task because its hidden under several vague menus.

Such scenarios are key indicators of ineffecient or insufficient user testing exercises; where the product or QA team failed to catch bugs or ask the right questions to test users- if at all they tested with actual users.

Alpha and Beta tests are part of the user acceptance testing (UAT) process. UAT is designed to help the product team investigate whether the software developed meets business and end user needs and works as anticipated.

User acceptance testing is important because:

• It helps in the detection of bugs so they’re quickly fixed before they annoy mass users.

• Its part of the quality assurance process that ensures you sort out any issues that could prevent your app from being blocked by app stores.

• Usability testing that ensures the app or website works in predictable and similar ways regardless of the device being used to access it.

• It helps you test the performance of your software when it’s being accessed by multiple people; things like loading speed, errors and seamlessness are critical to check.

• It helps in spreading product awareness since it can be used as a marketing strategy with your targeted end users. A well organized test process can help you acquire and retain your first group of real users.

There are a few fundamental differences between Alpha and Beta tests. In this article, we’ll focus on how to plan and execute beta testing like the boss you are.

But first;

Differences between Alpha and Beta testing:

Alpha Tests	Beta Tests
​Usually performed by internal employees	Usually performed by end users who are not part of the company/org.
Its perfomed in an internal testing environment (deveoper’s site)	Performed in a real-time environment that’s available to the public
Minimal or no focus on security and reliability	Testing is focused on reliability, security, seamlessness and robustness
Ensures the quality of the product before moving to beta testing	Also focuses on product quality, but with user’s feedback to ensure readiness for mass real-time users.
Critical issues are directly fixed and addressed by developers.	Feedback is collected and can be addressed immediately or on a rolling basis based on urgency and product roadmap.

Three Beta testing tools you can leverage:

1. TestFlight: You can invite up to 10,000 testers using just their email address or by sharing a public link. Testers can download it from the AppStore. Up to 100 apps can be tested at a time. TestFlight supports apps for iOS, iPadOS, macOS, tvOS, watchOS, and iMessage.

2. Firebase Crashlytics: This is a real time crash reporting tool, helps you prioritize and fix your most pervasive crashes based on the impact on real users. Crashlytics also easily integrates into your Android, iOS, macOS, tvOS, and watchOS apps.

3. TestFairy: Works on both iOs and Android and allows users to report bugs by shaking their device or just clicking on a button and sketching on the screen. TestFairy includes an enterprise-grade app distribution platform for iOS and Android that is highly configurable, tightly secured and can sync with your corporate Single Sign-on.

How to map out your beta-testing roadmap:

Stage 1: Getting Launch Ready

​Tasks	​Goal
Key Features Developed	There needs to be something to be tested. And you want the critical features ready to be brought to the test group.
Acceptance criteria well defined	Acceptance criteria guides the testers to measure whether the feature meets the specified expectation. Well defined criteria can be answered using a yes or a no.
Software/App Pushed to Test Environment	The initial test environment for Alpha testers is usually a sandbox or a developer’s site. The App/Sotware should be available in that environment to allow for testing.
Alpha testers added to sandbox/test environment	Alpha testing is done in a closed environment with limited access. The testers (usually internal team members) have to be added into the environment so they can access the app/software for testing.
Internal Alpha-testing of App completed	Internal alpha testers mark yes or no on the acceptance criteria of specific features to indicate whether they work as stipulated or not.
App pushed to production	Once the alpha test is done and changes made, the developers push the app/software to production (ie AppStore/Playstore) or soft-launch the website so it can be accessed by the public/beta testers.

Stage 2: Beta-Planning

Task	Goal
​Create beta- test SMART goals and tasks	Specify the tasks you want to test through the beta-test expercise, and the goals you want to achieve with the exercise. Some key goal-setting questions could include: What do you want from this beta program? Will you be engaging these users on a weekly, monthly, or quarterly basis? How will you be engaging them? Will it be simply for new features or innovation and new product releases? How often will you require them to give feedback?
Finalise on the testing strategy to be used	​This could include specifying whether it’s virtual, in-person or both; how long the testing period will last, what rewards (if any) will be offered to the testers, and where the testers will be found.
Assign team responsibility in the testing process	Ensure every stakeholder knows their role and the associated deadlines; this includes any review or approval processes (such as budgetary approvals)
Narrow down specific dates and deadlines	​You want to be have a limited window so that testers stay engaged and prioritize the process.
Decide on any beta-testing tools needed, where appropriate	​Test your new software on multiple platforms and beta-testing tools. Depending on the software, this could involve testing it on both MAC and PC computers, both iOS and Android mobile device, and Chrome, Edge, and Safari internet browsers. The software could behave differently across different platforms. Different testing tools/platforms allow you to test for specific things, so choose approriately.
Create a communications plan/strategy for the beta-testing exercise.	Timely, clear and targeted cross-functional and external communication makes for a seamless testing exercise.

Stage 3: Legalities and Shortlisting

Task	Goal
Prepare an NDA and Participation Agreement for testers to sign	You don’t want your idea stolen before it makes it to the market; or to get sued because you processed a tester’s personal information without express permission.
Create ideal test-user persona	​You want your target market to make a significant part of your test user group, so you know that you’re building something YOUR users want.
Cordinate with internal teams to help shortlist potential test users	Customer success, social media team and sales team can help curate targeted, precise list of users who fit your test group persona
Reach out to communities where test users are likley to be	You want to let people know you’re ready for launch and keep them excited to receive a solution to a problem they’re facing. Excellent marketing
Settle on dates and locations of beta-testing	You want to communicate, well in advance to your test users the logistics around the exercise.

Stage 4: Beta-testing

Task	Goal
Specify internal team members to beta-test and their locations (if in-person).	Relevant team members need to own specific parts of the testing exercise; and team can map out availability.
Organize logistics for team members (digital or physical)	Prior preperation helps everyone feel in control and prepared to do their best work.
Organize logistics for testers, if necessary (internet, location, transportation and meals; if in-person)	Users should feel that you care about their convenience.
Communicate relevant information with beta-testers consistently	You want as many shortlisted participants as possible to show up for the testing exercise.
Conduct the testing over the specified range of dates	Time to know whether you’ve built something worth having.

Stage 5: Beta Feedback

Task	Goal
Bug reports	Fix them before they annoy and frustrate everyone else.
Submit Feature requests or improvements	You know what your users want more of; add it to your backlog and you’ll refine it later.
Journal/Note-take	Take real-time notes of your observations and insights you gather as you carry out your exercise
Evaluate Feedback	Ensure you come up with the correct POV when you look at the feedback in toto
Map required changes where necessary	​Groom the backlog of feedback into now, next and never columns of changes to be made.

Stage 6: To the Moon!

User testing as a beginner:

As a PM, you'll encounter your very first user testing exercise. Or, an opportunity to stop limping through the process and do it well.

I've created a user-testing playbook that helped me facilitate cross-country testing exercises, leading to my promotion just 8 months into my first product role. This playbook is now available on Gumroad and is filled with Notion templates that will help you plan the logistics, communicate with users effectively, and record their feedback.

With this playbook which includes other resources that are helpful for a new PM, you can take the guesswork out of user testing and ensure that your products are designed to meet the needs of your users. Don't miss out on this opportunity to take your product strategy to the next level.

You can get the Testing Textbook here:

Access the Testing Textbook