How to avoid crashing on take-off by testing your product with real users
You want to ensure your product doesn't suck
Have you ever visited a website and you were left wondering whether it was built by a chaotic bat; or downloaded an app you were excited about but soon after aborted mission because, “who built this thing and what were they thinking?”
No?
Okay, at the very least you have encountered a situation where an app you like or need was inconvenient in specific ways. Either it takes forever to send an OTP, or a button hides a key component you want to use, or for the life of you, you cannot remember where you need to click to do a specific key task because its hidden under several vague menus.
Such scenarios are key indicators of ineffecient or insufficient user testing exercises; where the product or QA team failed to catch bugs or ask the right questions to test users- if at all they tested with actual users.
Alpha and Beta tests are part of the user acceptance testing (UAT) process. UAT is designed to help the product team investigate whether the software developed meets business and end user needs and works as anticipated.
User acceptance testing is important because:
It helps in the detection of bugs so they’re quickly fixed before they annoy mass users.
Its part of the quality assurance process that ensures you sort out any issues that could prevent your app from being blocked by app stores.
Usability testing that ensures the app or website works in predictable and similar ways regardless of the device being used to access it.
It helps you test the performance of your software when it’s being accessed by multiple people; things like loading speed, errors and seamlessness are critical to check.
It helps in spreading product awareness since it can be used as a marketing strategy with your targeted end users. A well organized test process can help you acquire and retain your first group of real users.
There are a few fundamental differences between Alpha and Beta tests. In this article, we’ll focus on how to plan and execute beta testing like the boss you are.
But first;
Differences between Alpha and Beta testing:
Alpha Tests | Beta Tests |
Usually performed by internal employees | Usually performed by end users who are not part of the company/org. |
Its perfomed in an internal testing environment (deveoper’s site) | Performed in a real-time environment that’s available to the public |
Minimal or no focus on security and reliability | Testing is focused on reliability, security, seamlessness and robustness |
Ensures the quality of the product before moving to beta testing | Also focuses on product quality, but with user’s feedback to ensure readiness for mass real-time users. |
Critical issues are directly fixed and addressed by developers. | Feedback is collected and can be addressed immediately or on a rolling basis based on urgency and product roadmap. |
Three Beta testing tools you can leverage:
TestFlight: You can invite up to 10,000 testers using just their email address or by sharing a public link. Testers can download it from the AppStore. Up to 100 apps can be tested at a time. TestFlight supports apps for iOS, iPadOS, macOS, tvOS, watchOS, and iMessage.
Firebase Crashlytics: This is a real time crash reporting tool, helps you prioritize and fix your most pervasive crashes based on the impact on real users. Crashlytics also easily integrates into your Android, iOS, macOS, tvOS, and watchOS apps.
TestFairy: Works on both iOs and Android and allows users to report bugs by shaking their device or just clicking on a button and sketching on the screen. TestFairy includes an enterprise-grade app distribution platform for iOS and Android that is highly configurable, tightly secured and can sync with your corporate Single Sign-on.
How to map out your beta-testing roadmap:
Stage 1: Getting Launch Ready
Tasks | Goal |
|---|---|
Key Features Developed | There needs to be something to be tested. And you want the critical features ready to be brought to the test group. |
Acceptance criteria well defined | Acceptance criteria guides the testers to measure whether the feature meets the specified expectation. Well defined criteria can be answered using a yes or a no. |
Software/App Pushed to Test Environment | The initial test environment for Alpha testers is usually a sandbox or a developer’s site. The App/Sotware should be available in that environment to allow for testing. |
Alpha testers added to sandbox/test environment | Alpha testing is done in a closed environment with limited access. The testers (usually internal team members) have to be added into the environment so they can access the app/software for testing. |
Internal Alpha-testing of App completed | Internal alpha testers mark yes or no on the acceptance criteria of specific features to indicate whether they work as stipulated or not. |
App pushed to production | Once the alpha test is done and changes made, the developers push the app/software to production (ie AppStore/Playstore) or soft-launch the website so it can be accessed by the public/beta testers. |
Stage 2: Beta-Planning
Task | Goal |
Create beta- test SMART goals and tasks | Specify the tasks you want to test through the beta-test expercise, and the goals you want to achieve with the exercise. Some key goal-setting questions could include: What do you want from this beta program? Will you be engaging these users on a weekly, monthly, or quarterly basis? How will you be engaging them? Will it be simply for new features or innovation and new product releases? How often will you require them to give feedback? |
Finalise on the testing strategy to be used | This could include specifying whether it’s virtual, in-person or both; how long the testing period will last, what rewards (if any) will be offered to the testers, and where the testers will be found. |
Assign team responsibility in the testing process | Ensure every stakeholder knows their role and the associated deadlines; this includes any review or approval processes (such as budgetary approvals) |
Narrow down specific dates and deadlines | You want to be have a limited window so that testers stay engaged and prioritize the process. |
Decide on any beta-testing tools needed, where appropriate | Test your new software on multiple platforms and beta-testing tools. Depending on the software, this could involve testing it on both MAC and PC computers, both iOS and Android mobile device, and Chrome, Edge, and Safari internet browsers. The software could behave differently across different platforms. Different testing tools/platforms allow you to test for specific things, so choose approriately. |
Create a communications plan/strategy for the beta-testing exercise. | Timely, clear and targeted cross-functional and external communication makes for a seamless testing exercise. |
Stage 3: Legalities and Shortlisting
Task | Goal |
Prepare an NDA and Participation Agreement for testers to sign | You don’t want your idea stolen before it makes it to the market; or to get sued because you processed a tester’s personal information without express permission. |
Create ideal test-user persona | You want your target market to make a significant part of your test user group, so you know that you’re building something YOUR users want. |
Cordinate with internal teams to help shortlist potential test users | Customer success, social media team and sales team can help curate targeted, precise list of users who fit your test group persona |
Reach out to communities where test users are likley to be | You want to let people know you’re ready for launch and keep them excited to receive a solution to a problem they’re facing. Excellent marketing |
Settle on dates and locations of beta-testing | You want to communicate, well in advance to your test users the logistics around the exercise. |
Stage 4: Beta-testing
Task | Goal |
Specify internal team members to beta-test and their locations (if in-person). | Relevant team members need to own specific parts of the testing exercise; and team can map out availability. |
Organize logistics for team members (digital or physical) | Prior preperation helps everyone feel in control and prepared to do their best work. |
Organize logistics for testers, if necessary (internet, location, transportation and meals; if in-person) | Users should feel that you care about their convenience. |
Communicate relevant information with beta-testers consistently | You want as many shortlisted participants as possible to show up for the testing exercise. |
Conduct the testing over the specified range of dates | Time to know whether you’ve built something worth having. |
Stage 5: Beta Feedback
Task | Goal |
Bug reports | Fix them before they annoy and frustrate everyone else. |
Submit Feature requests or improvements | You know what your users want more of; add it to your backlog and you’ll refine it later. |
Journal/Note-take | Take real-time notes of your observations and insights you gather as you carry out your exercise |
Evaluate Feedback | Ensure you come up with the correct POV when you look at the feedback in toto |
Map required changes where necessary | Groom the backlog of feedback into now, next and never columns of changes to be made. |
Stage 6: To the Moon!
User testing as a beginner:
As a PM, you'll encounter your very first user testing exercise. Or, an opportunity to stop limping through the process and do it well.
I've created a user-testing playbook that helped me facilitate cross-country testing exercises, leading to my promotion just 8 months into my first product role. This playbook is now available on Gumroad and is filled with Notion templates that will help you plan the logistics, communicate with users effectively, and record their feedback.
With this playbook which includes other resources that are helpful for a new PM, you can take the guesswork out of user testing and ensure that your products are designed to meet the needs of your users. Don't miss out on this opportunity to take your product strategy to the next level.
You can get the Testing Textbook here: