On Episode 46 of the Build & Analyze podcast1, Marco Arment talked a bit about beta testing and how hard it is to find good beta testers who give great feedback.
Beta testing isn’t something that we usually see other developers openly talking about, and our experience is very different from Marco’s, for the better. What follows is the story of how we approached beta testing from the beginning.
Step I: Recruit
We immediately knew that we didn’t want everyone testing the app. I tend to sign up for beta testing when developers publicly ask for testers, but I’m a terrible tester. Like most people, I just want to see the shiny new stuff. We didn’t want that for Listary. We wanted good, frequent feedback, and we felt that our chances would improve if we hand-picked our testers and invited them one by one. That’s what we did. It probably created a lot less buzz than publicly asking for a huge number of testers, but it payed off in the end.
Mixing
We wanted some of our testers to be close friends, because we’re comfortable with calling or meeting them out of nowhere to ask for feedback. Besides seeing the app simply as users, most of these friends were developers or designers and could give feedback about what could be improved, not only what was broken.
We also wanted more people from the industry that weren’t close friends and whose work we greatly admired, people who knew a lot more than ourselves and could teach us how to do things better. We were just starting and had virtually no “name” in the community, so it would be hard to get everyone to accept.
We didn’t have any other choice, so we wrote custom emails to each person we wanted to invite, knowing that it wouldn’t be so easy as asking friends. Some people didn’t answer back, some did but were too busy to test another app, and some accepted. This will always happen, and there’s no reason to be annoyed.
Research
We learned that it works best with people you feel will like the app even before trying it. There’s no way to know this for sure, but a little research helps. This is how we got Craig Mod to help us. I had been following Craig’s work for a long time, and four months before the beta testing started, when Listary wasn’t even a working prototype, he tweeted:
Any suggestions for simple todo apps that cloud sync (iOS<>desktop)?
Simplenote doesn’t cut it. I like tapping boxes to check things off.
We saved that for later, and when the time came, we emailed him something along the lines “We made the app you asked for, wanna test it?”. Who would say no to that?
We also kept an eye on the people who replied to him expressing the same wish, and we searched Twitter frequently for terms such as simplenote lists or iphone lists app. Sadly, we didn’t find many people, but it may work better for you, depending on the app.
Step II: Engage
Once everyone was on board, we told them it was extremely important for us to get honest feedback, and it would be perfectly OK if they hated the app. It’s very important receive we’re-not-afraid-to-hurt-your-feelings opinions. It’s not usual because people don’t like to be rude, but it’s the best way to learn.
With each update, instead of just writing the changelog, we wrote them nice emails regarding what had changed and why, what we were unsure about, and what was still in progress. We feel this encouraged people to reply with their opinions, and that’s exactly what happened.
Step III: Feedback
We got amazing feedback from these people. Seriously, incredible feedback. This is where our experience completely differs from Marco Arment.
We exchanged long back and forth emails about the placement of the buttons, their icons and meaning. Thanks to the feedback we got, we dramatically improved the experience for new users, and simplified the item detail screen. What is now an obvious “Add Items” screen wasn’t so obvious back then. Our testers noticed things that were misaligned by one single pixel. We even received multiple mockups with different layout possibilities for every screen in the app.
We couldn’t ask for more.
These people, with their keen eye for detail and perfection, spent their time to help us, and we’re thankful. We did a bunch of things really well, but it was them who forced us to go the extra mile. For us, they were our first users too, and interacting with them was also our first experience with customer support.
Lessons
We can’t know for sure what would happen had we proceeded differently, but we learned a bunch of things with all of this.
- Engaging with people was crucial. We feel that people became more interested in the app because we treated them like individuals, not a bunch of random users. We wrote them often, and we did our best to reply to their feedback as soon as possible.
- Happy testers are happy customers. By the time the app was launched, most of our beta testers had become customers, happy enough to spread the word about Listary.
- Industry experts are very helpful. Not only they see what’s wrong, they see room for improvement in what’s good. At times, it felt that they were part of the team too.
- Mixing close friends and industry experts is good. We had the best of both worlds: close and distant feedback, everyday phone calls and lengthy emails.
Step IV: Improve
Testing shouldn’t end after the app has launched. When we started working on version 1.1, we removed a few testers and added a bunch of new ones. This time, it was so much easier — we simply asked existing users if they wanted to test the app, most of them industry people as well. And the feedback continues to be awesome as always.
Beta testing doesn’t need to be a pain. It was a fundamental step in the development of Listary, and a great way to meet very nice people and learn with them. It was also an opportunity to practice our customer support skills. Just try to understand what people like and don’t like, and be nice. Oh, and use TestFlight, of course.
We’d like to see more developers sharing their own experience. We want to learn and improve our beta testing. If you’re a developer, get in touch and tell us what worked and didn’t work for you. Thanks.