Results of the Flagfox 4.0 beta tester survey

Any feedback, suggestion, bug reports, problems....
Post Reply
User avatar
DaveG
Flagfox Developer
Posts: 723
Joined: Wed Oct 03, 2007 9:06 pm
Location: Philadelphia, USA

Results of the Flagfox 4.0 beta tester survey

Post by DaveG » Sun Feb 21, 2010 9:12 am

I said I'd post this somewhere when I was done, though I doubt many people will care too much about it beyond Richard. ;)

A report of the survey results is accessible here. (no names shown)

The beta testers pool got all the way up to over 200, which is right around what I was hoping for. This was great, and I got more testing and feedback, though not quite as much as I would've liked. The survey... however... let's just say the submission rate was pathetic. Grand total: 13 completions, 11 if you drop me and Richard, less if you drop translators. The sad part is that this after I started yelling at the beta testers to fill it out in their first run page. It was around a 4% submission rate, so now it's up to 5% or so. I'm just dumbfounded that people can find and install a beta version, clearly labeled as for testing and feedback at every turn, and yet almost none of them bother to fill out a short anonymous survey. There were also 13 more partial submissions from people who filled out the first page and then gave up. Again, I find this astonishing.

The mind-boggling part about this is that if I were to have done this with a normal release, even if I got an even lower submission rate I'd be over 60000. The free version tops out at 250 a month. :roll:

It wasn't a total waste though. The fact that the traceroute action was broken in an early beta was reported in there, which led me to fix it quickly whereas I might've missed it otherwise. There were also a few other interesting bits in there.

Things I've learned from this stupid survey:
  1. People are lazy, and even if they went out of their way to get to this point, they're still lazy. Granted, I can be lazy on occasion myself, but I was hoping for at least a double digit submission rate from so-called beta testers...
  2. To deal with the lazy, I should probably ditch all superfluous questions and boil it down to a short and useful survey on one page. I had a whole bunch of "it'd be nice to know" questions in there that apparently weren't helping me. I'm also pretty sure that while the survey itself was rather short, especially if you skipped optional questions, that by virtue of the fact that it was on 3 pages it made it look longer than it was.
  3. Have only one "anything else" box. The last page was mostly empty optional boxes with free-form questions. Probably made it look too complicated.
  4. Explicitly tell users to not answer "no" or "n/a" in optional boxes.
  5. Not allow "other" fields for multiple choice answers if I can avoid it. Asking users in-general to enter something specific seems dubious.
Actual useful non-bug-related feedback in the survey submissions:
  • Virtually nobody cares about Fennec (Firefox Mobile). This surprised me. I was actually vaguely considering investigating writing a Fennec version, and now I'm going to not bother, at least for a long while.
  • User managed separators/menus for customizing the context menu aren't fantastically popular either, but not as clear a "no" as Fennec.
  • The average rating of the current list of default actions is exactly in the middle ("Looks good to me"). Exactly the same amount rated it above as rated it less. This proves to me that this question was either completely useless or the list really is just statistically perfect.
  • Nobody completing the survey had poor fluency in English and not a one was in the Western Hemisphere with me. (except for some unfinished responses) Respondents were primarily European with one Chinese.
Richard: Can you read anything else important into these tea leaves, or are these results even more vague than I'm seeing?

Now that I've typed out this rambling post, I'm thinking that I wasted too much time on it... :doh:

Post Reply