Figuring Out what to fix

This is a question I get from authors:

I get some reviews that love something, another review says that they hate that exact same thing.  Who should I listen to?

Well, I’m not an author.  But I do know that authors obsessively read everything related to themselves and their work:  Goodreads, blog posts, Amazon Reviews, etc.  They obsess over them.

In my World, I am a web developer.  We also obsess over our apps, performance, feedback, etc. and this sounds like A/B Testing.

A/B Testing means that you produce two versions of something, an “A” version and a “B” version, and see which works better.  From this, you know how to optimize your app to improve conversion rates, user engagement, or whatever other metric you’re testing.  Sounds great in theory, right?  But like most things in theory, practice proves more difficult.

An example:

Relevant portion:

Statistical significance is often the thing that most most testing software and people running tests focus on but there’s an equally important thing that is overlooked: statistical power. Or running the test with enough traffic so the result is meaningful and not chance. A page conversion change from 5% to 5.5% (an increase of 10%) could mean thousands of dollars more to the business, but to be certain about that result 62,000 users would need to go through the A/B test

You don’t know have a large enough sample size to determine if what you are doing is useful or whether you’re getting random statistical noise.
There’s another problem:
Time spent doing this is an opportunity cost, you could spend your time doing something else.  How wise is it to obsessively test what random users may or may not like?  So, I’ll share some secrets of the trade that might help out.

The “Three Jellybeans” approach

Google invented the “three” jellybeans approach, and it’s simple.  Google figured that most people reacted more strongly to things they dislike than things they like.  Given this, if they implemented a new feature, they needed three “white” jellybeans (user likes) vs. “black” jellybeans (user dislikes) to implement a feature.  If they received anything less, they rolled back the feature.

You can read the entire article here.

So what’s the problem?

The first one is something I already reviewed on my reason’s why Amazon Unlimited can be a problem for readers.  Amazon’s information is “low information” signals, reviews don’t give any useful information.

Amazon lacks a lot of features that we see in Facebook.  For example, Facebook lets you break down advertising based upon similar likes, age ranges, interests, gender, location, etc.  Amazon has no such breakdown.  Thus you can’t tell if the reviewer is someone who falls into your demographic or if they just randomly picked up the book.  From an optimization standpoint, it makes more sense to go after a specific demographic than trying to hit as broad a demographic as possible.

At Google, the same sort of advanced breakdown is available.  It’s why Google Analytics is one of their most popular programs.  Amazon has nothing like Analytics, no breakdown of average pages read, average reading speed, which sections were skipped over, what page people stop reading on (bounce rate), etc.

Because Google was able to supplement their “likes” with lots of hard data, they could make an informed decision.  Amazon lacks any hard data.


Find good beta-readers

A step up might be to find good beta readers.  But there are problems inherent to this approach.  The adjective “good” means a beta reader who can spot problems with the plot, logic, sentence structure, etc.  Most beta readers cannot give that kind of high quality feedback, which puts them back at the same random noise level as most Amazon reviews.
Stephen King’s advice is to marry the best beta-reader you can find and then your problem is solved.  May not be practical for everyone.

Join a writer’s group or groups

The Fantasy Writers group, the Indie Authors Group, and many others are dedicated towards writers.  You can find other authors on there and the advice you get from authors is what I would call high-information signals.  You can also Facebook follow your favorite authors and see what websites they recommend and follow.


Hire a Developmental Editor

The job of a developmental editor is best explained by Alan Rinzler:

Developmental editors offer specific suggestions about the core intentions and goals of the book, the underlying premise, the story, character development, use of dialogue and sensory description, the polish, narrative voice, pacing, style, language – the craft and literary art of the book.

As with talking to other authors, a developmental editor is a form of high-quality signals.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s