Determining Product Effectiveness when Measurement is Difficult

As a product development lead, my validation tasks are further complicated by the nature of my particular area of focus, social products, which specifically includes both on-site user activity (commenting, voting and sharing) and off-site behaviors on social networks that have little or nascent measurement systems. So how do you  measure the un- measurable? Here are three ways to tackle this particular beast, based on what we’ve learned equating user social behavior to television tune-in:

1. Connect the Data and Run the Numbers

This one may sound obvious, but it is typically very difficult to achieve. In a perfect world, we would create an analytics  system that connects all of the disparate internal and external data sets, from the public and private social actions of all TV fans to the individualized TV tune-in history  of the same users. This is a tremendous challenge due to the scale of both social activity and television viewership, not to mention the litany of players on both sides  of the television content.

That said, this data access is the holy grail and is worth pursuing. At this company, we’ve done some remarkable things to connect the un-connectable and it’s been  invaluable to our strategic and tactical business decision-making process.

2. Break the User Path into Bite-Sized Pieces

While end-to-end - or should I say Tweet- to-remote - data analysis is a huge task, you can still derive tremendous product insights by taking validation one piece at a time. For example: What social factors best lead to television viewership? This is a ball of string with many starting points but folks are beginning to find insights.

Last year Nielsen and Twitter connected Tweets with actual Nielsen ratings and announced a causal relationship between tweets about TV viewing. Even though the  research details revealed that the effect was limited to 29 percent of shows, it was a step in the right direction. And while Twitter is only one social network of many,  they are particularly chirpy when it relates to social and TV.

Another example of bite-sized  measurement: What online events lead a user to take a social action? At Viacom we can track many of the social events that originate from our digital properties using modern quantitative analytics packages. These packages offer great scale and we can split-test audience segments to prove our  ideas are valid. That said, off-property social events dominate the social fire-hoses, so there, our visibility is limited.

3. Talk  to Users

Qualitative conversations with your customers and users is absolutely invaluable. As the best modern product development methodologies point  out, you are not the user. Instead, you are a biased business stakeholder that is likely too deeply invested in the idea to give it an impartial valuation. Share your ideas with end users early in a straight-forward, non-leading way. Design thinkers recommend using low-fi physical  prototypes that are literally stuck together with scotch tape and pipe cleaners. These interactive and tangible representations of your best ideas often generate quick  ah- ha moments, especially when they are miss- (or perhaps correctly?) interpreted by the end user.

In my experience, the most important goal of working directly with the end user is not getting them to immediately prove out your idea as valuable. Instead, working  with users allows you to quickly sift through many different ideas, some good, some bad, some biased, some pure, some half baked, and some ready for  prime-time.Sorting the wheat from chaff early allows more time for steady incremental revisions of the better ideas.