Why Not Do It By Hand? Manual Processing Vs Automated Listening
Executive Summary
Marketing, customer experience, and social media professionals often hold the misconception that manual analysis of customer conversations is an adequate and accurate way of analyzing the voice of their customers to derive actionable insights. However, a closer look at the process of manually monitoring customer comments – “doing it by hand”— reveals increased error rates, high costs, and delayed time to insight. This paper explores the shortcomings of manual conversation analysis, and demonstrates the value of automation technologies like OpenMic to increase analytical accuracy and time-to-insight across customer listening initiatives.
Why is Manual Analysis Inaccurate?
A person can generally classify a single user-generated comment more accurately than any automated system. But for classifying hundreds of messages, automated systems do just as well as humans. For classifying thousands, or hundreds of thousands of customer verbatim comments, automated performance is always better than manual. In addition to improved accuracy, automated classification can perform the task in a fraction of the time and cost as human efforts. This difference is reflected in the high cost inherent in market-research oriented services that analyze feedback, social media conversations, and other voice of the customer data.
The factors which render manual categorization ineffective are a combination of subjectivity and fatigue. When a person classifies text documents, the first one encountered gets lots of attention and careful thought. The second gets not quite so much attention, the third less still… and by the time a person gets to the 100th document he or she has begun to suffer from wavering attention and fatigue. By the time a person gets to the 500th, the accuracy with which they assign categories has seriously deteriorated. Beyond a few thousand documents, the main result of manual categorization isn’t better information – it’s eye strain. Moreover, every individual will categorize records according to their own interpretation and experience, resulting in often-inconsistent results (e.g., see real-world examples below). Even a single individual’s judgment will drift over time, and it is not uncommon to apply different weightings of topic and sentiment to the same messages. Automated categorization produces greater accuracy, with greater consistency.
