How Natural Language Processing helps us understand consumers better

Natural Language Processing is one technology solution that can help you understand what your customers say and do that at scale. But how does it work, what do you need to be aware of, and what roles do humans play in a Natural Language Processing analysis?

This article covers:

• What is Natural Language Processing?
• Where should NLP be used?
• Using NLP efficiently
• How to get buy-in for Natural Language Processing?

What is Natural Language Processing?

In short: NLP is an acronym commonly used for Natural Language Processing.

Natural Language Processing is an area of artificial intelligence and is the process of enabling machines to learn and understand human language. NLP is an important piece of text analytics strategies that help researchers understand the context behind what consumers are saying.

“Fundamentally, it’s about a machine trying to understand what us humans are trying to say,” said Andy Barraclough, CTO and founder at Voxpopme, on an episode of “Reel Talk: The Customer Insights Show.”

Jenn Vogel, CRO at Voxpopme and host of “Reel Talk,” added that there’s also a line where the machines’ abilities end and human intervention becomes necessary.

The nuances of human languages can be mindboggling – even to people.

Take the response of:

“It hasn’t rained in weeks.”

Andy said that could be taken as:

• it’s been a great summer
• negatively if said by farmers, because fields need water

Using NLP, machines are learning to understand and then analyze those subleties.

Where should we use NLP?

It’s essential to think about what the goal is, Andy said. At times, NLP might replace something. Or it might enhance a process.

Like anything, Natural Language Processing has areas where its strongest and some areas where it can still improve.

“It’s come leaps and bounds like a lot of technology has,” Andy said. “But our language as humans is constantly evolving, the way we speak to each other, the language that we use, the words that we use. The slang, accents, and everything around that is constantly evolving.”

In other words, it’s hard for machines and humans to keep up with all those changes. And that shortcoming is something we want to keep in mind and understand.

“There are different scales that we can rely on,” Andy said. “It depends on what you want to get out of it.”

Many of us already interact with NLP in our day-to-day anyway. Think of Siri and Alexa, for example. And sometimes they understand what we are saying or are trying to say correctly. Other times, they don’t.

Read next: Market research technology: What’s the role of market research automation?

Understanding some of the advantages and disadvantages of when NLP works well and where it could still improve are good to consider. And also, remember that improvements happen – including in machine translation, which I will discuss shortly. Others areas where NLP is already used includes chatbots and feedback sentiment analysis.

Chatbots

NLP tries to understand what people say when talking to the bot. The responses are then based on the machine’s understanding of the conversation. NLP in some chatbots certainly has come a long way. I remember cases when a chatbot would respond to me: “I don’t understand. Can you please rephrase/use different words?”

The trick is that the machines adjust to how people talk, not the people having to adjust their language so the machines can understand them.

Sentiment analysis

I use NLP weekly during ongoing consumer studies of trending topics – for example, when we asked about favorite mustard brands.

The system analyzes what people say and assigns it to positive, negative, or neutral statements to offer the researcher an overview of the makeup of statements.

Machine translation

Machine translation is another example of NLP and is more helpful than not being able to communicate with people who speak a different language. It’s also still evolving, Andy said. Take the example of translating from one language into another and then back. The final translation wasn’t always great in the past. That’s because of the nuances of language and how things get translated and interpreted.

But, there seem to be improvements. By way of example, I translated this English paragraph into Hindi:

I want to try Natural Language Processing more and more, but I don’t have the right budget right now. Also, my team is always understaffed. Need ideas!

In Hindi, it was listed as:

मैं नेचुरल लैंग्वेज प्रोसेसिंग को अधिक से अधिक आजमाना चाहता हूं, लेकिन मेरे पास अभी सही बजट नहीं है। साथ ही, मेरी टीम में हमेशा स्टाफ की कमी रहती है। विचारों की आवश्यकता है!

Then translating that back into English got me this:

I want to try Natural Language Processing as much as possible, but I don’t have the right budget right now. Also, my team is always short of staff. Ideas needed! (Emphasis added to highlight changes)

So there were some changes, but the meaning of the paragraph essentially stayed the same, accurate and understandable.

Using NLP efficiently

For NLP to be most effective for you, the data – including the context of the data need to be provided. How else could we analyze the “it didn’t rain for weeks” comment if we don’t understand who we are talking to and what the context of that conversation is?

That’s not always easy to do, said Andy.

“That’s why we need to understand what we are trying to get out of these systems and roles we have to play as humans,” Andy said.

For example, an NLP analysis can get you an overview of trends. In the case of the sentiment analysis, the breakdown of positive, negative, and neutral comments gives me an excellent synopsis of where the comments are trending.

Some people might become skeptical when something wasn’t categorized as they thought it would be.  On the flip side, they might also be pleasantly surprised by advances in NLP. I was surprised by how close the machine translation ended up being – which was way better than the last time I tried that same scenario, which was probably years ago.

Certainly, we might be able to see things that NLP didn’t see when we manually go through responses. But, on the flip side, NLP can also catch things we may have missed in our manual review. And, NLP will get us some trend ideas quicker than any manual review would.

Read next: Trendspotting: Seeing and understanding trends with consumer insights

Consider the amount of data that you have available. Andy explained that NLP works well with larger data sets while training it on smaller sets can be hard.

How to get buy-in for Natural Language Processing

Understanding and communicating NLP’s strengths and how they fit into your goals is certainly one way. Trust in the data is always an important bridge to cross as well.

Quick wins

Start using NLP in certain projects and then share the useful results with stakeholders and executives. That can be a quick win – sharing something that worked and showing its impact on the overall actionable insights being presented.

It’s unlikely that human involvement can be completely stripped out of the process involving sophisticated data.

“Seeing the data is also important,” Andy said. “You can spot-check the data, which can be quite useful.”

That can also build trust with the team when spot checks and the NLP output mesh.

Saving time

Also, be clear where NLP can help when it comes to:

• saving time on certain tasks, which can then be used elsewhere
• creating analysis that wasn’t as easily possible previously

Showing these results can help with buy-in and acceptance of new technologies in a business.

Different perspectives

NLP can also add perspectives on a situation that may have been absent. Think about the projects where three different people have three different views on what insights are actually being discovered. NLP can offer a data-backed approach here.

“Perhaps it will open up different routes of innovation, customer discovery, and understanding,” Andy said. “It’s about how do we leverage it. Maybe it’s another perspective.”

The speed

Many of us are under the pressure of constant and fast results, and NLP can speed up the process of getting insights. That doesn’t mean NLP should be used for every project. It also doesn’t mean every project needs to be done quickly. But the option of NLP and agile qual can generally help us speed up projects.

“From a product perspective, by iterating through an idea, you are uncovering different sets of information that provide you the ability to make another decision,” Andy added. “I think when we plan out too far into the future, we blinker ourselves from the ability to see more from the information we gather in a different use case. This can help you uncover something new.”

Read next: How to follow the design thinking process to be more relevant to customers

Sometimes, we try to get to that perfect answer in research, Andy said, but getting insights along the way can be so helpful in making better decisions.

Data integrity

Many companies have plenty of data, but as somebody once said: “Garbage in, garbage out.” Bad data won’t help us understand our customers better. But cleaning up our data, understanding where it’s coming from and what it includes can help. The more we know about the data we have and the more confident we are about its accuracy, the more useful NLP will be.

“We must be conscious of where we are getting our data,” Andy said. “The data can potentially have a negative impact on what gets pulled out.”

But there’s plenty of data that companies already have that has the potential to be used to train the AI. For example, you could use:

• existing customer interview video or audio
• open-end responses collected over recent months or when still timely even years

In summary, NLP can help us scale our consumer conversations. That can happen through new asynchronous video feedback, remote interviews, and even existing content that can be analyzed anew using Natural Language Processing.

Like any new technology or method, it has disadvantages and advantages. Understanding those and how they can bring new perspectives to the customer can help us be even more successful.

Like this article? Read more like it here.