We’ve been told that Natural Language Processing (NLP) will change the world for a long time now. Most of you will agree that it has certainly done so; be it Google Search, Translate or Amazon Alexa, as end users, we have been using NLP systems on a daily basis and have learnt to appreciate all the benefits they provide. But to a large degree we have taken those technologies for granted. You type your search query on Google, and you get hundreds of different answers almost instantly, while in the background there are tons of complex operations and computation taking place within a few milliseconds.

If we take a step back and think about how those NLP systems were implemented, we will discover a myriad of complexities that product teams had to address. Even though NLP has many different applications across many industries, product teams need to make sure that the NLP systems they are trying to implement can perform in real world production environments.

Practical Applications of NLP

There is no shortage of real world NLP applications! Some of them include chatbots, machine translation, text summarization, text generation, semantic search and speech recognition. Most of these applications are being tackled by famous products such as Apple Siri, Amazon Alexa, Google Translate, Google Search and more.

As a product team, you have specialized needs that can be solved by NLP systems. For example, how do you offer your customers a customized search engine, in which they can quickly search items in your inventory? How do you build a chatbot that is just as smart as Siri, but can answer questions related to your own Customer Support needs?

Today, NLP systems can provide a lot of value to companies by reducing operating costs, improving their overall efficiency or enabling them to understand trends and capture business critical insights.

Modern product teams are tasked with implementing NLP systems across different business needs (operations, sales, marketing, etc.) and across many industries. However, it’s not always clear how these NLP systems can be implemented when teams are faced with challenges around accuracy, maintainability and scalability.

Building business-specific NLP

Contrary to most commercial products that are built by B2C tech powerhouses, modern product teams need to feed their own dataandprocesses into an NLP system. It is obvious that tech companies that specialize in NLP software have the resources to develop an in-house solution, but what happens in the general case? How does a product team, which is not specialized in NLP, offer a high quality NLP system to their customers?

Let’s take the example task of building an AI chatbot that can automatically respond to customer questions. Let’s consider Apple and a product team within a Fintech company.

Apple has specialized NLP and an “army” of software/NLP engineers, who can take on the challenge and build such a chatbot (Siri). Apple’s team built Siri from the ground up with the single objective of answering Apple-related questions for Apple’s customers.

On the other hand, a smaller Fintech company does not have such a specialization and has to rely on third party NLP products, which they will use to create their chatbot. The company needs to select a product that can ingest the company’s data, understand various processes and output a chatbot that can answer questions specific to the company.

Building a custom NLP system from scratch is a demanding task that most product teams avoid. They consequently turn to third party SaaS platforms that are specialized on a subset of the applications that can solve their business requirements. The hardest and biggest concern modern product teams face is how non-NLP experts can adequately select an NLP product that satisfies their needs, which functionalities guarantee long-lasting success and in what way these products will be used in production.

Different Types of NLP products

Assuming that a product team has identified a need which can be addressed by an NLP product, the next step is to do market research and identify a pool of potential candidates. NLP products can be split in two general categories:

Build it yourself

A solution like that is usually open source and contains the building blocks for getting started. Following this approach a company needs the involvement of NLP engineers and project managers, who will architect a solution and implement customized algorithms by tweaking the existing building blocks provided by the solution at hand. A custom solution is usually selected by product teams only when their problem is very specific and at the same time either have the necessary software development team resources or are willing to outsource.

Pros:

Fully customizable solution that meets even the most complex and unusual business needs
Complete outsourcing of the project entails that the company’s product team spends little time or effort

Cons:

Very Expensive
The maintainability of the NLP system is very hard
The NLP system gets outdated after a few years, unless there is continuous development by the product team

Overall, custom solutions are rarely selected by product teams, since SaaS products cover most business needs and always remain up-to-date with the latest technological advances.

SaaS Product

SaaS products are usually the best choice for product teams, since they are specialized on a specific NLP application. SaaS products are typically cloud solutions and can be either multi-tenant or single-tenant platforms. In cases where a business has security concerns or particular needs (financial institutions, healthcare organizations) some SaaS products offer an “on-premise” installation. In a nutshell:

Multi-tenant cloud: Company data, NLP models and resources are in the same cloud infrastructure as other companies using the SaaS product
Single-tenant cloud: Company data, NLP models and resources are in isolated cloud infrastructure
On-premise: Company data, NLP models and resources are located in the company’s infrastructure

Pros:

More affordable
Quick time-to-value
Easy to maintain
Easy to scale
Up to date with new features and NLP models

Cons:

In rare cases, custom business needs may not be met

The key characteristics of NLP systems

When it comes to evaluating NLP Systems, there is no one-size-fits-all. Every product team has different needs, but they need to look out for some key characteristics when evaluating whether an NLP product meets their needs.

Customization

One of the most important traits of Machine Learning and NLP algorithms is whether they can learn from any dataset. Hence a product team needs to make sure that the system can ingest custom data and understand internal processes and limitations. As a consequence, the NLP SaaS platform would output business-specific NLP models that have been trained on business-specific data and are able to solve the original business-specific need.

Different NLP applications require a higher level of customization. For example, the “text summarization” application requires much less customization than the “chatbot” application, since long articles are summarized in a universal manner. Chatbots, on the other hand, differ significantly across different companies, and consequently, product teams need to customize them.

Let’s consider the chatbot example from above, where the product team from a Fintech company implements a customer support chatbot using a SaaS NLP product. It is necessary for the chatbot to be trained on company-specific data, as it needs to understand questions coming from the end customers that are related to the Fintech company and be able to respond in a manner that follows the company’s processes.

In NLP terms, two different Fintech companies would input different data and processes into the SaaS product and would get two different NLP chatbot models as output.

Low Code, No Code, Open Source

When a product team selects an NLP product, they need to make sure that they have the skills to use it. These are the three alternatives as far as the required skills for using an NLP product are concerned:

No Code: The product team does not need to have any software development knowledge, in order to use the NLP product.
Low Code: The majority of the NLP product’s functions do not require any software development skills. However, some of the offered functionality requires software implementation. Low Code products do not usually require heavy implementation efforts.
Open Source: The NLP product provides most of the models and algorithms, but all the software engineering work needs to be done by the product team. Even though the product team would not require to have NLP research scientist, software engineers with knowledge of basic NLP principles are needed to setup and use the product.

How much data is needed?

It’s not just about the quantity of data, but also about the quality.

So, before we answer the question “How much data is needed?”, let’s make the following distinction:

Labeled Data: Data points that have been tagged with one or more labels by a human. For example, a data label might indicate whether a sentence has a positive or negative sentiment.
Unlabeled Data: Raw data points that do not contain any human tagging

Let’s take a contact center as a real world example.

The collected historical chat-logs between customers and call center agents are considered unlabeled data. On the other hand, the dataset containing customer sentences along with a positive or negative sentiment label is considered labeled data.

As you can guess, Labeled data is much harder to acquire and demand a lot of time and effort as they require human involvement. Even though unlabeled data does not always exist for some companies, they are much easier to collect.

When product teams consider using an NLP product, they need to make sure that the amount of Labeled data needed is as little as possible, because large labeled datasets are expensive to produce.

The most desirable NLP products are the ones that require as little data labeling as possible from the product teams. NLP products that can either utilize unlabeled data or require a few labeled data points are preferable, since product teams need to spend the least amount of time configuring and maintaining them.

Recent advances in Transfer Learning [LINK] in NLP have given rise to popular models like BERT and GPT-3, which are “pre-trained” on vast amounts of unlabeled data and are “fine-tuned” on a few task-specific labeled data points, yielding state of the art results.

Time to Market

The time to market is defined as the time it takes for a product team to configure an NLP product until it is deployed in production. All the characteristics described above with regards to no/low-code products or how much data is needed influence the time to market.

Again, let’s consider the chatbot example for a Fintech company. In the extreme case, where the product team chooses an open-source NLP product which requires large amounts of labeled data points, the time to market would be months.

In the ideal scenario, the product team would choose a no-code NLP product that only requires unlabeled data (past chat-logs). The time to market in this case would be a matter of days.

Maintainability

Maintaining an NLP system used in production is one of the most challenging tasks. Business needs, processes and datasets constantly change. As a result, NLP systems need to be continuously updated and improved. Just as software systems need to continuously get updated, NLP systems have to be re-trained and re-deployed multiple times to ensure long-lasting success.

Maintaining NLP models is one of the largest bottlenecks when it comes to product development and one of the largest sources of cost for companies, in terms of inefficient use of the product team’s time.

In the chatbot example from above, the product team of the Fintech company needs to be able to continuously update the answers the chatbot gives, in case their processes change but also make improvements to the NLP model. How easy is it to teach new skills to the chatbot? How fast does the chatbot learn? How can you add new chatbot flows without breaking old ones? How do you detect bad customer journeys in order to improve the chatbot? How does the model improve; is it manual or automatic? Does the product team need a dedicated person to maintain the system? These are some very important questions that need to be considered, as depending on the NLP product, the product team will either streamline the maintenance process or incur a heavy and sometimes unbearable cost.

Scalability

Inherently, when evaluating software systems, we think of scalability as how many concurrent requests it can handle in a given period of time (e.g 10 concurrent users per second). A modern product team has to always plan for success. Therefore, the scalability of an NLP system is sometimes a make-it-or-break-it characteristic. The scalability of Machine Learning models used in production is usually one of the hardest things to achieve, which is one of the main reasons why only 20% of machine learning models are eventually used in production.

Depending on the NLP application, product teams need to make sure that the NLP models that will be used in production can handle the load. In the chatbot application, the NLP model needs to be able to handle multiple concurrent requests (throughput) very fast (latency). This enables the chatbot to serve many concurrent customers with a low handling time.

Data Privacy

A product team needs to make sure that their customer’s data are protected and are not being used for the improvement of NLP models without their consent.

Especially in the case of multi-tenant platforms, product teams need to make sure that their data and their customer’s data are not being used to improve the overall NLP models of the platform.

Conclusion

In the past few years, there have been significant advances in Machine Learning and NLP, which already have a significant impact in our everyday lives. Modern product teams can capitalize on readily available technologies and apply NLP systems across many functions of their business yielding an innovative transformation where intelligent software systems provide a scalable solution for many of their problems. However, product teams need to be able to holistically evaluate NLP systems and make sure that these systems provide long-lasting value in a dynamic business setting.