AI Sentiment Analysis: PostNL Validates New Personalized Chatbot Experience

PostNL has successfully used its chatbot, Daan, for several years to support customer service. As the national postal and logistics provider in the Netherlands, PostNL handles millions of customer interactions each year.

In 2022, PostNL, with whom we were already building the PostNL app, approached us seeking an innovation boost: how could the user experience of their existing chatbot be improved and expanded? This challenge was perfectly suited for our rapid prototyping methodology.

Two early learnings: personalisation must be introduced carefully in service contexts, and sentiment analysis only adds value if its signals are both accurate in Dutch and directly usable in the chat flow.

Sentiment analysis challenge for customer satisfaction

PostNL wanted to elevate its customer service by making the chatbot interaction more personalized and effective. Together, we identified the potential value of sentiment analysis, an existing Machine Learning (ML) technology, to better help end-users and increase customer satisfaction.

However, the challenge was twofold, making the idea risky for a large investment:

1. Novel context: Sentiment analysis is typically applied to measure mood in unstructured text like online reviews or social media posts, not within the real-time, fluid context of a customer service chatbot.
2. Language barrier: This analysis is often performed on English texts, requiring us to validate the feasibility and quality of ML models for messages in the Dutch language.

This complex, high-uncertainty challenge was ideal for a Jumpstart, a focused, short-term commitment to build and test a technically working prototype with real users. Jumpstart is a methodology, a fast, cost-effective pressure cooker approach.

Machine learning solution with user testing

We approached this by combining our operational knowledge of customer service with our ML capabilities. The objective was to validate the feasibility and user acceptance of applying sentiment analysis to the live chat environment.

The solution was structured as a comprehensive one-week Jumpstart consisting of two parts:

• Technical prototype & data study: We validated the technical feasibility by investigating whether sentiment analysis could be applied within the existing chatbot technology stack. We anonymized historical data and analyzed it with various machine learning models to determine which was most suitable for the Dutch language and the chatbot context.
• Usage test: We used the resulting prototype in a user test to understand how users related to a chatbot that could recognize their mood and tailor responses accordingly.

This approach allowed us to rapidly answer crucial questions regarding the role of sentiment in the conversation and which forms of personalization worked best for the end-users.

Improved chatbot user experience and validation

The one-week project delivered a three-fold result that provided PostNL with clear, data-driven answers and minimal investment risk:

• Validated concept: PostNL validated that, with a small investment, the innovative concept of sentiment analysis was technically possible to integrate into their existing chatbot infrastructure.
• Model selection: We identified the most suitable ML model for the Dutch language and context based on quality testing of historical, anonymized data.
• User insight: The usage test provided rich insights into user behavior and expectations regarding personalized chat interactions, guiding PostNL on which personalization strategies would increase satisfaction.

Our team proved that experimentation provides answers quickly. This success provided PostNL with the strong validation needed to make an informed strategic decision regarding further investment in personalized chatbot experiences.

Key Success Factors

• The Jumpstart methodology: Building and testing a working prototype in one week provided quick answers to feasibility and user adoption questions.
• Addressing novelty: Successfully applying established ML technology (sentiment analysis) to a new context (real-time chat) and language (Dutch).
• Data-driven validation: Testing the quality of the ML models using anonymized historical data ensured the final decision was based on empirical evidence.
• Shared learning: The emphasis on the experimentation mindset and shared insights empowered the PostNL team to drive the next steps autonomously.

Involved Eidra companies
This project was a collaboration between experts from Curamando and Q42.

Read more about Curamando’s AI Automation offerings.