Why are auditors increasingly using complaints & customer feedback data?Nov 20, 2019
It’s not often that internal auditors get the opportunity to help in identifying revenue leakage.
Most audits focus on compliance matters or loss prevention.
So, when the opportunity arises, we need to act quickly.
Partnerships are a growing area for large service-based organisations, especially those with strong brands and established customer bases.
Working with specialist service providers with adjacent or complementary offerings means that:
- You don’t have to do it yourself. You can earn referral revenue from your partners for referring your customers to them.
- Importantly, your customers benefit. They can easily access those services and can rely on your curation, rather than having to vet individual service providers themselves.
But reconciling referral revenue can be a challenge – you don’t always have the data that you need to determine whether your partner is paying you for all eligible referrals.
And you can’t rely fully on data from the partner. Yes, there is a trust relationship, but you need to verify the data too – mistakes happen all the time.
One of the data sources that we can use to overcome this challenge is customer feedback, including complaints & queries.
Because the customer sees your organisation as the primary party they are dealing with, they may call you to discuss issues or queries they have with those third-party services.
This can then be used to determine whether referral revenue has been earned and paid for those customers.
It’s not an exact science. You can’t catch all the anomalies. But it has worked for us and we’ve successfully used it to find potential errors. The errors are then investigated to determine root causes, helping to identify other similar errors.
The result – lost revenue recovered, higher confidence in future revenue.
Why are auditors increasingly using complaints data?
Most auditors that use complaints data are interested in one or more of these three key benefits:
- Customer complaints can provide an alternate perspective for a range of audits e.g., identifying revenue leakage like in the example above, triangulating customer requests that had not been acted on, highlighting control gaps.
- Understanding complaints data enables audit to help management improve the complaints process, e.g., how complaints are reported on.
- Exploring the complaints data enables audit to identify strategy achievement blockers – presuming that the strategy focuses largely on customers. For example, your complaints data may point to previously unidentified problems in how services are delivered.
In this article we explore:
- the historical problem with using complaints data and why this is no longer a challenge
- what the data typically looks like – understanding complaints data
- another use case (triangulating customer requests that had not been acted on)
What the article doesn’t cover:
- The third benefit above (strategy achievement blockers) – this is an interesting angle, but we’ll leave it for a future article
- Complaints data that is recorded manually e.g., handwritten notes in a physical file. This is not impossible to analyse but converting the physical data to electronic data is a separate topic. The solutions in this article assume that the data is already in electronic form.
The historical challenge
The core problem is that most of the valuable information is in free text form.
This has historically been difficult to process because it does not have the typical structured format that most analytics requires i.e., known value ranges for each data field.
Why this is no longer a problem
Natural Language Processing (NLP) can be used to help structure the free text.
But isn’t NLP an emerging field?
No. You can, with relative ease, use NLP libraries and applications to help process free text data. Of course, as with most data analysis, you need to know what you are looking for and have some practice with how to find it.
But it is no longer an insurmountable challenge. And you don’t need expensive technology either. Your favourite open source analytics language / program will probably have options for NLP – this includes Python, R, KNIME and others.
Let’s understand complaints data
There are two broad areas that you need to understand: scope (what is included in the complaints data?) and process (how are complaints recorded?):
The first thing to understand, before delving into the complaints process, is what the definition of a complaint is.
There are differences in the determination of what exactly a complaint is – ranging from any enquiry categorised as a complaint, to only recording a complaint as such if the word “complaint” is used.
Negative and positive customer feedback
Negative customer feedback only
Official complaints only
Very specific customer “wants to complain”
Understanding the scope is important because it helps you understand what is, and isn’t, included in the complaints data.
The broader the scope, the more useful it will be.
If the scope is narrow (or very narrow) you may want to find alternate sources of customer feedback, beyond the “complaints”.
Once we understand the scope, we know what we can expect and whether to include additional sources of customer feedback. We then delve into the processes associated with each of those sources. It typically looks like this:
The key components in this diagram that we need to understand are:
- Channels – how feedback is captured e.g., email, telephone call, hotline.
- Record-keeping – how feedback is documented and recorded e.g., in a business unit specific database or a central database.
- Decision-making – how feedback is triaged and routed internally e.g., through a central assessment team, ad hoc decisions by the complaint recipient.
- Response – the various ways in which a response is prepared based on the nature and gravity of the feedback e.g., a formal commitment to the complainant about rectification, a negotiated outcome with the complainant.
- Reporting – how feedback is dealt with and reported on – are relevant serious matters escalated, is the complaints profile transparent e.g., reporting to a governance committee, ad hoc reporting to management on serious matters.
Understanding the process is important because it helps us make sense of the data when we are working through it – fewer assumptions, faster analysis.
It also helps us find improvement opportunities for management to consider.
For example, too many routing paths, feedback hidden and not escalated.
How we use complaints data for audit – another example
This previous article outlines how we can overcome the “too many false positives” challenge.
The case study that is outlined in that article focused on offset accounts – in short, this is where customers can elect to have two eligible accounts linked, with the balance in the deposit account set off against the balance in the mortgage account, resulting in lower costs for the customer.
To find accounts that should have been linked, but weren’t, customer feedback was one of the key data sources that we used. The concept was dead simple:
- Find any customer that mentioned certain keywords or phrases e.g., “offset”
- Check whether they had accounts that were eligible to be linked
- Check whether the accounts were linked
Of course, we performed a range of other eligibility checks, but it wasn’t difficult.
Importantly, the value that was produced by audit was clear.
Where to start
If the two examples above have inspired you, here are a few steps to get started:
- Understand how complaints and/or customer feedback works.
- Find an audit for which you may be able to use customer feedback data.
It may be easy to find, or you may need to stretch your imagination a bit.
- Experiment – the best way to learn.