Nedbank has upgraded its data monitoring systems after external auditors found horrendous differences in the retail unit’s analytics platforms. Nedbank’s head of retail decision science support, Eugene Liebenberg, discusses the challenge of transforming de-centralised data monitoring systems into one single platform.
With responsibility for all marketing, attrition and prediction data models in Nedbank’s retail unit, head of retail decision science support (DSS) Eugene Liebenberg was in charge of a pile of de-centralised data systems.
His team wanted to improve their analytics models and methodologies, so Monocle Solutions, a risk assessment and optimisation firm, audited the models.
“What they came back with was, basically, quite horrendous,” Liebenberg told RBI.
“They found most of the units were de-centralised, they did not follow the same methodologies and many of the big operational models based within Nedbank were saved on normal desktop computers.
“Nobody knew who had developed the models. Some of them did not have owners, so if there had been an issue, no one would have known who to go to or how to fix the problem. And some models also did not have any documentation on them.”
Liebenberg said that Nedbank had been working with SAS for a decade and already deployed its Predictive Analytics Suite to increase analytical choices for marketing modellers and SAS Enterprise Miner for production models.
So it “made sense to go with SAS”.
The vendor provided Nedbank with “a little roadmap to consolidate” their de-centralised data models: SAS Model Manager.
According to a SAS spokeswoman: “The system offers a range of models stored in one place, so Nedbank’s intellectual property does not go onto peoples’ laptops and out of the front door.”
The spokeswoman said the system helps predict customer behaviour, reduce churn, minimise credit risk, combat fraud, reduce asset downtime, anticipate risk and plan intervention methods.
Nedbank’s retail division began implementation of SAS Model Manager in September 2009. It integrated the software in the unit’s infrastructure within two weeks, deploying it for complete lifecycle management.
Liebenberg said: “In Model Manager, we can monitor the performance of all models over time. We can see if a model is degrading or actually improving.
“A lot of people perceive model management as just managing the models. It is not only about that. Model management is managing your data, your infrastructure, your software and your people.”
“If you don’t manage the people that actually develop the models, you will not be able to deploy in a short time. It would take three to four months. But because we have the knowledge sharing and cooperation between the whole business units within retail, everything is running quite smoothly.”
Everybody in the retail DSS team has access to Model Manager, so everybody can see what the next step will be, who is signing off, and what documentation has been done to date and what is outstanding.
Liebenberg said that this communal access opens up “communication channels” among analysts, while monthly cross-business discussions focus on how to improve a current model.
“We previously couldn’t do that,” he said.
Since the implementation of Model Manager last year, the retail unit witnessed a “20 to 30% improvement” in model performance.
But convincing the bank’s analysts and executives to share knowledge and making them adapt to the centralised model was “the most difficult thing to do”, he said.
“Executives don’t understand data monitoring reports or results that were developed out of statistical models,” Liebenberg added.
“So part of the implementation process was a mind shift, to understand what we were doing, why and how we were changing it. It did not matter how you did it [collect data], just as long as it worked.”
“A lot of the analysts, including myself, never shared knowledge at some stage, because we felt that the model we had built was ours and that people were going to steal our work.”
“Making them realise that sharing data and sharing modelling techniques would help them improve their own skills was quite a challenge. It took about six months for them to understand why we were centralising the models.”
Liebenberg said that the reluctance to share knowledge was not only across the bank’s business units, but also within retail division itself.
“We had a lot of different data departments within retail and even between them the analysts did not want to share.”
He said that analysts believed secrecy created job security because they were the only ones who knew how the models worked.
But the unit did keep some ‘uniqueness’:
“What makes us [the retail DSS team] unique is that our modellers file their own data as well,” he said.