WHAT’S IN A NAME? THAT WHICH WE CALL OPERATIONS BY ANY OTHER NAME WOULD SMELL AS SWEET…

The unique challenges to overcome when building out AI Ops

By Diana Rodenberger & Rich Sokolosky, Sentier Analytics

First, let us start with the premise that AI (artificial intelligence) has become a “need to have” to stay competitive for commercial life sciences organizations. AI will be at the heart of moving to a digitally enabled, omnichannel-driven effort to improve patient and physician engagement. Just because the need is known does not mean the underpinnings to make it happen are in place. 

When we first started talking about using a random forest model to assess sales and marketing activity impact, analytic leaders looked at us like we had three heads. That has changed. In the intervening years life sciences companies have hired data scientists when they could find them, and built teams to answer important business questions through machine learning. This has produced many successes. Give the team a question, they’d answer it. Give them another…and so on. The AI developed by these teams has been single-question focused, encapsulated, and very dependent on the data scientists doing the analysis. Reusability and automation have not been the primary concern of these efforts. It was simply enough that the one-off answers generated significant value. 

Going forward, however, AI is being tasked to routinely and systematically identify the right parties, determine their preferences, coordinate activities, forecast outcomes, and measure results -- all while dynamically adjusting to changing conditions. Enter AI Ops (AI Operations). Just like there are structures and processes in place for data and other operations, there now needs to be an approach to AI Ops. But will it smell as sweet? 

To be clear, AI Ops is not referring to a standardized end-user platform (i.e., an omnichannel platform where channel owners can log in and perform their coordinated and assigned duties). AI Ops are centered around the operationalized machine and deep learning engines that provide the actionable insights to the platforms on an as-needed basis. These engines are constantly enhanced, customized to each market/company, and drive significant competitive advantages. So again, will AI Ops smell as sweet? AI Ops brings some unique challenges that make it different from other areas of operations.

1. Constant and complex change is a fact of life

“Train the model.” This is a phrase that will become ingrained in your head. So you train it, stand it up and use it. But when the data changes – a new competitor brand enters the market, new HCPs are added to the list, a new campaign is launched, etc. – the existing model no longer accurately predicts outcomes and you have to re-train the model. That is fine if the training works but what if the model still does not accurately predict outcomes? Then you have to check, and potentially adjust, a myriad of inputs including the data, the model settings, and the version of the python package you are using. Or in the extreme case, revisit the modeling approach. All this is to say that change in AI is constant and pervasive across a complex and multifaceted environment. It ultimately means that there are more dynamic factors to account for when operationalizing AI.

2. Fully automated? Probably not…

When you think of operations, you think of automation, and rightly so. But AI is a different beast. There are a lot of moving parts to account for. And it is true that much of that change can be automated. Certainly there are ways to dynamically re-train the model, automatically assess and change different machine learning parameters, and control for the software version used. It has been our experience, however, that even with automation in place there are still many instances where the results are not in line with accuracy or rationality expectations and data scientist intervention is required. AI operations require a level of institutionalized curation in order to be successful.

3. AI produces directional and acceptable thresholds of right, not concrete answers

What is right when determining profit? If you sold the product for $30, and it cost $20 to make, the profit is $10. Done. There is nothing to debate and it is easy to quality control. What is the right predicted lift if HCP Digital spending is increased by $2M? Run a straightforward regression and it predicts 1,000 TRx. Run a simulation through Random Forest and it yields 500 TRx. Run a third approach and it says 750 TRx. All three point to a positive lift which gives the overall direction of the outcome. There are also ways to best assess the amount of lift to publish but it is never 100% accurate. We are talking about predictions after all. If the result cannot be verified, how do you determine the “rightness” of the answer? New metrics need to be developed and monitored to manage the determination of success with a focus on acceptable error bands.

4. Managing the code, inputs, hyperparameters, and results is important 

In this post, we have just scratched the surface of the many inputs needed to deliver an AI-driven analytic. All of these inputs together make up what we call a “run” at Sentier and all need to be tracked for every run. When developing an initial model there can be hundreds of iterations and after reviewing the results, the data scientists need to be able to identify which set of inputs is the best. When running a model in production, these inputs are critical if something does not go as expected and adjustments are required. With AI there are many changing inputs and they all need to be systematically versioned to effectively manage operations. 

Conclusion:

At Sentier, we have been constantly refining our version of AI Ops to better deliver analytic services to our clients. With respect to the challenges listed above, we have conquered some, are halfway done with others, and in some cases are just getting started. While models/engines themselves tend to be proprietary, we believe that sharing ideas and approaches to delivering AI benefits everyone. We look forward to collaborating with our clients, leaders in the space, and peers to continue to improve AI operations as our combined knowledge evolves. Please feel free to reach out for a conversation.

----

Sentier is trusted by life sciences companies to provide a data-driven understanding of their promotional and digital efforts. For the past 5 years, we have been delivering disruptive insights through innovative AI-powered services and solutions. Always at the forefront of the most effective data integration and machine learning approaches, we have created unrivaled models. As a result, our clients are better able to optimize their marketing and sales resources, and build stronger customer relationships.

We are committed to the concept of High-Velocity Decision Making and believe that it can only become a reality with a strong and innovative analytics component.

We strive to be the leaders in the actionable application of new and emerging data and analytics approaches and to remove the barriers to our clients of benefitting from these solutions.

We believe that the ethical application of our services will benefit patients and doctors as well as the pharma and biotech companies we serve.