5 Steps to Maximize Business Impact with Machine Learning

By cnvrg.io team

One of the most enjoyable feelings I experience as a Data Scientist is watching the error rates fall as I work on my modeling projects. The iterative process of increasing the accuracy of a classification algorithm through feature engineering or driving down the mean squared error of a regression model through residual analysis is intensely satisfying. One can get lost, as I often do, as the hours and days pass in pursuit of the “perfect” model.

As a team leader and entrepreneur, though, I’m acutely aware that maximizing business value is of supreme importance. When I step back from my keyboard and close the Jupyter notebook, I force myself to think about whether my instincts to try a new modeling approach or run a new experiment are intended to improve business outcomes or to satisfy my ego. But simply asking yourself whether you should run that new experiment when you’re in the middle of a project is not enough. In order to maximize the business impact of machine learning efforts, you need a process that ensures technical work is aligned with business objectives in each of the project ideation, planning, and implementation stages.

Here are 5 steps you can take to ensure that your machine learning efforts are properly aligned with the business outcomes you wish to achieve.

Using a Machine Learning Platform for Business Growth

1. Determine the desired business outcomes

Before beginning any technical work, it’s imperative that you have a clear understanding of what the business wishes to achieve. One way to think about this is to imagine how the business would be different after the successful implementation of a project. Does the business wish to reduce customer churn? Is a department seeking to eliminate a manual task that keeps employees tied up? Is the Product team seeking to increase engagement in the mobile app to drive purchases? These are all examples of business outcomes that can be achieved through the application of machine learning.

Achieving these results begins with clearly stating the outcomes. Neglecting this step, say by rushing into the model building process, can lead to a great deal of time, effort, and resources put into building solutions that address the wrong questions. Clearly defining the desired business goals at the outset of a machine learning project provides a north star to follow.

2. Define the success criteria

Once the desired business outcomes are clearly stated, you need to define the measures of success that will track your progress towards those goals. These metrics will guarantee that your team’s work is moving the business towards achieving its goals. The best metrics are specific and measurable. Specific metrics reduce ambiguity and increase focus. And, the easier it is to measure your success, the more certain you can be that you’re achieving the results you set out for in the first place.

Suppose the business intends to reduce customer churn. Specific and measurable success criteria include a clear definition of churn and the desired threshold. For instance, a subscription service may define churn as the percentage of customers that cancel their subscription service within 3 months of signing up. A specific and measurable success criteria would be to reduce this number to 5% of all new customers within the next year.

As another example, consider a company whose operations team performs a significant amount of manual data entry. Individuals on this team are able to process approximately 20 documents per day with the current system. The business is determined to cut costs and would like to increase the number of documents processed per employee to 30 documents per day. The specific business metric is the average number of documents processed per day by employee.

While specific and measurable criteria are ideal, it may sometimes be necessary to settle for more subjective criteria. For instance, a business may desire to improve employee or customer satisfaction but it might be difficult to pin this to a specific set of metrics. In this case, you should explicitly state who is going to make this judgment call.

3. Translate the business success criteria into a machine learning metric

After defining the business objectives and success criteria, it’s time to define the successful outcome in technical terms. For machine learning projects, this means selecting and optimizing an appropriate metric that directly relates to the business metrics chosen in the previous step. Machine learning models are trained by optimizing towards metrics such as accuracy, area under the ROC curve (AUROC), or root mean squared error (RMSE). By selecting metrics that map to business outcomes, data scientists can optimize directly towards the goals the business wishes to achieve. Choosing inappropriate metrics that aren’t tied to business outcomes leads to project failure and business losses.

It’s important to note that selecting an out-of-the-box metric such as accuracy or AUROC may not be good enough. For instance, optimizing towards these metrics when facing a class imbalanced dataset can lead to suboptimal models. This situation often occurs in areas such as fraud detection where the number of examples of fraudulent activity is significantly smaller than the number of non-fraudulent instances. Selecting the wrong metric in this instance can lead to a model that looks great from the perspective of accuracy but still misses the most costly transactions. The importance of selecting machine learning metrics that directly relate to business value cannot be understated.

4. Establish a baseline

Regardless of the metric you decide to optimize, it’s necessary to establish a baseline measure of performance. This baseline provides a point of comparison that enables you to track your progress. It also allows you to judge the rate of return you’ll get by increasing the complexity of your modeling solution. Suppose you work for a real estate firm and are asked to build a model to predict the price of a house. You decide to optimize for RMSE and build a linear regression model with features including the square footage of the house, the number of bedrooms and bathrooms, and other information. While this linear model may not lead to the lowest RMSE, it does have some attractive qualities. The coefficients of the model can be used to explain the importance of the input features and the functional form is very easy to explain. If building more complex models leads to only marginal improvements in RMSE, you may even decide to deploy the simpler linear model because of its explainability and simplicity.

5. Deploy an MVP quickly and begin iterating immediately

In his Rules of Machine Learning, Google Research Scientist Martin Zinkevich describes best practices for building machine learning products. Early on in a product’s development, Zinkevich stresses that developers should “Keep the first model simple and get the infrastructure right” (Rule #4). Deprioritizing modeling gains allows machine learning engineers to sort out the numerous infrastructure issues that need to be solved before the benefits of machine learning predictive models can be fully realized. The simple model provides baseline metrics and behavior to compare against and a reliable pipeline gives you the ability to test more complex models.

Zinkevich isn’t the only heavy hitter to emphasize this iterative approach to building machine learning products. Peter Skomoroch, CEO of SkipFlag and former lead data scientist at LinkedIn and AOL, recently stated that developing machine learning algorithms without user feedback is risky and can lead to unintended consequences. Instead, teams should “ship a complete MVP in production ASAP, benchmark, and iterate”. Andrej Karpathy’s advice for picking a first model is “Don’t be a hero.” The Director of AI at Tesla says that in the early stages of a product, developers should copy and paste the simplest architecture they think will solve the problem. “You’re allowed to do something more custom later and beat this”, he continues.

Conclusion

In order for machine learning projects to have a massive business impact, practitioners need to be relentlessly focused on asking the right questions. Rather than dive directly into technical details and implementation, data scientists must ensure that they understand the business problem as specifically as possible. This involves speaking with stakeholders and translating their needs into technical requirements. Once that’s done, data scientists should work to integrate their work into products as soon as possible and get feedback from customers. This feedback loop will guarantee that data scientists are optimizing towards the success of their business rather than meaningless vanity metrics.

Clearly defining business goals, choosing appropriate measures of success, and deploying solutions iteratively are each important steps towards building great machine learning products. But machine learning solutions have a host of other requirements, many of which are different from the requirements of traditional software application development. For instance, whereas the marginal cost of software approaches zero over time, machine learning models require additional investments in orchestration to ensure model performance doesn’t degrade below acceptable thresholds.

About the author:

Luigi Patruno is an expert data scientist and entrepreneur focused on helping companies utilize machine learning to create competitive advantages for their business. He has built and deployed thousands of machine learning models at Fortune 500 companies, startups, and academia and has taught graduate level courses in machine learning and data engineering. As the founder of MLinProduction.com, Luigi creates and curates content with the goal of educating data scientists, data analysts, and machine learning engineers about best practices for running resilient machine learning systems in production. If you’re interested in learning about how to build, deploy, and run production machine learning systems, sign up to follow his blog at MLinProduction.com.