hello and welcome to how to succeed with predictive AI I'm Abby lunberg editor and chief at MIT slan management review and I'll be moderating the event machine learning is the engine of predictive AI yet too many machine learning projects fail at deployment generally that's because they're viewed as technology rather than business projects and organizations often fail to Foster The crucial connection between Business and Technology functions our speaker today is a leading expert in the effective
and profitable use of Predictive Analytics and machine learning Eric seagull is the author of the newly released book just in the last couple of days called the AI Playbook mastering the rare art of machine learning deployment he'll explain what business stakeholders must do to succeed with AI Eric is a consultant in former Columbia University and U UVA Darden professor and in addition to the AI Playbook he's the author of The highly acclaimed Predictive Analytics the power to predict who will
click buy lie or die and he's also the founder of the machine learning week conference welcome Eric thanks Abby thanks everybody for coming I'm very excited to speak to you about the uh uh problem and potential with machine learning deployment where it's routin failing to deploy but um here's what I think is the antidote so first in 20 seconds here's why predictive AI or Predictive Analytics is important business needs prediction prediction requires machine learning and machine learning depends
on data let's put that in Reverse you start with data you give it to machine learning also known as predictive modeling and it generates from that data it learns from the data a predictive model that then produces pred predictions which is why we also refer to these Enterprise use cases of machine learning as Predictive Analytics or predictive AI in that way we in these types of use cases that's where you have the potential to improve almost any large scale existing um operation and so it delive
rs these benefits boost sales we cut costs combat risk prevent fraud fortify Healthcare streamline manufacturing conquer spam and win elections as Morgan vaer put it in the forward to my new book the AI Playbook machine learning's practical deployment represents the Forefront of human progress improving operations with science so in a nutshell predictive modeling machine learning methods learn from data they create what I depict in these slides as a golden egg and the golden eggs are the predict
ive models that are then meant to be deployed to integrate in into existing operations and better Target Marketing sales activity fraud detection Etc changing operations in order to improve them that's the deployment that's the operationalization that's where you actually realize and capture value from the number crunching but routinely they they fail in fact um most new Enterprise machine learning projects fail to re reach deployment and therefore fail to capture value entirely IBM recently cam
e out with re uh industry research results saying that the average Returns on AI projects is lower than the cost of capital and in my own um uh work participating at to help produce a couple um uh industry surveys to data scientists the data scientists turn out say that tell the same story they make the model with the intention of deploying it and then so often it fails to get deployed basically stakeholders get cold feet if they don't get their hands dirty their feet get cold that'll be the the
me today so really we have a major unmat need in the industry which is that there's no established well-known widely adopted Paradigm practice framework playbook for running machine learning projects end to end from con from inception to successful deployment that's well known to business stakeholders in fact in general business stakeholders haven't realized that it takes a very particular specialized practice which I'll be outlining today and secondarily and maybe more fundamental in order to p
articipate end to end in that life cycle of the project the business side must collaborate deeply and first ramp up on a certain Semite technical understanding of what it means to integrate to deploy a predictive model its predictions I'll I'll explain what I mean by exactly what that Semite technical understanding is today so I'll start out by defining machine learning predictive predictive AI briefly but then in the red we've got the problem and the green is the solution right so ml projects a
re routinely failing to deploy I'll cover that semi- technical understanding and then with that understanding the stakeholders could then participate in and and practice Paradigm that I call bis ml business practice for running machine learning projects so let's turn to the definition I'm reading this book machine learning to my son his name's Ka he's he's now much older he's now uh three and a half um that's a real book which I highly recommend for babies but not for toddlers because I really d
idn't like the definition and there's a lot of definitions thrown around around these terms let let's turn to a more concrete definition well first a more informal definition but a step in the right direction is actually the full long title of my first book Predictive Analytics the power to predict predict who will Click by lie or die let's turn to a little bit more concrete actionable um practical definition machine learning and these types of predictive Enterprise use cases it's technology tha
t learns from experience learns from experience okay by experience I mean data data is a collection uh of of uh incidents that have occurred it's a long list of Prior events it encodes the colle Ive experience of an organization from which it's possible to learn to predict to generate a model to predict the outcome or behavior of each individual and by individual I mean humans or otherwise a certain lowlevel granularity for from the organizational perspective customers patients business Vehicles
image equip piece of equipment other individual units transactions that could be fraudulent locations you might drill for oil satellites that might run out of battery any and all individual um outcomes or behaviors on that indiv idual level in order to drive better decisions and that's the rub that's where you're actually acting on it that's the deployment piece so each individual you see at the bottom gets a number usually in the form of a probability it's a predictive score and the higher the
number the higher expected chance that the individual will Click by lie or die commit an act of fraud any outcome or behavior of which there may be uh value for improving operations which generally consist of many individual decisions predictions the holy grail for that so now that I've sort of defined the area and these types of use cases I'd like to ask you all to quickly um respond to a poll and I've broken it down to four options with regard to your background and experience with these type
s of Enterprise machine learning projects uh you've never been involved with a machine learning project you've been involved with one but it didn't deploy which is not uncommon you've been involved with them and most deployed but didn't necessarily show proven value or you've had mostly successes you've been involved most did deploy and most did have have established proven um business value okay so okay this great so the intention today is really to be accessible for anybody who's not been invo
lved with machine learning um and in fact that's what I'm talking about it's a meta discussion what is it that we need we all need to learn in order to participate so that's great that more than half of you have never been involved and indeed of those who have been you see this skew where there's a majority of cases where they Trust but didn't quite get deployed so there's a certain disconnect and that's really what we're going to be talking about today um so thank you Sean if you could go back
to the slides so the main two technical steps of a machine learning project the sort of culminating steps are to create the model to learn from the data that's the number crunching that's the main rocket science and that's what I'm showing on this slide so data is is input into the machine learning software predictive modeling software and it generates this golden egg predictive model and then in the second sort of culminating step um you're deploying them all you're acting on it you're using it
to make predictions and then acting on the on the predictions so technically for each individual whether it's an individual human or a car that might H need need a repair or a train wheel that might be faulty whatever it is on that level of detail you take characteristics of that individual's input this is where you're using what's been learned you're acting on or applying what's been learned the model then takes that information about one individual at a time and then generates the predictive
score for that one individual so those are those are the sort of two culminating technical steps right so the the problem is in getting that second part deployed where you're actually producing those predictions and acting on them right we have this unmet need and here's the rub which is that sort of both sides both data professionals and business professionals the data science scientists the machine learning experts who actually operate the machine learning software and prepare the data for it
and their clients the stakeholder the people in charge of running large scale operations that stand to be improved with predictive model um uh deployment both kind of point to the other side and say this this kind of running or management or business level process is not my job and I'll get I'll clarify where that is but it sort of rests in the No Man's land and this is the last main remaining ingredient before we can get more widescale success in deployment and part of the problem is that what
we're talking about is probabilities and acting systematically over many probabilities because the output what I've been calling a predictive score really is just a probability a value between zero and one or zero and 100 same thing that says How likely is this outcome for this particular individual you can think of a predictive model that's been generated from the data also you could think of it as a probability calculator for that individual for whatever you're trying to predict for this proje
ct so by the way just to be clear so by by predict we really just mean put a probability on I should also be clear that by predict we don't just mean literally predicting the future the alcher behavior that will happen in the future but the word also applies for a situation or a diagnosis does does this health care patient have this diagnosis should they be given a positive diagnosis is this transaction fraudulent you know or is it legitimate it's not exactly the outcome or behavior in the futur
e but the same we use the word predict either way predict whether the transaction is fraudulent so just to be clear on that so here's the problem with um probabilities you know the world's not exactly so friendly and excited to adopt them there's a certain intimidation and and and technicality that seems to shroud them right so let's go to Empire Strikes Back And The Helpful robot C3PO Hey sir you know there's only a one in 329 to one chance that we're going to survive navigating through this as
teroid field and then our hero Han Solo never tell me the odds right and I'm like thanks George Lucas you're not you know you're kind of giving this stuff a bad name so now let's turn to maybe not quite as popular movie but but a very popular movie Moneyball based on the book where the Oakland A's baseball team turned out to do uh much better than anybody expected because they were acting on and successfully deploying the number crunching and the Analytics the probabilities about which which pla
yers to put up to bat first and and and all these decisions that go into uh running a a baseball team during the game um the thing is is that although this movie celebrates the math and its successful deployment it's also the epitome of glossing over the math right so how far have we gotten into sort of bridging this Gap um you know in that between the Beauty and the geek right between the the manager and the data science can you can you guess which actor um played the data scientist in this mov
ie now in terms of sort of more pop science we've got Nate Silver's very famous book signal in the noise and it espouses probabilistic thinking as a general thing but it's mostly focused on more what I would call forecasts where it's one-off predictions which way is this a political election going to go is the economy going to go up or down rather than lots of individual predictions and that's the defining characteristic of these predictive use cases that's why it's so actionable because those p
redictions directly inform each individual case so we need to think not just about probabilistic thinking but a much more rare concept at this point which is probabilistic doing right systematically acting on probabilities repeatedly over many many individual cases um here's another example where probability was misunderstood Nate Silver's prediction of the 200 uh uh 16 presidential campaign right where at the last minute he gave Hillary Clinton a 71% chance and then she lost and everyone say th
e death of data and his model's terrible whatever not at all you know if if Trump had a 29% chance of winning that's far from a long shot that means almost one out of three cases that look like this will have that outcome in fact a 29% is closer to a 50 toss up then a sure lose it it the take away from that particular number 29% is uncertainty right and in that way his model succeeded it said uh can't make a very clear call other models by the way we're saying 99% Clinton maybe those were overco
nfident um but he kind of got um uh chastised unfairly in the media and I think it's from a widespread misunderstanding of Simply what a probability is 7 out of 10 is not definite three out of 10 is not a long shot but let's get a little more specific about what it means to deploy a machine uh a machine learning model for Enterprise use case um this is where I'm going to start telling you about this semi- technical understanding that all Staker holders need to understand and it comes down to Sim
ply three things about how this probability what the probability means and what it's going to do what's predicted how well and what's done about it which to data scientists technically are known as the dependent variable the metrics and the deployment so I'm going to go through a few examples of these just to make it concrete but what we're talking about here isn't the rocket science it's how to make use of how to capitalize the rocket science what you need to understand about how it's going to
change your business so for example one of the probably two main marketing applications of a predictive model is response modeling where you're predicting who's going to buy will the customer buy have contacted for all the examples I'm going to giving it's a yes no prediction a binary prediction will the customer bu contacted yes or no and then what's done about it okay for those above a certain threshold of probability like kind of likely enough to buy then send a brochure and just think about
that for a second I'm not saying they have to be 99% chance of buying if the overall population has a 1% or often it's like a 0.1% chance of responding right most direct man is junk mail it goes right into the recycling container but if you find a pocket that's five times as likely to buy maybe they're only 5% likely to buy you don't have a definite prediction of whether or confident prediction of whether they'll buy but you've tipped the odds so that's what you that's that that's the numbers ga
me you're playing so in this markting example you're looking at an individual customer today saying hey should I expend the cost of contact should I send them a brochure and spend $2 contacting them if I do what's the probability of a positive outcome and I'm going to use that to drive the decision now let's turn for a moment to metrics if you took all of your prospects for this marketing example and you scored them with a predicted model How likely are they to buy if contacted and then order th
e list from from most likely to buy down to least likely that would correspond to the left to right the x axis of this um profit curve so what this curve shows you the the top curve is that as you start contacting those most likely customers your profit is going up pretty steeply because you're relatively speaking getting a lot of responses relative to the expenditure of contacting them then you kind of reach diminishing returns and then the and then the uh curve starts to go down you're only lo
sing money as you continue to contact people because you're just not getting enough responses so obviously when you see a curve like this you might say okay well let's stop at uh 20% we're going to maximize our profit great we're going to only Mark into the top 20% so a couple things about this curve first of all it doesn't tell you absolutely what to do with deployment it it informs the decision it tells the story of what would happen if you happen to be actually contacting them in that order i
n practice you don't actually contact them in that order but by showing in that order it makes you decide where you going to draw that line so for example you might not want to draw the line 20% you might say hey let's go all the way over to 72% where we break even we can effectively Market to 72% for free arguably a lot better than marketing to 100% and costing us $550,000 so it tells the story it turns out that this kind of graph which shows a a business metric a kpi which is the profit of the
marketing campaign is very very rare in practice only technical metrics that that report on the absolute performance of the model how well does it predict so I'll get into some examples of that but um the problem one of the main disconnects is that we need to move to metrics that relate directly to business value like this so the most popular metric that you always hear is about accuracy and it turns out that accuracy is usually impertinent and misleading and I call this the accuracy fallacy an
d there's a lot of overblown headlines that use the word accuracy and imply predictive performance that's infeasible and overblown so here's for example real headlines AI can tell if your gay artificial intelligence predicts sexuality from one photo with startling accuracy linguistic analysis can accurately predict psychosis AI powered scans can identify people at risk of a fatal heart attack almost a decade in advance this scary AI has learned how to pick out criminals by their faces so the pro
blem with these headlines is that they convey that the model can predict with very high confidence for both positive and negative cases and usually be right about it for both types of cases but for these types of outcomes or behaviors especially human behaviors that will require Clairvoyance we don't have Clairvoyance we can't expect computers to have them either so let me break that down a little bit with one of the particular examples the Stanford study that predicts SE ual orientation in thei
r data 93% were straight so if the model just always predicted straight and never predicted gay it would be correct 93% of the time that's a 93% accuracy without ever correctly identifying somebody in the in this case the minority class which is gay but their claim was 91% and out of context it conveyed to Casual readers to lay readers even technical readers who weren't looking at the details of the technical paper that this thing could tell from a photo what your sexual orientation was whereas
no it couldn't so let me let me be a little more specific if you tune the model to use it to correctly identify 2third of the members of the minority group which are gay in this case for this particular example which it could do but only with a really high false positive rates of all the times that actually outputed gay it would actually be wrong half the time and if you wanted to identify more than two3 it would be wrong even more frequent in um frequently so there's this problem when something
happens less than 50% of the time if it's a minority thing that you're trying to predict which is the case for most uh applications that's a hard thing to correctly predict with high confidence the accuracy fallacies per perpetrated over a huge number of cases heart attacks and whether there's a disease in the corn crop and Alzheimer prediction and a million of the examples so I've got an article in the Scientific American Blog the accuracy fallacy by the way you will you're you have access to
the PDF of these slides please take note that when you download them in the top right there's a little comment thing and then you'll see the notes that were in my original PowerPoint file below the slide but they're in the top right little corner click thing and for a lot of these slides they'll you'll get the link to the original article or some additional information including this slide this topic is also the opening of the metrics chapter of my book the AI Playbook so let's talk about this I
've been saying there's these metrics accuracy is actually just a technical metric other other very commonly used ones are Precision recall and something very popular called area under the curve those these are these are the only metrics that data scientists generally work with and and they only tell you the relative absolute performance predictive performance of the model you know relatively compared to say a baseline of random guessing and the fact that it does better than guessing is indicati
ve of potential value but let's also measure the actual value in business terms profit Roi savings numbers of customers acquired or saved this is a move that's absolutely needs to take place and in fact it's the very topic of a new article that just got published on on slow management review yesterday what leaders should know about measuring AI project value so I really get into that in the article um the difference between business and Technical why technical dominates wrongly and we have to al
so be looking at these business metrics so let's look at one particular example with fraud detection and by the way here's here's those two pieces what's predicted and what's done about it to Define this Enterprise use case uh you predict whether a transaction is fraudulent and then if it's likely enough to be fraud well then you're going to either put a hold on it or you're going to audit it right uh there's some action to be taken based on that predicted probability so the problem is you're tr
ying to find this balance between false positivism is false negatives in the case of fraud detection a false positivist says it's fraud but it's not that that's those are two different kinds of Errors right so now the card holder has been inconvenienced that cost something to the bank whereas the other error is probably worse it says it's not fraud but it actually was which means the by the time they realize it's too late the criminal got away with it the bank has to eat the cost of the loot so
I'm just going to skip I'm going to give you a real brief tour of some arithmetic and leave you if you want to dig into the details the takeaway here is that it's just arithmetic it's not rocket science and it's not calculus it's just arithmetic to translate the value of the performance of this model from a technical metric to in this case cost savings um using the fraud detection model for card payment card transaction fraud detection excuse me um so if you had a a medium-sized bank had had iss
ued 100,000 cards and and one out of thousand cases were fraud 0.1% then you would have um it turns out about a $50 million cost to the bank if there's no fraud detection so here's the savings with fraud detection you could actually save $34 million and the calculations I'm not going to step through these in detail but the calculations are basically you get this fraud detection model it does it predicts a lot better than guessing it usually when it predicts fraud it's actually not fraud Fraud's
so infrequent it's very hard to predict it correctly without also incorrectly predicting it a lot of the time but still if it only happens one out of a thousand times and yet almost half the times you predicted it actually is fraud that's a really good rate and if you do the math and add up those false positive and false negative costs it turns out that in aggregate you're going to save uh $16 million um so the cost is going to go from $50 to $34 million if you take into account those two differ
ent kinds of errors and the costs of them so I'll leave it to you if you want to dig in you can also go to this spreadsheet which is the same example and you can also look at the slow management review article that published yesterday same example if you want to see what that math is but my main point here is it's not rocket science not even calculus it's just [Music] arithmetic okay so let's get into so as I mention is what predicted how well that's the metrics what's done about it what's predi
cted and what's done about it that defines the use case should I drill for oil here what's done about it decide whether to drill for oil right uh sh is this credit applicant likely to be um a bad debtor what do I do about it decide whether to actually approve their application for credit right that that what's predicted and what's done about it that's the pair that defin finds the business value proposition and the business use case but the first of those two the prediction goal is where you st
start to actually dig into a good amount of detail so you can't just predict for response modeling will the customer buy of contacted you have to say let's get really precise about what we're predicting for the purposes of this project well okay if Center brochure will the customer buy within 13 business days you have to set a time window with a purchase value at least this amount $125 after shipping and not return the product within for refund within 45 days you have to put in all the qualifier
s and caveats they're business relevant they're pragmatic they can't be informed by a data scientist operating in a vacuum this is where you need collaboration by an understanding from business side stakeholders participating in the project so what's predicted how well what's done about it actually corresponds with the first three of the bis ml Paradigm that I that I lay out in the book and that I'm espousing in general as as what really we need I'm issuing claron call we need a standardized fra
mework Paradigm for businesses to follow and an understanding for a need for one in the first place so those three factors that Define the project and that uh where business stakeholders have to get involved and understand some of the concrete details correspond to the first three steps establish the deployment goal which is that pair what's predicted what's done about it establish the prediction goal which is the first of those two but get a lot more specific about it and then establish which m
etrics pertain and what your standards are for those metrics then the other three are the main three technical steps of any machine learning projects two I've already mentioned prepare the data train the model from that data right learn that golden egg model from the data and then deploy it actually start using it to make predictions and acting on those predictions to improve operations this requires change management right we need to this this deployment means change to existing operations not
just number crunching right now the world focuses on the number crunching the rocket science part this is the awesomest kind of Technology most SC data scientists like me and I got been in in machine learning for more than 30 years totally got into this because that is the coolest kind of Technology learning from data to find Trends and patterns that hold out in general it's really really in in that sense the computer literally learns because it discovers things that are real not just particular
to the data but that hold in general in new unseen situations it's the cool science but right now we focus entirely on that instead of its use right and in fact that's sort of the that's sort of the syndrome students data science students come in and they flock straight to head Hands-On what they want to do is load the data and this is the first thing most books and courses cover load the data and start making a model but no that that skips all those those pre-production steps those business st
eps to establish exactly what you're predicting and why and then that definition in full detail of what you're predicting which is then by the way manifest by the data prep the data is not going to prepare itself so you can't skip over that stuff it sends a false narrative and it has us focusing too much on the technology now data scientists are going to say no no no this isn't my job my job is just to make the model um those are management issues of course the model generates self-evident fact
it's valuable it'll be deployed are they crazy whereas the business professional saying no no no those details what's predicted how well what's done about it I delegate all the details of this project which sound really hairy intimidating to my experts the data scientists so therefore the hose and the faucet are failing to to connect it's so ironic by focusing so much on the on the model the modeling science rather than its deployment it's like being more excited about rocket science the than th
e launch of the rocket that's where we are in the world the business professional also says well well to drive a car I don't need to look under the hood which is true I also have never looked under the hood I don't know how to change a spark plug but to drive you do need expertise momentum friction how a car operates rules of the road expectations of other custo of other drivers and how they expect you to behave um that's a very particular amount of expertise you need that that analogous experti
se to run to drive a machine learning project successfully through to deployment more than anything Enterprise machine learning projects don't need better technology they need you they need you after you've ramped up on the semi- technical understanding to participate deeply in the end to end project that's the missing ingredient it's you and it's I I can't cover all of it in in a half hour you know it's basically books worth that's why why I wrote the AI Playbook book so to wrap up we need to r
eframe machine learning projects right now it's focused it's considered a technical project which consists of these technical steps prep the data train the model and deploy it it's missing the whole other side of the project the business side right we need bis ml or something like that we need everyone to understand that there's a particular framework an end to end practice that we're on the same page and participating collab op ating deeply together from end to end with business side stakeholde
rs who've ramped up so it's not two-sided project it's a first and foremost it's a business project that has a technical component we need we should rename machine learning projects to operations Improvement projects that use machine learning as a a critical technical component it's a business project first so that's these are my conclusions gain a semitech technical understanding what's predicted how well what's done about it you need to get into that level of detail when the stakeholders don't
get their hands dirty in that respect they get cold their feet get cold and that's what happens the project dies because it it fails to get approved for deployment and then with that understanding in place follow this specialized Playbook a practice end to end the whole life cycle needs that deep collaboration so that's what I cover in the book I'll land on this during Q&A because this is what I'd like you to meditate on um I'll also mention that um I've been running this conference machine lea
rning week since 2009 formerly Predictive Analytics world it's first week of June in Phoenix and our we have a new sister conference the a uh uh generative AI world so with that um I'll give it back to Abby and we can have uh some [Music] questions great Eric tons of of great comments and questions coming in from the audience and I'm going to start out with um a couple of questions around you know you've clearly you've clearly articulated that this needs to be a business pro project um and that
there is both the business and the technical um aspects to it that has there has to be a lot of collaboration around that so I guess a couple of questions around you know governance and Leadership um one is so for the folks in the audience who are from the technical side what can they do to help um engage their Business Leaders to if they're if they're not viewing it as a business project or they're not as engaged as they should be what can can the folks from the technical side do to help them b
ecome more engaged well it's easy it would be easy for me to say don't take no for an answer um because I'm not the one in your situation but my answer is H don't take no for an answer so it's absolutely critical so you know and and a related a related question is sort of who does this who's supposed to be the leader of the project I'm agnostic about that depends on organization the way that you know who who conceived of the project and how it evolved and how it emerged and it can come from all
different levels and sides of the organization and there's a lot of stories that came from the executive or came from someone in the middle or came from the data scientist or came from the operations person um it varies like mad but the fact is um what what I'm advocating for is to follow an understood collaborative business practice that's end to end from conception to deployment um and if it's not being filed that's a big problem it's sort of if it's if you're following it you're doing the rig
ht proper planning from the get-go of exactly how operations are going to change and that's the missing ingredient somebody's got to be running the project in that way with that kind of structure so if not you and why not you by the way think about that but if not you you got to make sure it's happening so one of the stories I tell in the book is you know when I was uh early in my Consulting career and I've been an independent consultant for 20 years now um I had a online dating uh business and
I got them to hire me to do and they hired me to do some number crunching and some a Reconciliation and then I was like let me do churn modeling because we can predict which of your paying premium level customers are going to cancel and they're like oh that sounds like a great idea and they were doing well they had a lot of cash so I got paid and and then when I uh showed them the model in a PowerPoint and in this sense the power is stuck in PowerPoint right and I'm like look this is what could
happen when you deploy it and they're like oh cool that's that sounds interesting and I'm like so you going to do it they're like are we going to do what I'm like you know you gonna so was that that was the conversation this is what happens over and over again they're like you have to you want us to start a whole new operation in this case a new marketing campaign a retention marketing campaign by predicting which customers are going to cancel um so there was this disconnect and so I was at faul
t because I even though I'm only the data scientist I'm the one who sold the project in the first place right and I hadn't anticipated um you know what it would take to convince decision makers to actually make an operational change and in this case a whole new initiative um and what's worse than that is I didn't learn my lesson my next main project that I had worked really well but not because of me because the people who hired me already had a really good plan for how it would be deployed I go
t lucky in that regard it took me a while to learn this lesson this is and this is so there's there's inertia right people are fixated on this awesome technology I'm a former academic and at that point as a green consultant I was in love with the technology and sure I still am but now I'm more in love with getting it actually deployed great yeah and you did also just answer uh Peter's question so um we we'll so there's there's a couple there are a couple questions coming in around value um you y
ou defined I think you clearly laid out that everyone's got to Define their own value what is it that you're trying to predict and then how do you measure it so there's a couple questions one is from um sangan asks says the comment is it's hard to evaluate business values immediately after deploying AI so what do you mean by proven and how long does it usually take to get there like what's what's the right sort of time frame for measuring the value that's a that's a great question I mean it depe
nds on the operation right if you're doing uh if you're changing the targeting of ads it could take hours right before you know how many people are click clicking on ads um so that could be really fast uh if you deploy a a campaign a direct mail campaign that's better targeted with a machine learning model you know depending on the context maybe a few weeks or a few months um you have to find out and give them time to actually respond the customers to respond and then see how it goes so there's
a sense where you have to wait until after deployment to see how well it worked but the practice is of course to stress test the model and give your best estimate of how valuable it could and should be to forecast its potential value before you decide whether and how to deploy it um and in that regard we need to already move to those business metrics what matters for a marketing campaign the overall profit of the campaign that's often one of the measures profit Roi number of of customers saved n
umber of dollars saved for fraud detection these are really really obvious things this speaks this is the lingua franka of of business and it's unfortunately strangely still not the language of data science although in in that survey I mentioned um uh it turns out data scientists generally understand this right they when you ask them what's the most important metric they start with those business metrics like profit and Roi in their lists but then when you ask them which are the most common metr
ics that you actually use they say technical metrics it's partly inertia it's partly um uh cultural but it's also a lack of sort of common practice and a lack of Technology I'm actually co-founding a company called good or AI to actually address that so that we can be evaluating models with those business ter with those business metrics in addition to uh technical metrics so there's what happens inside companies and then there's what happens within the ecosystem of vendors and and Consulting com
pan companies as well and ESR asks how do you see platforms and products evolving and on the other side how are organizations or service providers maturing in terms of framework methodology that weaves together the imperatives that you're talking about um with their products and platforms so it's kind of a market question um yeah I got sorry I'm not I'm not quite getting the gist of the question could you read again I mean yeah um so so the first part was about how how do you see platforms and p
roducts evolving but then the this I think the meat of the question is how are organizations or service providers maturing in terms of framework and methodology that weaves business imperatives into their products and platforms okay thank you yeah yeah sorry I just couldn't parse it the first time so a lot in there yeah certainly I'd rather answer the second question first because my focus isn't on platforms and products my message my claron call today is to be focusing on business process this
is an organizational problem first and it's an organizational project first and a technical project second yes there are technical problems uh if you want to integrate I mean if you've got things serving ads in real time or making fraud uh detection decisions as far as whether to authorize a card transaction in real time that operational system now has to actually get changed based on the score output by a model so in real time it needs to evaluate the model there and then and then make some dec
ision based on the output of the model so there's real um uh infrastructure issues and deploy you know um structural issues but again I I see those as the secondary issue to the broader like the the the dog that Wags The Tail has got to be the organizational point and the practice the the the project management practice because if you're following it then you'll properly plan if there's a deployment stall because there's a lack of the right technology it's because it wasn't foreseen and planned
for from the get-go so that's the advantage of starting very clearly not just with the general business objective and most people already know that yes you have to start with uh you know your objective is to retain more customers or decrease the cost of of of uncaught fraud or whatever the objective is that's great but that's literally only the first of six steps that I outlin in the bis ml framework um you then have to go into the detail to the end of that including exactly what's predicted and
what's going to be done about it and how is that going to be operationalized so let me put it this way the book's six main chapters outline those six steps you can't just read step chapter one and be ready to do step one you also have to read chapter six to understand what deployment fully entails because it's those nitty-gritty details you need to jump into from the beginning in terms of how is this going to deploy what's it going to change do we have the right Engineers so we have to plan for
that so in terms of that second question of how are organizations service providers Evol evolving in terms of a framework uh so far not that great but hey the book just just published on Tuesday this is why I wrote the book right because and this and maybe more importantly this is why I coined the wonderful Buzz word bis ml Five magic letters business practice for running machine learning projects because there is no well branded notion that's out there and well known to business stakeholders a
bout even the need for a particular uh business practice so that's why I'm trying to get out there I would say no it's not going well we this is this is the change I'm I'm asking the world for because routinely especially outside of big Tech and some very particular leaders projects are failing to deploy so back to um inside the organization and keying off of something that you just said about the project management piece Jessica asks product managers sit between Business and Technology how have
you seen this role influence ml projects do you find that including product managers increases success with that skill set um yeah I would say that uh learning the semi- technical getting Semite technical understanding U is necessary I'm not claiming it's sufficient right I don't know if there's I don't know if there's really a a Holy Grail or a Panacea but in the very least you need this um in terms of sitting between Business and Technology pro pro uh product manager for business analytics pr
ojects in particular sometimes it's called an analytics TR uh translator there's a few other kind of relatively new kind of role description and names um arose By Any Other Name we need this translation to happen whether it's a third person sitting in the middle or it's just two main people it's the Deep collaboration right so I I I find it difficult to prescribe exactly the structure of the humans uh because it varies so much and the needs vary so much from organization to organization project
to project whereas the process Remains the Same or at least the required uh fundamentals of the process if you're going to get that there does tend to be a disconnect because there does tend to be sort of two types data and business person there's there is a Continuum between the two but it's it's very much a bodal distribution right even if everybody in the company is super technical and has a master's degree in machine learning um the fact is when you're executing on the project if you're in t
he weeds if you're actually doing the data prep and modeling it's very hard to be doing that and also keeping in in your keeping seeing the force for the trees and keeping perspective very very hard there's a reason we collaborate and C create these things called companies which are groups of or people instead of operating in a vacuum alone we need to collaborate you need to have people in different roles so the roles are intrinsically going to have sort of data and business sides to one degree
or another what's at stake here is the lack of a connection between people in those roles and the need to collaborate deeply not on the core rocket science not necess I mean it's great to understand about um internal internal combustion and I know the principle of internal combustion even though I don't know where the spark plug is in my car um so you don't need to get into the weeds of the algorithm of how it learns from data but the general principles are there more importantly you need to kno
w how to operate the vehicle you need to know those um semi- Technical pragmatics and get involved in those at a level detail for the each particular project so let's build on that for a bit what what's the best way for someone to ramp up so that they can participate in ml projects so so what's the first step where where do they start and I know they can read your book um and learn a lot there but but really what was break it down to that where to get started okay I can't say read my book well I
just said that take my course no um uh you know I I I um I just got interviewed uh about one of the five books I recommended machine learning there's an article that just launched I think this week and they told me I couldn't recommend my own book I was like oh right I'll do it anyway um two of the uh two of the books on that on that list are in the same category in general there's far and few between books and material and courses and content on this level this is this is a missing ingredient
because it's so much more comfortable for a data scientist to focus just on that number crunching part and it's so much more comfortable for business people to sort of operate at a higher level of abstraction which I kind of call you know buzzword city um uh but we need to bridge that Chasm so you know it's and the same thing with courses that I'm familiar with it's few and far between they get into that that unted un Uncharted Territory which is some where in the middle right I mean what I'm re
ferring to is Semite technical um to business stakeholders it may at first seem super technical and to data scientists it'll seem like high school algebra they'll be like what whatever that's not technical why are you even using the word technical um there's a spectrum and the spectrum is really wide or long and so there's this middle area both sides need to move towards the middle so I wish that I had um more resources and I also to point you towards and I also wish there were more successes so
assuming that someone is coming from the business side and uh was trying to get that understanding working with their colleagues in the in the organization um the question is how how can I convince data scientists to explain things more clearly like what is that there there's a translation Challenge in a lot of organizations so what can someone from the business side do to gain that understanding with their technical colleagues you know um another way to sort of tell that story is the data scie
ntist says hey look I made the model that you wanted to predict misinformation or fraud or whatever it is it has look how good it is has an area under the receiver operating characteristic curve of 089 isn't that exciting and then the stakeholder is like what you know can you translate that to English oh it means it's predicting a lot better than guessing oh okay so but how much value is it going to generate if I actually deploy it if I change operations according to it data scientists are not p
repared to answer that question um but if you point them to yesterday's slow management review article where I'm like look it's just the and the the article you have to go to a sidebar that pops open at the very bottom but it gets into that same arithmetic I flashed on your screen today the data scientist is going to understand that someone needs to hold their hand and be like please would you now now you can just translate this it's not it's not impossible it it's just a willingness and and a c
hange to culture um they need to be convinced no technical metrics are not sufficient you need to also provide an estimate of business value in business [Music] terms so there are a couple questions coming in around data and one of them is from Raquel asks Eric what is your view on using synthetic data to train predictive models that's a you know I need to ask if anybody can help me because I I've seen this I haven't spent a lot of time on it but everything that I've seen about synthetic data I'
m like huh I don't somebody needs to explain to me the synthetic data could think because the only thing that could be generating the data is a model so if you already have a model how could the model that you generate from the data be any better than the model you had in the first place so I don't get it that's where I'm at maybe there's something I'm missing if anybody find me on LinkedIn tell me what I'm missing point me to the article that that that'll explain to me the the value of syntheti
c data I mean data is expensive especially when you have to manually lb label it like for is there a traffic light in this picture uh very expensive and also bad labor conditions for the Labor uh the labelers in many cases right so that's a question a lot of people say the solution is synthetic data um because it can be generated automatically I'm like if it can be generated automatically you don't need to model on it so I don't get it yet I think there might be something I'm missing so uh we ha
ve time for just one or two more short questions Eric another data question I thought that was funny I didn't actually capture it but I remember it the question was around can you use the AI to clean your data so you know we know we need a lot of data to train models can can the model also help to make sure that that data is the right data and it's it's accurate um probably in some situations there there there already is uh an intrinsically circular logic problem just the same as with synthetic
data there in a way right um but let me let's just take a step back the word noise can mean two different things it can mean in uncertainty or outright incorrect values and the inputs to the model um only help predict so well right we don't have the ability to predict perfectly it sort of goes to Chaos Theory you can't predict whether somebody's going to click by LI or die with really high confidence but you can predict a lot better than guessing and that translates into business value um and wh
at that means is that you can s you can there sort of feels like there's this Randomness to to the to the values and the way they associate with the outcomes um but that Randomness even if the data is totally sound and there's no errors the randomness is because we have limited knowledge as humans and always will so what appears to random is just is this uncertainty that comes out of a a lack of complete understanding and knowledge um from that respect there's not really a big difference between
noise in the sense of incorrect value and just normal uncertainty and um a lack of understanding in general and because of that machine learning methods are really robust to noise if the noise I'm talking about is in the input variables the things you're predicting from but as far as the thing that you're predicting the output of the model What's called the dependent variable in the training data you know who's going to cancel whatever the thing is you're trying to predict um that's where you c
an't have noise and the model's value and and reliability is going to completely hinge on I'm Excuse me that's where you can't have errors so you need to make sure that that data is sound and that is just pragmatic it's reality it's empir this is empirical science and that's the empirical part if it's wrong it's wrong and but it's it needs to be ground truth or there is no truth that you have access to so there's no way for another model to to determine that in general now there may be some crea
tive uses of generative AI where you kind of say hey look at this distribution you know can you can you tell me a probability that this is noise and it may help Target um human activities that investigate where there may be um bad values something like that well you just heat up the final question quite nicely um we we can't have any sessions on AI these days without talking about generative Ai and um Heather asks how does this Paradigm change when you layer in generative AI while it is grounded
in classic machine learning the output is very different it is although you can also use you can use generative I and we've done some studies on this of um to as a predictive model just the same it's like what are the chances this statement is misinformation you can basically in one way or another ask at that um uh and it may perform even better because it's a language heavy task but more generally um generative AI where let's say you're using it to create first drafts of writing or of code um
very different kind of um apples and oranges uh but broadly the same kind of framework needs to apply you need to reverse plan the antidote to Hype is focusing on concrete use case and concrete value you need to plan from the Geto exactly how it's going to operationalize which practice which process uh you know such as writing a 100 letters a day to customers need could stand to be improved you need to evaluate it with pertinent metrics how well does this particular language model work with thes
e employees doing this particular task how often does it create new errors you have to proofread everything as a human human in the loop um so there is generally the same kind of theme but BML is very much about the predictive use cases predictive is where you stand to improve ex most of your existing large scale operations versus generative where you're creating basically first drafts or writing or or images and videos and this kind of things um very different they both build on machine learnin
g which learns to predict right in this case you're predicting for each what word should I write next or what token really um or how should I change this pixel while I'm the process of uh generating this image um so the core technology of learning from data to predict very much the same thing but just use in a very different way um well that's great Eric yeah yeah um this has been such a great session so much good information and of course lots more in the book which just published this week um
so thank you so much Eric for being here with us my pleasure thanks Abby thanks very much for having me thanks everyone for attending thanks so much
Comments