Using and leveraging artificial intelligence is essential today, but C-suite executives want to know what value AI will bring to their organizations, and how to maximize return on investment. As organizations expand their use of generative AI and large language models, the pay-off can range from increased productivity to higher revenues. We dive into the nuts and bolts of how to measure ROI from AI, monetize the benefits, and make these investments pay off.
Category
🤖
TechTranscript
00:00Hello and welcome.
00:01I'm Andrew Nuska, the Editorial Director
00:03of Fortune's long-running Brainstorm event
00:06series and community.
00:07Since 2001, we've convened the technologists, executives,
00:11and investors for a cross-disciplinary conversation
00:15about the future, and that's what we're here to do today.
00:18So thank you all for being here.
00:19I know it's very early and very late for some of you.
00:22This is the second of two lead-up events
00:24to our Fortune Brainstorm AI conference
00:27in December in San Francisco, with Accenture
00:30as our Brainstorm AI founding partner.
00:33So a special thanks to our corporate partner
00:35for this conversation, again, Accenture,
00:37Brainstorm AI's founding partner,
00:39and to Moksit Ashraf, Accenture's Group Chief
00:42Executive for Strategy.
00:44Also speaking today on today's panel,
00:46we have in the blue corner, Liana Bailey-Crimmins,
00:50the State Chief Information Officer and Director
00:52of the California Department of Technology
00:55for the state of California.
00:57Thank you for being here, Liana.
00:59Renata Marquez, Chief Information Officer
01:01for Latin America at Natura and Company.
01:04Ken Washington, Senior Vice President
01:07and Chief Technology and Innovation Officer
01:09at Medtronic.
01:11We've got a fantastic panel, as you
01:12can see, for this important conversation.
01:14A one-note of housekeeping before we begin.
01:17Please use the chat.
01:19We are involving you as much as we're
01:21involving ourselves in this conversation.
01:23So offer your thoughts, your comments, your reactions,
01:26and especially your questions.
01:27And if I could urge you, don't wait until the end
01:30or wait until the time is right.
01:32When you are inspired, when you are intrigued, please weigh in,
01:36and I will funnel all of your thoughts into the conversation.
01:39All right, let's begin.
01:41Moksit, I'm going to start with you.
01:43We're here to talk about generative AI
01:45and the ROI involved.
01:47I just got back from a dinner in San Francisco,
01:49a fortune dinner in San Francisco with a bunch of CIOs.
01:52And everybody was talking about how
01:53to justify all of this spending around AI.
01:57So before we even get there, I want to ask you,
02:00where do you begin?
02:03If you want to start using AI in your organization,
02:07what is the structure, the scaffolding of thought
02:10you should take to answer that question?
02:14Absolutely, Andrew.
02:15And thanks for having me.
02:16And a pleasure to be with you, with Ken, Liana, and Renata.
02:20Looking forward to the dialogue.
02:22I think in terms of where you begin,
02:25the first thing is just taking stock of where we are today.
02:29I think by all measures, we have traversed
02:33a large part of the hype cycle.
02:34And there is a lot of discussion around, show me the money.
02:38And that's evident in some of the earnings falls
02:42that we've seen over the last couple of quarters
02:44where the focus has shifted from just the pure usage of how
02:48much AI I'm doing to what is the ROI that's
02:51getting generated.
02:52And I think that was very evident on many of the calls.
02:56So in terms of maybe another state of the play, if you will,
03:05is that as we look at companies and where
03:08they are in their journey, we find that 90% of the companies
03:13are doing something with AI or now generative AI, which
03:16AI is, of course, a continuum and Gen AI is one part of it.
03:20But we only find 20% to 30% of the companies
03:23are systematically building capabilities,
03:25whether they are things like centers of excellence
03:28to develop use cases or build competencies.
03:32But only less than 20% are actually scaling those cases
03:37or plays at an enterprise level.
03:40And at the same time, what we're finding
03:42is that less than 20% of the companies
03:44are getting anywhere close to the targeted value
03:47that they are seeking from their AI investments
03:52or from the use cases.
03:54So where do you begin?
03:55I mean, our view is that first and foremost,
03:59you have to lead with value.
04:03In other words, decisions should be
04:06made on the basis of the must-win business challenges
04:10and strategic impeditives and how
04:14AI can address those challenges.
04:18The approach, the prevalent approach
04:20has been more a hammer looking for a nail approach, which
04:24is what can I do for me and then picking
04:27a set of disparate use cases.
04:30And so that is leading to a lot of value leakage, if you will.
04:37So to give you an example of the types of business
04:39challenges, if I'm a pharma company,
04:41I'm looking at compressing my drug discovery time.
04:44If I'm a retail company, I'm looking
04:45at enhancing the in-store customer experience
04:50or management of the shelves.
04:51If I'm an asset intensive company,
04:54like with an energy or utility that I'm
04:56looking at managing my assets better
04:58or driving my capital programs better.
05:00So lead with value.
05:02The second aspect is you have to think big.
05:06As you're setting up the framework.
05:09But you also have to act fast.
05:12Companies that are consistently, we find at least,
05:14are delivering higher ROI are the ones
05:18that are able to scale their AI cases.
05:23And they know the pathway to making
05:26AI big in that particular application,
05:30big in their organization.
05:31So I obviously talked about some of the industry
05:34specific opportunities.
05:37And there are many across different sectors.
05:39I touched on only a few.
05:41But there are a few opportunities
05:43that are applicable across many industries
05:46and are actually truly getting traction, especially
05:48in the world of generative AI.
05:50So I'll give you a few examples.
05:51Knowledge management and retrieval is one.
05:54Content generation for marketing and even areas
05:57like legal is another.
05:59Customer experience and contact centers is a third one.
06:02Technology and software lifecycle development,
06:05compressing that time frame is another big application.
06:08So think of those plays as a second imperative.
06:11And how do you scale them?
06:12Now, if I understand you correctly, just to clarify,
06:16think big, you're talking about impact on the organization
06:19and not so much cost, OK?
06:21Because I'm feeling the CFOs on the call getting very worried
06:25about the terms, think big.
06:27Yeah, but think big, but start small.
06:29Because you can say start small.
06:32But know the journey to becoming big, the journey to scaling.
06:36You should be clear, right?
06:37There are uncertainties, especially
06:38when you talk about the specialized plays.
06:40And I know it's in vogue to actually talk about the big
06:45things that Gen AI is going to solve for,
06:48including the ones that I referenced,
06:49like drug discovery and capital programs.
06:51But the pathway to getting there is still
06:53a little bit uncertain, right?
06:55The data models and the domain-specific models
06:59are yet to be developed in their entirety.
07:02A couple of other points I would say is,
07:06and this one I feel is very important.
07:09It's not just about the tech.
07:10It's about the transformation, right?
07:13Technology is an enabler.
07:14So as you're thinking about it and you're setting up
07:16your organization, think of technology as an enabler.
07:19But the value is really unlocked when
07:21you're reimagining or reinventing functions
07:24or processes or ways of working.
07:26So as you're thinking about directing your investment,
07:29you have to be very clear as to what
07:32the nature of the change that you're
07:34seeking to drive through AI, right?
07:36And one thing I point out often is
07:38that the technology cost often people
07:41think is the largest part of the equation.
07:43It's about 30% in what we have seen and done.
07:48The rest of the 70% resides in things
07:50like building the right capabilities, training,
07:54change management, and the rest.
07:57A fourth aspect is, and it's yet another critical one,
08:01is to invest in enhancing trust and responsibility.
08:05There is obviously enhanced risk with generative AI
08:08in particular.
08:09And we have seen more regulation.
08:10Regulations have been talked about since 2016 in this space,
08:13but they have really picked up steam.
08:15But there is also a growing sense within the workforce
08:20that there could be more harm done in some cases.
08:23So the intrinsic trust in the organization
08:26has to be built. And what we find
08:28is less than 2% of the companies are
08:30investing in a holistic, fully operationalized,
08:33responsible AI program.
08:35And the last one is, I would say, top down, senior,
08:40or call it executive sponsorship for what
08:43you're going to invest in and how you're going to run
08:45the program is important.
08:47We actually did, we looked at the companies
08:49we have worked with.
08:50And for those where there was CEO level sponsorship,
08:54the ROI has been about 2 and 1⁄2x versus the ones
08:57that did not have.
08:58So those are a few.
09:00That's true in everything.
09:01If you don't have top level support for any big initiative,
09:04you're not going to get very far.
09:05The difference that I found that we've seen with this
09:08is there's so much hype around AI and generative AI
09:13that it could lead to all kinds of confusion
09:16and inappropriate allocation.
09:19So it's really important to also include in your recipe
09:23to get started an education campaign about what
09:26is and isn't possible and what you should and shouldn't do.
09:30So the responsible application of this technology
09:33is critically important and also clarity
09:35around what really this technology is
09:39and what it means to your business.
09:41So we took a very early effort to do that education
09:47and to adopt some principles around responsible use.
09:50We called it our AI compass.
09:52But we leveraged the hype because the hype was there.
09:56We leveraged it to create a campaign that we called
10:00hyperautomation that included generative AI
10:02and other technologies like robotics, process automation,
10:06and process mining, and traditional AI as well
10:09to lead to the kind of ROI that matters,
10:12like direct cost savings, cost avoidance, and revenue
10:15growth on our products.
10:16So this is a really exciting time,
10:19but you really have to avoid the potential pitfalls
10:22right up front because there's so much uncertainty.
10:24And you're creating new things with this
10:29because there's a ton of hype and there's
10:32a lot of potential confusion.
10:33I'm glad you came back today, Ken.
10:35That's why I often talk about AI as a continuum, right?
10:37You have diagnostic AI, you have predictive AI, generative AI.
10:41And so while generative AI is perhaps
10:44the most transformational aspect of it, to your point,
10:48you have to understand what are the different applications
10:51where you need the different forms of AI to be in play.
10:54Yeah, I want to bring Renata into this conversation
10:56because I know Netra has been working
10:58with AI in various forms for quite a long time.
11:00And you had to have these considerations.
11:04I don't know when, but you had to take these considerations
11:07into account when you were rolling it
11:08across the organization.
11:09So could you share with us a little bit
11:11about that decision stack?
11:14And my colleague has already shared very good insights.
11:19I agree that, of course, it started with all this literacy
11:23because of the hyper.
11:24So now everybody's experts in AI.
11:28And of course, as an IT organization,
11:32I have the responsibility to multiply the knowledge
11:36and federate all this capacity of the organization
11:41to generate value through AI, but on the correct way.
11:45But at Natura, Natura, for who doesn't know the company,
11:50Natura is a well-being well company.
11:53So you needed to be well with yourself
11:56to spread wellness around the world.
11:59So we are a cosmetic company.
12:01We have the banking pillar.
12:03We have another platform for the artists.
12:09So we have direct sales.
12:12So direct sales is 3.5 millions of beauty consultants
12:18in Latin America, stores as well, and e-commerce.
12:23So I'm just giving the context of what
12:27are the opportunities and risks of not using AI
12:32or using what AI can leverage for us.
12:35So then when you talk about the beauty consultants,
12:39it's a relationship-y business.
12:42So they connect with their customers
12:45using their charisma or their empathy.
12:49So how do we create a more exponential human being
12:56capability through AI?
12:59So I used to say that the AI is the amplified intelligence.
13:04It's not to replace the beauty consultants,
13:06but they can be more connected with their customers
13:12through AI.
13:13So what we did here is first to divide this AI,
13:18define a strategy.
13:19I think that's important to think big, as Ken mentioned
13:24and was mentioned before, and define this strategy,
13:29dividing two hemispheres.
13:31One hemisphere is the impact to the customers or beauty
13:35consultants, our stakeholders, and the internal impacts.
13:40And then we have also two sides, so two hemispheres
13:45and two sides.
13:46And one side is that they change the game AI
13:50and what is the everyday AI.
13:52So everyday AI is related to how we can be more productive, how
14:00we can do faster, how we can operate in different ways.
14:06But it's an everyday thing that we do,
14:08but we'll be doing differently.
14:10And changing the game could be a new avenue of impact
14:15to our customer, to our consumer, to the environment,
14:20to the production.
14:22So that's the change the game.
14:24So we put this together, and then we
14:27start to think big and think about all the possibilities.
14:31And based on that, we start, OK, so let's think big
14:36and start small.
14:37And then started to develop all these.
14:40And it's more the change game is the generative AI.
14:46And AI, we are doing this for more than decades.
14:50But now I think that with this literacy to the business,
14:54we have more people adopting and using AI to do their job.
15:00So that's the change.
15:02But first, it started with this strategy.
15:05And the strategy means all the data foundation,
15:08because without the data foundation and governance,
15:12we would not do that.
15:13So foundation, governance, data for all,
15:16that's our program to spread knowledge.
15:20And then Natura AI, that's all the AI portfolio
15:26that we are managing and implementing
15:30in each different aspect.
15:33And of course, measuring results.
15:35Without the business results, we cannot.
15:38It's just a lab, right?
15:39And we are not a lab.
15:41Right, right.
15:43There's a few points that you raised,
15:44and out of that I want to come back to.
15:46You're the first person to say data, so you get the prize.
15:50We're going to come back to that because it's going to come up.
15:52But I want to bring Liana in here.
15:54Your organization is a little bit different
15:56than everybody else's in terms of,
16:00you know, I'm still stuck on the think big
16:02and act small part of things, Liana.
16:05So tell us how the state of California
16:07is thinking about this.
16:09Thank you, Andrew, and thank you, panelists, for a very,
16:14I was really resonating with Renata.
16:15So as we're talking about return on investment,
16:18obviously in a public and private sector,
16:20there's a lot more alignment.
16:22We think of financial like cost savings, cost avoidance,
16:25but also the human ROI.
16:27Especially being in government,
16:28we need to think about what we're doing to help residents
16:32when it comes to efficiency, helping our workforce,
16:35ensuring that the emerging technologies that we leverage
16:38are actually assisting the workforce to be more productive.
16:41And then also when we think of public policy,
16:43you know, how are we using ROI to help
16:46when it comes to climate?
16:48When I don't know if all of you know,
16:50some of you live in California,
16:52but CAL FIRE and UC San Diego,
16:55they partnered a few years back
16:57and Time Magazine recognized them as the invention of 2023.
17:03And what they do is they have thousands of cameras
17:05out in the environment.
17:06They use AI.
17:08And what they do is it actually detected
17:10the first two months that it was out there,
17:11it detected 77 fires before any 911 calls.
17:15So imagine when you're talking about natural resources
17:17and saving lives, when AI based on learning
17:22what the environment smoke detection,
17:24that it would text local fire departments
17:27before any 911 calls, that's a game changer.
17:30That's affecting human, that's affecting resources.
17:33And then I don't know if you've been at DMV lately,
17:35but there's been a lot of great changes there.
17:40The old stories of sitting in line
17:42and standing in line forever is by the wayside.
17:45And so we have the chat bot,
17:47we've all been using AI for quite a while,
17:49but Miles, the chat bot has been able to take millions
17:53of inquiries every other month.
17:56And based on that, we have a 94% success rate.
18:00So 6% only have to be escalated.
18:02So that means when people are working
18:04and they can't get done the DMV,
18:06no matter what time of night they can get their answers,
18:09you know, right in front of them.
18:10And that improves efficiency,
18:12that improves customer service
18:14because at the end of the day,
18:15government is about the public and helping the residents.
18:18So any technology we're looking at,
18:21it should not only be financial,
18:23it needs to look at the human RI,
18:25it needs to be also looking at what we're doing on policy.
18:28And I just feel like it's, we look at it all holistically.
18:31And I also agree with Renata.
18:34We deal with the data.
18:36I mean, we have a lot of data, as you can imagine,
18:37on the pension system, health systems, social systems.
18:41And it's very important that we use that data
18:43in a way that improves decision-making
18:45so we can vastly make decisions
18:47and that we can ultimately make the right,
18:50if it's vulnerable populations,
18:52how do we make sure that what we're doing
18:54is making lives better?
18:56Not necessarily, you know, dollars and cents
19:00on a statement.
19:02So we just have to be very careful
19:03that we don't just focus on the financial part.
19:06Yeah, but still very much measured
19:08as you raised, Liana, with the fire example.
19:11Ken, I wanna come to you.
19:12A question from the audience, from Stephen,
19:15forgive me, Franchetti, maybe you hit it, Franchetti,
19:17but he's chomping at the bit.
19:19He wants us to get into the nuts and bolts
19:21and talk about what were the specific initiatives?
19:25How did you create those KPIs?
19:26And that's true.
19:27You know, a lot of this,
19:28we're talking about general benefit,
19:29but the measurement, I believe that was your point, Renata.
19:32You know, the measurement is really, really key.
19:34Ken, I wanna know where you chose
19:37to make these choices in Medtronic
19:39and how you chose to measure them.
19:41If you can share that.
19:43I'd love to, and it'll build on where Liana was going,
19:46which is about, it's about the human impact.
19:49You know, we are a mission-driven medical device company.
19:52And so our prioritization is driven
19:55by where we can impact patient outcomes,
19:59improve, you know, ensure patient safety
20:02while, you know, eliminating or minimizing risk.
20:05And so as we looked at where to prioritize AI investments
20:09that kind of a return and that kind of impact on patients,
20:13we looked at areas where we could make a difference,
20:15you know, where we could improve patient outcomes.
20:18For example, in our endoscopy business,
20:21where we saw an opportunity to improve
20:25how patients who get colonoscopies are,
20:30you know, the colonoscopies using the technology
20:33that is currently, that was previously available.
20:35We applied AI to improve the detection of polyps
20:38in a colonoscopy.
20:40And on the best day, colonoscopies, you know, miss polyps.
20:47You know, one out of four polyps are missed.
20:50So using AI, much like Liana's case,
20:54where, you know, you use imaging and data and trained images,
20:58we trained the colonoscopy product
21:03to detect polyps using training data
21:07and improving the ability to detect polyps
21:11to improve patients' outcomes.
21:14So there was a very clear stat.
21:16You already had. Very clear impact.
21:18Very clear.
21:18You were not thrilled about it,
21:19and you thought maybe we could use AI to bolster that.
21:23Very clear impact.
21:24And another impact was in a cardiac rhythm
21:27where we have a device that measures patients' heartbeats
21:33for detecting atrial fibrillation.
21:38And this device, it's inserted just under the skin,
21:41and it operates 24-7.
21:45But because it's a very low-power device,
21:48without using AI, it can create false positives.
21:52Using an AI algorithm, we made this product
21:56reduce those false positives by a significant amount.
22:00And so it eliminated false positives.
22:03It created time savings for the clinician
22:06and time savings for the patient.
22:09And that allowed the patient and the physician
22:12to focus on things that really matter.
22:14So these are outcomes that come from applying AI
22:18to a medical procedure and a medical device
22:21that changes the patient outcomes
22:23and improves the lives of the patients
22:25and the clinicians that provide the services.
22:28Yeah, yeah, yeah.
22:29And we're just getting started.
22:30I mean, we see opportunities
22:32across our robotic-assisted surgery portfolio.
22:34We see opportunities in doing spine surgery
22:37with our software suite and our spinal robot,
22:42using AI to assist a surgeon to do a better job
22:47of correcting a curved spine,
22:52using robots paired with AI algorithms
22:56to actually plan the surgery on data-trained algorithms.
23:02So these are real examples
23:03where we see opportunities to improve healthcare.
23:08And so those are the product and medical device examples
23:13that I shared with you.
23:14But we have hundreds of examples
23:16of examples where we've applied AI internally
23:20to improve the engineer's ability
23:24and our employees' ability
23:25to create these products more efficiently
23:29and to take cost savings out,
23:31but also make their experience a better experience
23:34of doing the engineering and doing the design work
23:37in the software area, in the purchasing area,
23:40in the finance area, in the quality and testing area.
23:44And so, I mean, I could go on and on.
23:46We've literally gathered over 200 examples
23:49where we can apply AI to improve internal processes
23:53to make these products more efficiently
23:56because they really matter to the outcomes of patients.
23:58Yeah, I was gonna say, I think we all agree
24:00we want surgery rate success rates to improve.
24:04They're already great, but like keep it going, you know?
24:07The, Moksit, I wanna come back to you.
24:10There's a great question from the audience
24:12about how you conduct that value assessment
24:14when a solution is still in early stages.
24:16It might not be a product that's on the market yet.
24:18You know, you're just getting things going.
24:19How do you work with your clients
24:22to find the right metric to see that improvement?
24:25Because I think that's half the battle
24:27is knowing what to measure.
24:29Absolutely.
24:30So look, again, it goes back to my first point,
24:33which is you gotta start with an understanding
24:36of what are the business problems
24:38that you as an enterprise are solving for?
24:41So I'll take an example, you know,
24:43from say the energy industry.
24:45Capital projects in energy
24:48and also many of the other capital intensive industries
24:51are notorious for 30 to 50% cost and time overruns.
24:56And so you clearly have a metric
24:58and obviously these are long cycle projects,
25:00sometimes five, seven, eight, 10 plus years.
25:03So if you are able to eradicate all of the overruns
25:08and you can be in budget,
25:14the production or the productivity of these projects
25:20is realized or monetized much more rapidly.
25:24So how do you assess something like that?
25:27You of course have the starting point,
25:28which is what are the overruns?
25:30What is the timeframe in which you are able
25:32to do the project in your as a state today?
25:36And then how does AI and generative AI change the picture?
25:40So how did it change in this case?
25:42You know, a lot of capital programs have commonalities
25:46if you look at a large enough repository
25:48or pool of information.
25:50In the case of the energy industry,
25:52there are lots of deep water projects,
25:54things that are done offshore.
25:57And many of them are done as fit for purpose projects
26:01are very bespoke.
26:02As a result, the design time,
26:04and I'm talking in very simple terms,
26:06can be fairly extended.
26:07So instead of going through that whole process
26:10of specing every project,
26:14having the ability to retrieve information
26:16from all of the projects or as many of the projects
26:18as you can get access to,
26:20and looking at then the specific environment
26:25in which you are going to be making an investment
26:27in that capital program,
26:30and getting to 80% of that design in 20% of the time
26:34allows you to radically compress the overall timeframe,
26:39the design timeframe,
26:40which is often the driver for the entire
26:44sort of lifecycle of the project.
26:46It also allows you to pinpoint
26:49what are the specific needs in terms of parts
26:52and equipment that are needed
26:54and who the best suppliers are.
26:56So then you are also able to compress the timeframe
26:58for execution.
26:59But coming back to the original point,
27:02that in itself,
27:03if you look at asset intensive industries,
27:06if you can shrink your capital investment by 8, 10, 12%,
27:11as a result of compressing some of these timeframes,
27:14you have materially changed the ROI.
27:16And you often have these metrics which tell you
27:19how much the time is that your KPI could be,
27:22the project lifecycle time,
27:24secondary KPI could be the overruns,
27:26which often, of course,
27:27throw what might have been an acceptable ROI
27:30when you have a 30% overrun
27:32into a completely unacceptable ROI, right?
27:34So that's the way we do it.
27:37The other thing is, to that point,
27:42you do have to have a foolproof value management,
27:47value realization approach, right?
27:49So again, I know we are looking for very specific KPIs.
27:52It would vary by company, by objective.
27:55But what I tell everyone is, again,
27:58articulate what is the problem you're solving for.
28:01That's the first day.
28:03Ensure you can assess the value.
28:04And I understand in some cases,
28:06the value might be harder to assess,
28:08which is why you lay out these primary KPIs.
28:10You run scenarios on it, right?
28:12So if the compression of time is 20% versus 40%,
28:15what the value outcome could be.
28:17So you know what the sensitivities are.
28:19Then you have to be able to architect,
28:22what it would take to realize that value, right?
28:25Often, the value assessments you've done
28:28are purely focused on the tech
28:29and what the cost of the tech is,
28:31which as I said before, is 30% in most cases.
28:34And just a general sense for how much
28:37you're gonna move the KPI in a very localized environment.
28:40And then the fourth is activating it.
28:42So the whole value management has to go
28:45through a foolproof cycle
28:47across all of those parameters, right?
28:50And you gotta be open to some of the uncertainty
28:53inherent in the technology that's evolving very fast.
28:56So to Ken's earlier point, think big, act fast,
29:00and in some cases even fail fast, right?
29:03And move on.
29:03Well, that's exactly where I wanted to go next.
29:05Hinata, I'm looking at you
29:06when we're talking about failing fast.
29:08All of your organizations are trying to apply AI
29:12in different places to get these improvements.
29:15Not all of them are gonna work.
29:17So Hinata, I don't know if you've had an example
29:20that you could draw on
29:21where you maybe deployed a technology in this category
29:24and you went, you know, this is maybe not the right mix
29:27or it didn't have the improvements that we had hoped,
29:29but I'd love for you to share a war story if you have one.
29:33So there are a lot of new...
29:39AI is a journey, you know, AI,
29:42Gen and I, it's a chapter of AI, so it's a journey.
29:45So one thing that we do is first to promote
29:51this experiment environment.
29:53So we can learn, you can invest, learn, fail.
29:58So it's not try to prohibit everything
30:02because otherwise we don't innovate.
30:04So we have, for example, every Friday here,
30:08we have a problem.
30:08So I agree here, we need to start with the problem,
30:12what problem we are solving.
30:14And then learn from different technologies
30:18because today we also have the problem
30:21of a lot of offer of technology.
30:24There are tons of technology being offered
30:28to solve a problem.
30:29So what's the best one?
30:31I don't think that we should always find the best
30:36because sometimes we need the fast and not the best, right?
30:40What's the faster technology to delivery value?
30:44So then we created this environment of test learning,
30:48it's a kind of lab so we can learn with each other.
30:52But at the same time, we need to be very careful
30:54with how to scale things.
30:58Because then we test, we create some MVP.
31:01But of course, I believe that all companies
31:06face this type of problem,
31:07that we try to scale something that's not ready
31:11and prepared to be scaled.
31:14And then we started to have problems
31:15because then we have fails, you can impact the consumers.
31:22So one thing that we work here is, okay, this is the MVP,
31:27what are the objective key results expected?
31:31After that, let's understand what's the investment
31:36to scale because there are a lot of investments
31:38on the foundations layer.
31:40It's not just about the sexy AI,
31:44you have all the foundation.
31:45It's not?
31:48It's not, there are a lot of things,
31:50it's like a iceberg, right?
31:52So you just talk about the top of the iceberg
31:55about AI, GenNI and all the results
31:59because it's what people sees.
32:01But under the water, the major part is under the water
32:08is the foundation, the engineering, the data lake,
32:11the data governance, everything is under the water.
32:17And what moved the iceberg, guess what?
32:21It's not the wind, it's the ocean.
32:27I don't know the English, the ocean.
32:30That's okay, I'm fully with you on the,
32:32we're drowning in AI, medical, it's fine.
32:35So we need to, and that's why I needed to explain
32:38to the organization that what's the,
32:43all the layers of AI because it's not everybody
32:46that understand all this complexity.
32:48So it's our job also to invite people
32:55that doesn't understand all the dimensions
32:57to understand the dimension and work together.
33:00That's another key factor of success
33:04is how we put together technology,
33:07engineering, data solutions and business
33:11to work on this AI product
33:13because then you get the best of everyone
33:19and then we build something that's more sustainable.
33:23It's not just something that based on the hype
33:29or the technology that we like.
33:32We combine all these skills to work on this product.
33:37I wanna come back to the humanity in it all.
33:40And I promise I will
33:41because we can't have this conversation without it.
33:43But Liana, I wanna come to you and ask you about timelines.
33:46As you are all talking about these things,
33:50it is not lost on me that Ken,
33:52you work for a publicly traded company.
33:54Hinata, you work for a publicly traded company.
33:56Liana, you work for an actual state
33:59and Moksha, you work with plenty of companies
34:03with those constraints.
34:04We are not highly funded private startups
34:07who can burn cash privately without anybody seeing.
34:11We're all under public scrutiny.
34:14We can't just say, well,
34:16we promise it'll all work out in the end.
34:18And that's especially true for a municipality.
34:21So Liana, how are you thinking about deploying these things?
34:27Do you wanna get those small wins to get more buy-in?
34:33You can't commit too far in the future in state business.
34:36So how do you make sure that everybody
34:40is willing to come along for the ride
34:42in terms of ROI timeline?
34:44Well, it's really based on the thing
34:46that we've been talking about, start small and scale.
34:48So one of the things that California did,
34:50Governor Gavin Newsom signed the executive order
34:52September 6th, 2023.
34:55A lot has happened in California.
34:57I think we were really out there in the front of all this
35:01with a balanced approach because as we wanna look
35:04at benefits, we do realize there's risk.
35:06And I know with publicly traded companies,
35:09you're thinking of shareholders and your stock.
35:11I've got 40 million residents that instill trust in us
35:15and we can't take that for granted.
35:18And so what we did is we looked at it
35:20from a challenge base.
35:22And so we had a challenge statement and goals
35:25like we have vulnerable road users,
35:26there's skaters, there's motorcyclists,
35:28there's way too many fatalities on the roads.
35:30Could Gen AI be a way to solve some of those problems?
35:34I don't know about you,
35:35but anybody that's been in the Bay Area or LA,
35:36there's a lot of traffic in California.
35:38Is there ways that Gen AI can actually help
35:42address traffic bottlenecks?
35:44And as we're doing construction
35:45and can we make that a better experience?
35:48We also have hospital surveyors
35:50that go into all these hospitals
35:51and make sure that it's safe when we go in,
35:54they see violations or say they need to do a citation.
35:57Can we take some of that workload from them
35:59and cross-reference citations?
36:02So we have a challenge or goals that we need.
36:03We have homelessness, we have a lot climate goals.
36:06So there's things that we wanna be doing.
36:08And then what we do is we open it up
36:10to the vendor community.
36:11We want it to be an innovative idea from the vendors.
36:15And so we want small business, medium, large
36:18to come in and pitch their ideas.
36:20And based on those ideas, we'll drive use cases.
36:24Small use cases to test drive it in like a sandbox
36:27as Renata was talking about, how you can test drive this,
36:31make sure the human in the loop is completely involved,
36:35making sure that the business that knows
36:37what this output should look like is at the beginning,
36:40during the monitoring and also at the outcome level.
36:43And then based on the results we're seeing,
36:45is the data something that maybe we need
36:47to put different types of data in there?
36:49And we have a lot of data in the state of California.
36:52Making sure that it's explainable.
36:54Is it coming out with the right outcomes?
36:56And then providing a level of transparency
36:58to the legislature, to the world on what's going on
37:02under this microscope and then deciding,
37:06did it actually solve the problem we were thinking?
37:08Maybe Gen AI is not the solution.
37:10Maybe it's more traditional AI
37:12or maybe it's even an OCR product
37:13that we haven't implemented.
37:16And that's not necessarily, that's our term of filling fast
37:20because we're learning and then we can pivot.
37:22And the nice thing is what we do
37:24is we have vendors bake off against one another.
37:26So they have different solutions.
37:28So the business doesn't have just one option.
37:30They can look at multiple options.
37:32And then based on that,
37:33they know who they wanna scale with.
37:35And then they work with us to bring that
37:38into a bigger vision that we were just talking about.
37:40But Liana, so I'm glad you brought up
37:42some of these problems,
37:43not just because I viscerally understand them
37:45as a resident of Los Angeles County,
37:47but also because they're actually great examples
37:50of like complex intractable problems.
37:53You mentioned homelessness, you mentioned a few others.
37:56Let's go with traffic
37:57because that's a very, very tangible thing.
37:59And if anybody has visited the Bay Area or LA,
38:02you know full well what traffic is like
38:04in the state of California.
38:06What do you do in that scenario?
38:07Do you break up a complex problem like that down far enough
38:11so that you can in fact see that ROI, see that?
38:14I mean, I think about all of the factors
38:16that go into traffic on a day-to-day basis.
38:19And it's taken me years of residence,
38:21to understand the nuances of when I should leave my home
38:26to get where I need to go.
38:28How are you trying to break that down?
38:29Do you start in a very small municipality?
38:31Do you start, because quite literally
38:34all of the roads are connected.
38:38Yeah, so definitely what we're using
38:39in publicly available data.
38:40So there's a lot of data being collected.
38:42There's social media even.
38:43I mean, what's going on?
38:44So how do we collect the data
38:46and have generative AI look for opportunities?
38:48Is there ways, is there better times for construction?
38:51Is there better times if there's something going on
38:53related to accidents on how do you reroute traffic?
38:57All of that, because we're very data rich in this world.
39:00How can you do that in the analytical world
39:03to then generate new pathways and new ideas
39:05of thinking about it when it comes to traffic?
39:08And so when we think about scaling,
39:10it would be scaling in different areas.
39:14I don't know if I'd take on LA right away,
39:17but I'm not the director of-
39:18I don't recommend it.
39:19Director Jabaris will decide that at Caltrans.
39:21But I feel like once you take the data,
39:24it runs through the large language models
39:25and it can see what's coming out of it.
39:28That's where the business leaders
39:29are gonna be able to make the best decisions
39:32on how to use the technology
39:34or how not to use the technology.
39:36I think that's how we get out of the hype cycle.
39:38You have to test drive it
39:40and see is this going to be the right solution?
39:43Because we know Gen AI today
39:45is gonna look very different in six months,
39:47one year, two years.
39:49So maybe it might not be the solution at the moment,
39:52but making sure that we are thinking about it throughout.
39:55Because every hardware, software service
39:57is going to have some type of AI in it.
40:00And so what we've all heard is
40:01it's not AI that's gonna replace us.
40:03It's going to be the person
40:05that actually knows how to utilize AI
40:06that's going to replace us.
40:07So the same thing in government.
40:09I would say in 36 years,
40:12I guess I don't look like I've been in government
40:13for 36 years, but I've been for 36 years.
40:16This is an exciting time
40:18because we're not behind the cycle.
40:22We're not waiting five years
40:23behind all of the emerging technology
40:26to take advantage of it.
40:27We're very innovative
40:28and we're sitting with the UC Berkeley, Stanford,
40:31all of you in the industry
40:33to figure out how do we do this
40:34and how do we do this together now,
40:36which I'm very excited about.
40:38And Andrew, I was gonna build very quickly
40:42on what Liana was talking about.
40:45I mean, the point around the shift in the work,
40:51the worker and the workforce
40:53is one of the most profound shifts
40:55that's gonna enable the unlocking of value.
40:57And I know we often talk just simply
40:59about the technical stuff and stuff in the assessment.
41:03To give you an example, I mean,
41:04in the industrial space
41:05where you have heavy duty asset management
41:08and you have positions like maintenance engineers,
41:10I mean, we've found that you can actually impact the work
41:14of 40, 50% of that workforce.
41:16That doesn't mean that 40, 50% of the workforce
41:18actually gets eliminated,
41:20but a lot of what they do changes.
41:22So a maintenance engineer has to go through
41:24a lot of technical material
41:26to actually figure out how he or she is going
41:29to remedy a situation.
41:32Now that can be done by agents that are emerging.
41:36So the job of the engineer in that case
41:41evolves into someone that's comfortable
41:43working in tandem with these agents,
41:45which are digital workers,
41:47is also someone that could actually be more proficient
41:51in writing the right prompts
41:52to be able to understand what the situation is.
41:55So the skillset changes and hence the training
41:58and the learning requirements
42:00that Ken was talking about early on
42:02also substantially shift.
42:04I'm glad you actually went that direction
42:06because that's where I wanted to go as well.
42:08It's about the human in the mix.
42:10At this fortune dinner I went to last night,
42:11we were talking about this and one fellow spoke up
42:14and said, I need in the future,
42:18in an environment where this is across an organization
42:21quite thoroughly, I need better problem solvers
42:25and fewer people who can't do that.
42:28That's the skill that I prioritize
42:30if I can deploy AI in different places.
42:32We're talking about huge change.
42:35This conversation is about ROI,
42:36but you're not gonna see it
42:37if your organization is not ready.
42:40So Hinata, Ken, Leona, any of you on that,
42:45I wanna know, how do you make sure
42:48that you can get that AI in good use
42:52so that you can achieve the ROI?
42:54Yeah, let me build on that.
42:55Look, you have to instill the dialogue in the company
43:01that AI is part of everybody's job.
43:04That there's not like a little pocket of AI experts
43:07in the company that are building the AI solutions
43:10for the company.
43:11It needs to be an expectation
43:13of every person in the company.
43:16And so we have made that very clear.
43:18In fact, we've asked each of our business unit leaders
43:21to come with an idea for how they're gonna apply AI
43:25to make their business better.
43:27And we've asked every function in our company
43:30to come up with ideas for how they would apply generative AI
43:34and the tools that we've made available,
43:36which include the Microsoft Co-Pilot Suite
43:38and an internal version of ChatGPT we call Medtronic GPT
43:42to take cost out of their function
43:45or to improve the productivity of their employees
43:47or to automate something that's based on,
43:50things that could be automated.
43:53And as a result of that,
43:54we've gathered over 200 ideas
43:56and we're actioning several dozen of those ideas
44:02in our first round of funding
44:04to create the kind of return on investment
44:06that's gonna be tangible.
44:08But in the process of asking everybody to come to the table,
44:12we've created the narrative in the company
44:14around the fact that it's the expectation
44:17that this is everybody's job.
44:19And if you just think about the scope of our impact
44:23on healthcare in the world,
44:25it has to be everybody's job
44:27because we operate in over 150 companies.
44:31We touch more than 74 million patients every year.
44:39And if you just do the math,
44:40that's two people every second of every day.
44:43So two people every second
44:45get touched by a technology in this company.
44:48And so if we could make the technology
44:50that we put in the hands of these amazingly trained
44:53physicians and healthcare providers,
44:57it will have a profound impact
44:59on the healthcare outcomes of patients all over the world.
45:03And so it's just let that soak in
45:05and you realize that it's gotta be
45:08something that we take seriously.
45:09And this is why I'm really excited about the future
45:11and this is why we're baking this
45:13into the culture of our company.
45:15Yeah, yeah.
45:18Just a comment on Cairns.
45:22That's why here at Natura,
45:23we created this hub-spoke model of operations
45:27because we wanted to multiply this knowledge
45:31around the company, but in an organized way.
45:37Leading with this governance and not just,
45:40okay, go there, do whatever you want.
45:42No, we have the catalog of all the experiments.
45:46And the other thing is to share the user cases.
45:49That's something that help also with all the literacy,
45:54share the user cases and results.
45:57Of course, the OKR, the KPI results, ROE,
46:02will depend off the problem that we're solving.
46:05So you need to understand what's the problem.
46:07My problem is the customer service,
46:09is the cost of the operation.
46:11So we need to measure how many attendants
46:14we will be removing or what's the NPS
46:19of the consumers or consultants
46:22that we should be increasing.
46:23So depending off the problem that we are solving,
46:26we have a different indicators
46:28to consider on the return on investments.
46:33But the first thing is how to share this use case.
46:37That's something that we are doing.
46:38We have a monthly podcast, internal podcasts
46:41with all the use cases.
46:43So with that, people will start to look at the use case
46:47and understand how they can apply to different functions
46:53and things like that.
46:55And another thing that I'd like to share,
46:58I would like to complete Liana's comment
47:01about the data, expand the domains and everything.
47:06We are also living an era of opportunities
47:09to expand the domain beyond our company's area.
47:20Then you can see cars that have sensors
47:23that send the location and the weather condition
47:29to the weather channel.
47:31It's different industries, car and weather,
47:36or different channels like at Natura.
47:39In our stores, we have a scanner of the hair
47:42so people can scan your hair.
47:44And then we use this data in our R&D process
47:49to develop products with much bigger sampling
47:53of different issues or problems of the hair
47:58that we collect.
48:01So that's the, you know, we can boil the ocean
48:06with all the data thinking about, you know,
48:09the connections inter-industry, intra-industries,
48:13different industries as well.
48:15Yeah, I think the share the use cases
48:19is a wonderful tactic, I love that.
48:21And in fact, I'm going to flip it back
48:23to the rest of the panel and ask if each of you
48:26could share a very specific tactic
48:28you'd love to leave the audience with
48:30as we get to the back of this hour.
48:32Is there something like that that's very tangible
48:35that the executives listening in and watching
48:38could walk away and bring back
48:39to their organizations right away?
48:45Any?
48:46And I can start.
48:47I just started, sorry.
48:49I just mentioned about the hair that we are, you know,
48:53we are using that to fulfill the R&D.
48:58We also have data that we use for the artificial skin
49:04for testing our products.
49:05So this reduces dramatically the time into market.
49:10Another thing is the assistant to our leaders.
49:15Each leader, we have around 300 consultants.
49:18So they have an assistant,
49:20artificial intelligence assistant to help them
49:24to provide the best services and to their consultants.
49:28And this also increases the sales
49:32and not just the sales,
49:33but also being much more individually assertive
49:39of what that consultant demands in their sales.
49:46So there are different aspects
49:48from R&D, manufacturing, to the customers.
49:52Sure, sure.
49:55Muxin?
49:57Yeah, I was going to talk about one case
49:59and then come back to both a couple of points
50:02that were made and a few questions
50:04or comments in the chat, right?
50:06The first, I mean, one of the use cases
50:08we are actually internally working on
50:11as a professional services company,
50:12the marketing and communications part
50:14of our business is extremely important.
50:17So when we think about AI and generative AI,
50:20you obviously first think productivity,
50:22but then of course there's a huge element of growth
50:24that can come from it.
50:26We had recently announced a partnership with NVIDIA
50:29where we are actually now taking these Gen AI agents
50:33or autonomous agents, which as I said before,
50:35are digital, think of them as digital workers
50:37that can work with other agents,
50:39which are other digital workers,
50:40which can in turn work with humans.
50:43And we have started applying that
50:45to our marketing function.
50:47And as a result of that, we've been able to identify
50:50between 25 and 35% of the manual steps,
50:53depending on what part of the marketing
50:55and communication organization are you looking at
50:58to substantially improve productivity
51:00to the tune of 30 to 50%.
51:01But on the growth side, what it has done is
51:04it's helped identify five to 10% more opportunities
51:08where specific pieces of thought leadership
51:12could be positioned with the right clients
51:15such that we can have then new dialogues
51:18or new opportunities that can emerge from it.
51:21So you can both see the productivity
51:23and the growth aspect of it.
51:27The two other things that have stood out in the discussion,
51:30one is, I know we have talked a lot about use cases.
51:35The caution I sort of give out to the companies
51:41that we work with is let's not repeat the mistakes
51:45of the first wave of the digital transformation
51:48where there were lots of MVPs and POCs and MVPs,
51:52and there was a lot of education around them.
51:55And I've worked with companies that did 400 or 600 of those.
52:00But as I said, a very small fraction really scaled, right?
52:03So yeah, you have to do very fast tests,
52:06but these companies are actually looking at local pain points
52:09and trying to create a POC and unlock value
52:12and enterprise level value doesn't get unlocked
52:14just by doing small things.
52:16You gotta understand the business problem
52:17you're solving for.
52:19The second also comments on around ROI, right?
52:24ROI is not just about what the measurement looks like, right?
52:28Of course, we have to lead with value, as I said.
52:32ROI gets unlocked when you also just not get technology right,
52:37but you get, as I said before, the ways of working right.
52:40You change the processes, you reinvent the processes.
52:44You rethink the capabilities and skills
52:46as Liana and Ken were talking about.
52:48You create trust through responsible AI.
52:52You create a continuous engine
52:53to reinvent your organization to change,
52:56which by the way, less than 5%, 7% of the companies have.
52:59So I do wanna sort of debunk this myth
53:02that once you have figured out how you measure
53:04and you figure out the two cases
53:06that are technically very interesting,
53:08suddenly value gets unlocked magically.
53:12That is not the case.
53:13You gotta think about, again,
53:14not just the tech, but the transformation
53:17that is unleashed from it.
53:18The last point, and then I'll stop,
53:20is in all of this, I think we have heard clearly
53:23that data is essential.
53:25Our belief is that 99 plus percent of the data
53:29either is or will be democratized.
53:32And so the 1% or less of the data that's proprietary data
53:37is going to be the source of competitive advantage.
53:40And as you think about the future,
53:41as you think about creating value,
53:45to Liana's point about the technology is gonna evolve
53:48in six months, 12 months, 18 months.
53:51Building your data foundation such that you have
53:55then the agility to apply the right applications
53:59as the technology evolves is one of the fundamental,
54:02going to be one of the fundamental sources
54:04of success and hence competitiveness.
54:07And it's going to be essential to unlock value
54:10in a rapidly changing world.
54:11Yeah, all great points.
54:13Thank you, that's wonderful.
54:14Liana, I'm gonna come to you to close us out.
54:17A tactic that the audience can walk away with
54:20in your experience trying to find ROI for all of this.
54:25Even though this is a technology, put people first.
54:28I find that historically in all the years,
54:30it's always been the CIO or the technologists
54:33that have been pulling and working with requirements
54:36and pulling business around across the finish line.
54:38This is different.
54:39What we're doing is we're bringing the CFOs
54:42because this is gonna change how we budget,
54:44how we, even though in state we do one year budget,
54:47as you do prompt engineer, every prompt cost,
54:50the cost goes up exponentially.
54:52Do we have the right budgeting and financial minds
54:55in the room to talk about that?
54:57Are we having our chief of HRs making sure that labor
55:00is involved through all of this,
55:02that they feel like they're a part of it,
55:04that this is adding value to their jobs,
55:07not taking value away.
55:08So you have to have labor at the table.
55:11Your executives, your policy makers,
55:14legislators are very concerned.
55:16I think California had the most bills.
55:20And so trying to figure out, it was interesting.
55:22I'd sit around a round table.
55:23Vendors were actually ready to have some of this
55:26regulated as well.
55:27So I've never seen that before,
55:28but it's important that we're thinking about people first,
55:31that the technology is a means to delivering something
55:35that is gonna better society.
55:37Hopefully that's what we're all trying to do.
55:38And so the only way we can do that is to ensure
55:41that people are involved from the beginning to end.
55:43And making sure that they're the right people.
55:45Our governor is recently very much working with academia,
55:49as they're researching and doing great things
55:51on that forefront, ensuring that the policies
55:53that we're doing are being applied in the right manner.
55:56And I'm just gonna also put a plug.
55:58I'm really big on digital equity.
56:01We have the digital equity plan for the state.
56:03We forget one in five Californians do not have access
56:07to high-speed internet and they don't have devices.
56:10In some cases, we have tribes
56:12that don't even have electricity.
56:14As we do this great advancement in telemedicine
56:18with medicine and other things,
56:20we have to make sure we do no harm.
56:22That we have to bring people along and make sure
56:26that the gap, the divide that we saw during the pandemic
56:30does not get wider.
56:32And it is all of our responsibility.
56:34If you're in California or anywhere in the industry,
56:36we have to make sure that we are putting people first.
56:39Yeah, well said.
56:40All right, before we wrap,
56:41I wanna just offer my notes.
56:42I've obviously been taking notes.
56:44I am a journalist after all through this whole conversation
56:47and just some of the highlights
56:48that I thought were bare repeating.
56:51Lead with value.
56:53Think big, act fast, start small.
56:56Leverage the hype.
56:58Amplify humanity.
57:01Share the use cases.
57:03AI has to be a part of everybody's job.
57:07Create an engine for continuous change
57:11and actually, Liana, you wrapped up very nicely.
57:12Do no harm.
57:14So I thought those were all very, very good points
57:17to take away with.
57:18Thank you all so much, my panelists,
57:20for joining us to share your thoughts.
57:22Thank you to the audience
57:23for sending wonderful questions and feedback.
57:27It very much helped drive the conversation.
57:29And of course, again, a big thanks to Accenture,
57:32our Brainstorm AI founding partner,
57:34for partnering with us for this event
57:36and for Brainstorm AI in December.
57:38It will be held December 9th and 10th
57:41in San Francisco, Brainstorm AI.
57:42So if you want to register your interest to attend,
57:46please visit the link in the chat.
57:47It's right there to get to the Brainstorm AI website
57:51and send your information.
57:53Thanks so much.
57:54Have a wonderful day.