Brainstorm AI Singapore 2024: It Ain't Easy Being Green
Seth DOBRIN, Founder and CEO, Qantm AI Tim ROSENFIELD, Co-founder and co-CEO, Sustainable Metal Cloud Moderator: Clay CHANDLER, FORTUNE
Category
🤖
TechTranscript
00:00So I want to just kind of follow with a couple of questions for you, Tim,
00:04because these were really impressive.
00:07So you can basically get carbon emissions,
00:11electric power consumption, and spatial requirements
00:15down by about 50% for each of those three things.
00:19That's right.
00:20I couldn't quite hear you backstage,
00:22but did you mention the recent kind of metric
00:24that kind of benchmarked this?
00:27Yeah, I didn't actually mention that on the presentation.
00:30But indeed, it's one thing to come out with a system
00:33that is very energy efficient.
00:37Obviously, we live in a world of intense commentary
00:41around sustainability, and there's obviously the propensity
00:44for clean washing.
00:45It's just a fact of life.
00:47So it was very important for us that we could demonstrate
00:51to our users and, frankly, to folks that we're in dialogue with
00:56who might use our technology that this system,
00:59despite being generationally further ahead
01:02in terms of its consumption of energy,
01:04was as performant as it could be.
01:06So to your point, Clay, we worked with an industry association
01:10called ML Commons.
01:12ML Commons, for those that don't know,
01:15I'd kind of describe it like the Olympics of AI for geeks,
01:19basically.
01:20It's an open source forum.
01:23It was founded by NVIDIA and Google and Microsoft,
01:26and there's about 125 members now.
01:28And every six months, there is a competition
01:31to run different AI models and see
01:34how fast your system runs them.
01:36And forever and a day, that's been the main metric
01:39as to how this Olympics are judged.
01:43Moving forward, and this submission that we just did,
01:46ML Commons said, look, AI has a sustainability problem.
01:49There's a green problem with AI.
01:51So why don't we start tracking power?
01:54How much power does it take to run these models?
01:57The end output is we've been the first in the world
02:00to submit not just the performance,
02:02but the power benchmarks and have that verified
02:04by the industry.
02:05And it's a very exciting moment for us.
02:07That's great.
02:08You know, when we were visiting this data center,
02:10you made some comments that I thought
02:12were really interesting.
02:13I might ask you to share with the group here.
02:16The reason you came to Singapore was kind of the opposite
02:20of the paradox of plenty.
02:24You know, we often hear the story
02:26about how countries that are blessed
02:28with lots of natural resources actually
02:30don't develop that much because they don't have to.
02:32Singapore is the opposite of that.
02:35Maybe you could say something about why that influenced
02:38your decision to come here.
02:39As I mentioned, I'm from Australia originally.
02:42And Australia is a big country with a lot of energy.
02:44So sustainability and efficiency has never
02:47been high on the agenda.
02:49Singapore is a perfect environment
02:52to put to work the thesis that we need to do better.
02:57Singapore has power constraints.
03:00It has land constraints.
03:01It has a very strong drive at government policy level
03:05and within industry to do better on the sustainability front.
03:10So for us, moving our business to Singapore
03:14and setting up Singapore as our global hub
03:17was a good test because we've been able to deliver
03:20into Singapore more AI capacity than has ever
03:25been available before in a private cloud like ours
03:29thanks to this ability to use half the energy.
03:33So that's one of the ways that we've actually
03:35been able to showcase in a constrained environment
03:37like Singapore, this type of technology is needed.
03:40So it's interesting to me because this observation
03:43that it's precisely in energy and land
03:46and resource-constrained environments
03:49that you have to radically innovate.
03:52You can't just tweak the traditional air cooling system
03:55and get 3% or 4% or 5% increases a year.
03:59You've got to do something that's going to radically
04:01cut resource requirements.
04:04So it's kind of like if you can make it here,
04:06you can make it anywhere.
04:08Exactly.
04:09And I absolutely love Singapore.
04:11I live here.
04:13It is very much that ethos.
04:15If you can prove it in Singapore, it's hot,
04:18it's constrained, they want to do better on sustainability,
04:22you can prove it to the world.
04:23Seth, let me get you involved here.
04:25Seth, for those of you who don't know,
04:27is the former AI officer at IBM.
04:30He's written a wonderful book which is available,
04:32I think, in the lobby if you get a chance to pick one up
04:35and is really one of the most kind of far-reaching
04:39AI thinkers in the world.
04:41We're delighted to have him here.
04:44He is also the founder of a startup that is investing
04:49in sustainable AI technology.
04:51Seth, maybe you could say a little bit about that
04:53and why you have chosen that path.
04:56Yeah, so in addition to Quantum AI,
04:58which is a consulting company,
05:00I also have a venture fund called One Infinity,
05:02which is part of the reason I came here,
05:05is to raise for that.
05:06And we invest in responsible, safe, and green AI.
05:11And investing in green AI is important,
05:13A, because there's a lot of greenwashing going on.
05:16You can get on quote-unquote green clouds.
05:19That's moving on green carbon credits,
05:22which is multiple counting of it's not really green.
05:26And so just like a lot of things aren't really AI.
05:29And so we really thought it was important,
05:32My partners and I, to look at how can we support true
05:37Investments in things that are green, like really green.
05:41New hardware, new paradigms in how you support AI.
05:47And workloads around AI. So some of them like what
05:50Tim's building, completely new ways of thinking about how you
05:54Operate on silicon. So moving away from traditional
05:58Semiconductor technologies to new silicon technologies that
06:02Are a thousand times faster than today's gpus or new math
06:06Compression algorithms that are 50 times faster than traditional
06:10Matrix multiplication, which is how language models run today.
06:16Or even things as simple as I'm going to have gpu as a service
06:20And some percentage of those gpus are going to be available
06:24In clouds where they run on wind or solar or, oh, my gosh,
06:28If you put it in iceland, it can all be geothermal.
06:31And not only is it completely carbon neutral, in the
06:34Wintertime, it's carbon negative because they use it to heat the
06:37Housing and the environments around them.
06:40And so there's lots of very simple things that start-ups are
06:44Doing that are really, truly green.
06:47And so that's why we decided to start investing in platforms
06:51Like this. By the way, you can make a lot
06:55Of money on them.
06:59One of the things that tim and i discussed, which i hadn't fully
07:04Realized, part of the issue here is the people that operate data
07:08Centers, they're not engineers for the most part, right?
07:13They're real estate investment trusts, private equity funds
07:18That think of these structures as office buildings or more or
07:21Less an amazon warehouse or something like that.
07:23But as tim was pointing out, you really have to think about the
07:26Engineering and think of them as factories in the way that an
07:30Engineer would. How much, i mean, if we could
07:34Switch to these kind of liquid immersion cooling type
07:38Technologies and get away from the traditional air conditioned
07:42Model, what do you think, seth, about that?
07:45Is that going to be a solution that will really move the needle
07:49In terms of power and space consumption?
07:51I think there's multiple ways that you can tackle this.
07:54One is there's a lot of traditional data centers are not
07:57Just air cooled, they're water cooled.
07:59Water cooling creates problems of itself, right?
08:02If you go and use chat gpt today or clod, i don't know how many
08:06People you know this, but for every 25 to 50 prompts,
08:10Depending on how big they are, you use about a half a liter of
08:14Water just through evaporation.
08:16So they're liquid cooled, more efficient than air, but you're
08:21Still harming the environment. So we absolutely need new
08:25Paradigms of cooling. People say we're going to
08:29Consume 70 of the world's electricity.
08:31We've been running out of oil for 50 years.
08:33We're not going to run out of electricity.
08:35People like tim are going to invent new technologies where
08:39And we're going to hopefully fund some of them that are going
08:43To get us to where we're going to continually find new ways to
08:46Make sure that doesn't happen. It's going to be things like
08:49New cooling, new hardware paradigms, new math paradigms.
08:52It's going to be moving away. People talked about this
08:55Earlier today. These massive large models are
08:59Not what the technology was built for.
09:02They're cool, they're fun, but that's not the solution.
09:05That's also going to help the power. It's really small models.
09:08It's more efficient technologies. So we're not going to have this
09:12Problem of these massively large models using all these powers
09:15Because that's not what the solution is.
09:17That's very interesting. So a couple of things i hear there.
09:22One is, well, on this problem of kind of water consumption,
09:26You've sort of figured out how to recycle the water to make
09:29That more efficient. You don't use water, right?
09:33We use waste water. So we use waste water and i'll
09:38Describe a little bit for a second here on how it comes together.
09:43But this concept that you talked about, clay, which is
09:47Traditionally the data center, the end-to-end energy into
09:51Knowledge, whether it's cloud, whether you're using, you know,
09:55Whether you're doing a google search or whether you're using
09:59Whatever, apple maps, there's all these things in between,
10:03Which is this point, and you said it too, seth, there's this
10:07Form behind it, the actual factory that most folk don't
10:10See. We're using phones now and
10:12Today and you don't feel it and you don't touch it, but it's a
10:16Real-world impact. So this notion that there's
10:19Actually been a silo, there's a data center who builds a
10:23Building, there's a cooling engineer who builds a cooling
10:26System, which involves chilled water, there's a computer
10:30Manufacturer who builds a computer, there's a chip
10:33Designer who designs a chip, and each one of those people
10:36Says to the other one, you go figure out your part about this.
10:38You go figure that out. Someone will figure that out.
10:40When jensen comes up with a chip from nvidia, someone else
10:42Figures out the cooling. And it's been fine up until
10:46Recently. Problem is that graph that i
10:49Put on screen before, there's this insane amount of energy
10:54Consumption now going into these chips, and a verticalized
10:57System like ours is the way that we get over that.
11:01To your point, clay, and question about water, yeah,
11:05Most data centers, whether they're using it for direct
11:07Chip cooling or they're using it for air, they use chillers.
11:11And that's a real energy hog, especially in this part of the
11:14World where it's hot and humid. So when you use a chiller,
11:18You're really overcooking the amount of energy you need.
11:23Our system takes the building's effective waste water, so it's
11:26Called condenser water. So after the water has done its
11:30Job in the rest of the data center, it goes out into an
11:33Air-conditioned tower, and it gets recycled.
11:35We take it at its hottest point before it goes into that
11:38Cooling tower. We bring it into our hypercube,
11:40Which is a self-contained system, and then it does its thing.
11:43So in that sense, technically we use water, but we use water
11:46That was already in the loop, and we don't use chillers.
11:49So that's the energy saving that we get.
11:52What this kind of suggests is that there's been driving the
11:56Chips, you know, moore's law, which we talk about all the time,
12:00That are well at work in making them faster and better and
12:03Smaller, but that there hasn't been up until now a real moore's
12:07Law type dynamic in the way that we think about data centers.
12:11We build the big ones out, keep on building them out, and that's
12:14What we need. We need to start applying
12:17Engineering and science to condensing those.
12:19I think to a certain extent. So data centers got smaller for a
12:22Long time. For a long time, data centers
12:25Got smaller and smaller, so much to the effect that there were a
12:29Lot of places where they were trying to figure out what to do
12:32With the space in data centers. But now that ai has really taken
12:37Off, data centers, in some data centers there's still a lot of
12:40Room, but they're consuming more power because the chips are more
12:43Power hungry. So in a lot of places, it's not
12:47A space issue, it's a power and heat issue.
12:50So we shouldn't talk about a space, it's really a power and
12:54Cooling or heat issue.
12:56It's not that we can just put all the data centers in iceland
13:01Or sweden or denmark. I would challenge that,
13:05Actually, because do we really care about latency when we're
13:08Having a conversation? and we need to really think
13:11About when do we care about latency.
13:13When we're training models, we don't care about latency.
13:16When we're using models, when do we care about latency?
13:20If i'm having a conversation, so if my employees are having a
13:23Conversation with a system, do i care if it takes a half a
13:27Second versus a tenth of a second?
13:29Do i really care? is that worth impacting my
13:34Company's carbon emission? right?
13:37Right? that's a really interesting
13:42Insight because it's a focus, it shifts the focus away from the
13:46Operator of the data center or the manufacturer of the chips to
13:50The actual user of the data. And what you're suggesting, if
13:54I hear you correctly, is that we all need to figure out a way to
13:58Be more thoughtful about the trade-offs involved in using
14:01These five frameworks.
14:03It's part of the architectural decision.
14:05Getting back to what you said about my book, that's part of
14:07The conversation in my book. It's part of the
14:09Architectural conversation you need to have when you're
14:11Designing each use case. And it's part of the human
14:13Conversation. When you talk about human
14:15Centered design, panos was talking about the human ai.
14:21You have to think about the human. Who's the human going to interact with it?
14:25How is the human going to interact with it?
14:27What is the human decision? how important is that human
14:29Decision? all of these things play into the
14:31Architecture, which has implications on, we're talking
14:33About human centered, when we're talking about green, it has
14:37Implications to how you make that, how green that, you know,
14:42You would greenify that ai application.
14:46Yeah, so it's looking at the kind of demand side and the
14:49Psychology of the user. I use chat gpt all the time,
14:53And i've gotten really lazy about it.
14:56It's so good, it's so efficient that if i can't think of the
15:00Right word, i'll use it as a thesaurus when i've got an
15:03Actual thesaurus that's probably on the bookcase behind me.
15:07There's nothing about the way that that is delivered to me
15:11That service is delivered to me that makes me stop and think,
15:14Actually, you know what, for the amount of resources i'm going
15:17To burn to just do this, i should actually take out a real book.
15:21Or use siri. Siri is on your phone.
15:25One interesting unfortunate head wind to that, though, is the
15:32Growing narrative around data gravity and data sovereignty,
15:38Which is because i'm of the same opinion, in fact, you know, a
15:43System should be designed for its best use case.
15:46What we've designed are ai factories and not necessarily
15:49What you use for storage or what you use for general purpose cpus.
15:53There's different horses for different horses.
15:56But there is a very real and present, i mean, we see it with
16:00The chips act, but there's a lot more out there.
16:04There's laws heading towards parliament in india.
16:07There's a whole lot of countries that are very concerned about
16:10Making sure that data doesn't leave the country.
16:12And it might not always be the choice of the end user,
16:14Unfortunately. So there's a policy piece to
16:16That as well.
16:18Well, this sort of came up in our conversation with minister
16:21Cho about data sovereignty. If you're singapore, for
16:24Example, you can make the argument, why don't you just
16:27Move your data across the border to johor, it's cheaper, there's
16:30Lots of space. But there's lots of data that
16:33You basically, if you're singapore, want to keep on
16:36Short, your national health insurance data, the defense
16:39Data, the military data, anything that's proprietary,
16:42Commercial data, the big banks, these kind of companies.
16:45There are a lot of multinationals who will want to
16:48Set up a data center here but not have it across the border
16:51Because they're not sure that it's as stable as singapore is,
16:54Which has a better international brand.
16:56If you're multinational, you can't control data sovereignty laws.
17:00There's data sovereignty laws in the middle east, in europe,
17:04In south america, in africa. Yes, singapore can control it
17:08For singapore, but you can't control it for the rest of the
17:12World. So even outside of individual
17:16Countries, they're going to exist.
17:18And some of them are for good reasons.
17:20Some of them are just because. So there are technologies now
17:27That federated learning actually works now.
17:29We talked about it for a long time. It didn't really work.
17:32Now it actually works.
17:34If we were to think of just one or two things that we could
17:38Really focus on and prioritize to fix this problem of energy
17:42Consumption and carbon emission of data centers, what, in your
17:45View, would that be? i think, you know, especially
17:49When you're looking at generative ai, get back to the
17:53Foundation models, which is the technology underneath generative
17:57Ai was built for. Look at using small task
18:00Specific models. So focus models.
18:02Those use a lot of energy because they're smaller.
18:05Focus on the use case. What problem you're trying to
18:09Solve and determine what type of latency you can tolerate.
18:13And then i think, you know, focus on do you really need
18:17Generative ai? what's the right tool for the job?
18:20And i think those are the things that you can do today before we
18:24Develop new technologies.
18:26Really interesting. Very sophisticated.
18:28I feel like i've learned so much from this panel.
18:31Tim, i know you all are looking at bringing some of your
18:35Manufacturing capabilities to singapore fairly soon.
18:39And we'll be rolling out, as you mentioned, to thailand, india,
18:43And what was the other one? in australia.
18:46Further afield. Great.
18:48So it looks like it's a promising market for that kind
18:51Of technology. We're out of time.
18:53I think we'll have to leave it there.
18:55But i want to thank our two panelists for a fascinating
18:58Conversation about what really is a crucial topic for artificial
19:01Intelligence. Gentlemen, thanks very much.
19:03Thanks, clay.