(upbeat music) (upbeat music) (bright music) (bright music) (upbeat music) - Before I go and introduce our, his team and special guest today, I would like to ask, how many of you folks
are expecting to graduate around the horizon of 2028? Just roughly? You know, maybe at least
half of the class, right? So I guess just one hour ago, we actually wrap out
a very exciting event, hosting our guests from SK
Hynix, including their CEO. So we are, you know, they announced very exciting
development right
here in Indiana, which basically
will be their second home in US for the leading edge
memory chips production, what we call HBM, I believe
our guest speaker today will, you know, shortly introduce
this technology as a core of today's AI revolution, right? So, you know, by the end of, I guess 2028, around your graduation,
there will be, you know, hundreds and thousands of new jobs, exciting career
opportunities in this domain, you know, right here in Indiana. So very exciting times
indeed, and l
ook forward to, you know, seeing new
generation of workforce joining this very
important evolution, right? So how we can make our future
AI models more powerful, empowered by a fundamental technology, which we will talk about today. So if we look at this chart, right? Briefly, we have seen an explosion
of AI model sizes, right? Driven by the increasingly
large foundation models or, you know, transformer models, empowering today's chatbots like ChatGPT. But if you think about
the data movement re
quired to move the training data around supporting the training workloads, and as well as inference workloads, there is huge demand for memory, right? It is almost like how would
you support, you know, like heavy traffic between West
Lafayette and in Annapolis. So it's not only about capacity, right? Holding as much data as
possible on chip in the system, but also the bandwidth, right? How do you communicate
through all this tiny but powerful course inside
of a GPU into the center and edge devic
es? So it's almost like if you
expand your number of lengths for the I 65, right? From bus Latvia to Indy, you know, from two to three lengths
to let's say eight or 16 or even more lengths, you'll get a much better traffic, right? As a result, your efficiency
will be higher to support many of these emerging workloads, right? And at the core of this technology, our guest speaker today is Dr. Hoshik Kim. Dr. Kim is the vice president and fellow of the memory
system research at SK Hynix, one of the
leaders in
the memory chips, right? Memory technologies for
today's AI hardware revolution. It's our good pleasure
today to have him on campus to co-announce our special, you know, development in Indiana for the future jobs and opportunities for you folks, as well as giving us this
very exciting lecture and talk about what's
really behind this wave of AI revolution, and what's the role of
memory technologists behind, especially behind SK Hynix. Let's welcome Dr. Kim. (audience applauding) - Tha
nk you, Professor
Lee, for introduction. My name is Hoshik Kim. I am leading research memory systems and path finding research at SK Hynix. My team is based in San Jose, California. So I was originally based in
Seoul, Korea until last year, but now I'm relocated to San
Jose to work more closely with our customers and
partners in the ecosystem. So, because most of our
customers in the AI field are, you know, in the United States, you know, hyperscalers and partners
like Nvidia, Intel, all those p
artners are in United States. So we need to work more closely together with our customers and partners here. Also, we, as Dr. Lee
also has announced it, we have announced, we had a
monumental announcement today. We are gonna build a big
HBM fabrication facility in the state of Indiana. And then we are expecting to
open the operation around 2028. You know, it's just about the same time as you guys are graduating. So we are gonna have good jobs
and offering a good industry in the east state of Ind
iana. And then we will also invite
a lot of ecosystem personal into this area to make this a reality. So memory technologies. So how many have you guys
heard about DRAM and NAND? Oh, you know DRAM and NAND. Oh, good. Then how many have you heard about HBM? All right, good, good number. I didn't know anything about
HBM when I was in sophomore, freshman. As I said, I got a C on
my computer access class when I was in sophomore, and now I became a computer architect. So even if you are not
good at i
t right now, you can still change the world with chips. Alright, before I start and
talk about memory technology, I'd like to talk briefly about my journey into this semiconductor industry. So I graduated my
undergrad school in Korea and then moved to California to study my master's in PhD degrees. And at the beginning, I studied
in electrical engineering, and then later I was attracted
to computer engineering. So when I was in college, when I took my first
computer architecture class on my soph
omore, I didn't feel interested. I was not interested, attracted
to computer architecture, so I was a poor student. And then later as I studied electrical and computer engineering
in California, I found, I started enjoying my
computer architecture class, and then attracted to this area. And then I started delving
into that area more and more, and then later started VLSI systems and then design automation. And so I got my PhD in
computer architecture and then computer design of VLSI system. So du
ring these years, all the years, I learned about computer architecture, VLSI and software algorithm,
those kind of basic stuff. So I never imagined at
that time I would work in a memory industry at the time. And then I, my interest
and specialty has evolved. So after finishing my school, I joined Intel as a CAD engineer. These days, they don't
call it CAD engineer, they, people start calling about EDA, electronic design automation, because CAD, mechanical
CAD, architecture CAD, all day call CAD.
So they start to differentiate CAD of electrical system as an EDA system. So I was a CAD engineer because I majored, I specialized in CAD when I was in school. So I developed a lot of tools,
format verification tool, timing analysis, and even
spice circuit simulators. And then I worked on a defining design
automation methodology for Intel's own proprietary CP products. So during that time, I
learned about hardware design and software development skills because CAD is basically a software tool,
which is automating
design of VLSI circuits. So it's in between
software and hardware side. So you have to understand
the hardware circuit, and then you have to
understand the software and algorithm, computational theory. So it's in the middle, it's the good mix. So I studied CAD and then studied my
career as a CAD engineer, and then I wanted to do more than that. So I wanted more, I want to involve design
myself rather than, you know, setting, supporting design as
a CAD engineer, tool engineer.
So I joined the design
team, SOC design team, and started working on SOC design. But leveraging my background in CAD, I started working on SOC design as a design automation engineer, which means using variety of
automation tool like EDA tools. I was setting design flow
and methodology from the big, from spec to final chips. So I was working on the
design automation flow, and then I also worked on
a pre silicon verification of SOC and IPs. When you fabricate your
design into real chips, it's gon
na take millions
of dollars, right? Hundreds of millions of dollars. So you need to make sure
your design is correct, otherwise you are wasting a lot of money. So in fact, Intel, for
a company like Intel, about 70% of design cycles and resources are
spending on verification. So designing verification takes more time and resources than designing itself because you have to make sure. There was a famous incident in 1994, which you probably would
know, nobody knows about it. So Intel at that time, I
ntel developed a Pentium
processor and produced it. And then there was a famous
14-point error incident. So they actually sold a lot of chips and then some mathematicians found the, some error in 14 points calculation. And then they record it, and then it was costing huge
amount of dollars to Intel. And then they started in recognizing verification
is very important. So they started investing
a lot methodologies and automation to verification. So I was part of that. And then moving from a
design
automation engineer at that time, I decide to go back to
Korea for my family reasons. So I joined LG Electronics, SOC division, as an SOC design engineer. So at that time, LG was a very good at mobile
phone business and TVs. Nowadays, LG is not doing
business in mobile phone, but at the time it was very weak. So LG decided to build
their own mobile AP, mobile application processor. So I joined the team and I
led the development there. So I moved from automation to design side and architecture s
ide. And then I did a lot of SOC
platform architecture design, and also developed a memory controller. So SOC needs a memory
controller, memory interface. SOC also needs a DDL, DRAM. So all the electronics need DRAM. So I started designing a DRAM controller and DRAM interfaces. That's why I was introduced to memory from CAD engineer to here. So I developed a lot of
chips for phones and TVs. If you buy LG TV these days,
no, that is including my IP, my SOC I developed. So I learned that process, I
learned during that process, I learned how to design SOC product, actual product and the
high volume manufacturing. High volume manufacturing
is also very important. Designing is one thing, but
when you actually produce it and manufacture in high volume, it's totally different stories. You need, a lot of things
that need to be taken care of. So I learned that product engineering and it's high volume manufacturing. And then because of this, my
exposure to memory controller, later I joined SK Hyn
ix. The currently working, I'm
working at, working for. Now, I became a memory systems architect. So I'm currently leading various research and path finding activities
in the area of memory systems and solutions, including
software solutions for mostly data center
architectures and AI, and hardware and software. So these days, you are relying on, your daily life is relying
on a lot of data centers and cloud service providers, right? You are using Google, Facebook,
Instagram, every day, right? So
those, all the service
are now being supported by cloud service and data centers. And every data centers needs memory. And actually the most
computing bottleneck, computation bottleneck is
actually memory bottleneck. So if you later, if you further
study computer architecture, almost half of computer architecture topic is about memory architecture,
memory system architecture, because of Von Neumann architecture, have you heard about Von
Neumann architecture? Do you know Von Neumann? Right, righ
t. Von Neumann architecture
is basically all the data and programming code this in memory, and there's a computation
unit and the memory unit, there always have to be a transfer of data between computation unit and memory unit. So because of that definition, all the current computing devices based on that Von Neumann
architecture concept. So memory is the bottleneck. - Sorry, just that a little
bit because, can you try? Can I see if it's on? Yeah, it's on. - Okay. So, yes. So if you think about
A
plus B equals C, right? Let's say you do the computation, right? You need to retrieve
road A and B from memory and do the computation. And then again, you have to
write back the C to the memory, right? So A plus B is now very quick, is very, can be done very quickly, right? Because of the advance of CP technologies, they can compute very quickly, but actually loading data from A and B and then writing back to
see is taking more time. So all the bottlenecks
and memory interfaces. So, I'm workin
g on, right now, our customer systems rather
than working directly on the device itself, I'm working on more like a memory systems, which is our customer
system, like a data center, AI, hardware training, inference system. So that I analyze those system and then where the bottleneck is, and then mostly it's a
memory system, right? Memory bottleneck. So I propose a new
architecture and solution, how to optimize our customer's system, have better performance, better TCOs, better, low capacity,
low
power consumption. So I propose those ideas, and then we do a proof
of concept verification with our customer systems. And then designing, basically designing future
memory solution products. So my team is responsible
for defining new solution, next generation products for data centers, AI systems, and ET devices on IOT device. So during this process, I
learned about AI system, hardware, and software. Also, I learned about data
analytics and database systems because data analytics
and database
system is, if you may not, you may not heard much about
data analytics and databases, but today in data centers, all the datas, your Instagram data, Facebook data, all the datas are stored
in data analytics platform. And then if you want to do
an AI inferencing service, all the data, you have
to retrieve all the data from data analytics and database system, it consumes a lot of memory and storages. So we study data analytics systems and cloud computing data systems, all those heavy memory users,
I learned about it or
learned about system. So I evolved from CAD engineer to a verification
engineer and SOC designer and now a memory systems architect. And I don't know what will
be the next, my journey. So I'm personally interested in these days on an advanced packaging
and interconnect technology at the chip level and silver level. So my interest is still evolving
and my personal expertise and career is I think evolving. So be curious, what I want
to say to a (indistinct), is be curious an
d open
and willing to adapt to new challenges and
then always work to learn. So every time I change my
jobs, I change company, the reason, the criteria
I chose to the next job and next company was,
what can I learn, right? Rather than just whether they
give more money, more salary. That's not my choice criteria. So my choice of criteria is
how I can grow professionally, technically, right? So, yes. So when I worked at LG at designing SOC, I found that memory is a
bottleneck of all the performanc
e. So I started to understand and appreciate the technology
of memory interfaces and then I like to contribute to that, and I like to learn more about memory. So that's how I joined
it, joined the SK Hynix. And then now I started
to realize that, oh, all the bottlenecks and performance gap, personal gap is between
interconnect at the chip level, inside the chip, there's
a multiple IPs, right? The interconnect between
multiple IP is also important and chip to chip interconnect
is also important.
And in the data center,
solve node to solve node is, solve to solve interconnect
is also very important. That's actually another bottleneck, performance bottleneck in the system. So everything is very
hard topic these days. So I'm personally interested in that area. And then the risk facility we
have actually announced today is about advanced packaging
manufacturing facility. So advanced packaging is all
about actually interconnect. Packaging nowadays, packaging requires
heterogeneous integratio
n like logic process, memory process. And due to limitation of Moore's law, Moore's law is actually
nearing the limitation. So we cannot scale and we cannot put a lot of new transistors into
single dye anymore. So whether we have to
assemble multiple dyes, dye to dye, right? We have to assemble multiple dyes. And we have even talking about
stacking multiple dye in 3D. So that's what advanced packaging facility and manufacturing does. So to do that, everything
is about interconnect, how do we int
erconnect
multiple chips, multiple dyes. So this, that's HBM, actually, that's HBM. So HBM is High Bandwidth Memory. HBM is a crucial ingredient
of today's AI hardware, including GPU. This is actually picture
of HBM, GPU, Nvidia GPU, the left side pictures, Nvidia GPU. In the center, there's
a golden dye, right? That's Nvidia GPU. And then on above and below
that, there are three each, right? Black dye, three on top,
and three on bottom, right? That's HBM. So HBM and GPU are tightly
integrated t
ogether into single package to serve better, to serve GPU better in
terms of bandwidth and power and latency. So that's how you
integrate and interconnect between GPU and HBM. So in between, there's
a silicon interposer, there's another technology
which is integrating GPU and then HBM. And also inside HBM, if you
look at this right hand side, this is a 3D view of HBM. HBM is basically stacking
multiple dye on top of each other. So it goes up to like a
12-high right now, 12 stack. And at the bott
om there's logic dye, logic dye is actually,
include peripheral logic, IEO (indistinct), and those
kind of logic process, which is communicating with the DM dye. So that's another interconnect. So we call the TSV through silicon beer, if you see these little things. So basically we drill
holes in multiple dyes of memory chips, and then connecting each
other through a metal layer. So, that's another interconnect. So this is a particular technology, which is very challenging to
implement and in ma
nufacturing. So SK Hynix is actually number
one leader in this domain. So that's why current Nvidia chips, if you buy Nvidia and Nvidia GPU chips, a hundred percent of HBM is from SK Hynix. We are the sole provider of HBM. So that's why this HBM
is a crucial ingredient of GPU and AI system. So now that we talked about SK Hynix and memory industry a little bit. So I'd like to show
where we stand in terms of top 10 semiconductor makers by revenue. If this is our last year's record. So we ranked nu
mber six in total top 10 semiconductor makers. Intel was number one, and
then Samsung, Qualcomm, Broadcom, Nvidia. Actually, we were number three last year. And then we moved to number six now because of last year
was a historic downturn of memory industry. So memory industry has interesting. So memory is high capital expend, require high capital expenditure, meaning it's a very equipment
heavy, investment heavy. If you start building a facility, you have to invest like
billions of dollars, righ
t? And then let's say you decide
to increase the HBM capacity, HBM manufacturing capacity more today, then it takes more than,
like more than a year. It takes about two years, two to three years to produce
a new memory in manufacturing because you have to build
the fabrication facilities and then you have to
install the equipment. That's why even though we have
announced the big investment in Indiana today, the actual operation will
be happening in 2028. So predicting future is very difficult. W
e don't know what will
happen in three years, right? So this rapidly changing AI world, I don't know what will happen. And then what kind of
new processing technology or what kind of new algorithm
will happen in three years, right? So right now, AI is heavily
dependent on GPU and HBM. But will it continue to
depend on HBM and GPU in three years? Do we know that? I don't know, right? No one knows that, but we have to decide to
invest our investment today. So let's say we decide to invest, we fore
casted that HBM
will continue to prevail in this industry, then we decide to invest
HBM and expand our capacity. Then later, three years,
actually no one uses HBM anymore. Then we are in big trouble, right? So predicting future is very difficult, but very important for many companies. So that's why our team is
researching future system, future AI, where this AI will go to help our company to make a
decision, important decision, and to predict the future
and prepare the future. So last year becau
se of that, so that's the nature of memory company. Because of that, there is always you know, supply and demand issue, right? So every time we increase our facility, then at some point there's oversupply, then memory industry goes downtown cycle, and then later the demand increase. Then we are insured,
with supplies insured, and then our prices go up, you know, then we make a lot of money, right? And then there's a cycle
every two years or some or so. So last year was a history downturn, so tha
t's why our
company's revenue declined to six in the rank. But traditionally we are in
within five, within top five. So, but interestingly, if
you've seen top 10, oh, this year, Micron has dropped below 10. It was actually, Samsung
was another big memory, in memory manufacturer, and Hynix was another second actually. And Micron was also among
top five until last year. But now because of that historic downturn, Micron bought off from
this list, and then Nvidia, Nvidia and Broadcom jumped
into top
five, right? Nvidia and Broadcom was below
nine, below top five before. But Nvidia is because,
obvious because of GPU, and Broadcom hit up because
of Broadcom produced a lot of AI accelerator for bigger companies like Google and Microsoft,
those kind of big companies are developing their own AI chips, but actually that design
service is being done by Micron are broken. So that's why their rank went up. So the point I want to make is, among these top 10 or top five, top semiconductor in manufact
urers, memory companies are among
top five, top 10 companies. So when I was in college
or my early career, I didn't even thought
about joining memory. I never imagined I would
work for a memory company. Frankly speaking, I was
interested more like SOC design and computer design,
and as a CAD engineer, CAD engineers are dealing with automating complicated chip design by modern advanced computer algorithm. But memory was to me at, when I was, earlier when I was younger, to me, memory looks like a,
it's
a fairly structured design and full custom circuit design. So that was some, that
looks a little boring to me. So I decided to join CPU
company and SOC company, and then later I learned that, oh, memory is very complicated, there are various technology
and also it dictates, and then it dominates all
the overall performance. And then nowadays, because of this AI, event of this AI technology, people started to realize
memory is a key component because all GPUs, as you know, Nvidia GPU is ins
ured right now, but one of the big cause
of that short supply, short production is because
of HBMs supply issue. So they need more HBM, even
though they have GPU production. If HBM is, they don't have enough HBM, they cannot produce a GPU because GPU and HBM has to be integrated
together in same packaging. So, and there are a lot of AI
hardware accelerator companies and startups, including hyperscaler, many big tech companies
like Google, Microsoft, those companies are also
building their own AI
chips. They try to diverge from Nvidia, but they all need, at the end
of the day, they all need HBM, all the, they realize
that all the bottlenecks and memory interfaces. So we are in, we need to
expand our production of HBM. So that's, people started
to realize the importance of memory technology. And then we are actually working
with many, many companies, many, many chip companies, including hyperscalers, Nvidia,
you know, Intel, startups, they all request more memory interface, more memory s
upply from us. And then they want to research together to innovate the AI
hardware system together because they realize this
memory is a key component. So that's the point I want to make here. And then talking about
memory company and SK Hynix, we are second in DRAM market. Samsung is about 40%, we are about 35%. And then there's a NAND business. So do you recognize the
difference between DRAM and NAND? DRAM is a volatile memory. So it's basically, it's
used for computation. And NAND is a non-vo
latile memory, meaning that once you
store some data in NAND, even if you turn off the power,
then the data is reserved. So NAND is used for storage, right? So DRAM is used for computation. So if you think about your iPhone, if you store your photos, those photos will be stored in NAND and then if you do, if you
want to do text messaging, if you want to do, if you
want to unload your posting, or if you wanna do something gaming, then those data is from DRAM. So DRAM is for computation, but NAND
device can
have a higher capacity. They can hold more data
in a smaller spaces. But DRAM, but NAND interface is slower, accessing data from NAND is
taking more time than DRAM. That's why DRAM is used for computation. And DRAM is more expensive and NAND has a relatively
higher capacity than DRAM, and then more cost effective. So we are also number two to
three depending on the time. So we are about number two to three in NAND business right now. So memory, I said memory is
everywhere around our l
ives, right? Memory goes to cell phone, AI system, and even head mount display
like Oculus and your cars, automotive, computation,
data center, and everywhere, everywhere needs, if you need, if you have compute device,
memory is everywhere. So now we are relying on, our life is heavily
dependent on our data centers and cloud services, right? They are heavily using memories and DRAM. Data centers are crunching
and producing massive amounts of data every day in real time. So even though, I mean, t
hey
are using a lot of memory and data, but they are also
making new datas every day and every time, right? You put more data,
especially in this AI world with generative AI technology, they are generative AI itself
is producing new datas, new information. So that all goes through
memory and storages. So this AI, big data, cloud computing, all requires increasingly higher bandwidth and capacity from memory to manipulate this
ever-growing amount of data. Higher bandwidth, meaning as
Professor Lee
talked about, they need more data in a
certain period of time, right? In a second, we need more data
to be consumed and processed. That's high bandwidth and capacity is a large
amount of data to store. However, this memory is very expensive. If you think about data, server costing, overall server costing data centers, about 30 to 40% of data
center server courses, a memory cost. That's why this memory
company's making good money and at least among top
five to top 10 companies in overall semicon
ductor company. So nowadays, because of
this data center world, world is heavily relying on memories and the data centers and AI. Now memory takes the center
stage of the computer game. It was in the center from
the beginning, but no, but for the time being, people
concentrate on CPUs and CPU. Now because of this AI systems, AI world, people started to realize the
memory is a key component. So memory goes to the
center of the stages. Now we call it memory
centric computing era. So one makes AI,
data makes AI, right? So to make AI system smart,
you need to train your system. Motion learning is training
your system and data, you have to, machine has
to understand your action, your history, right? Everything, best data in the form of data. And then data is making AI possible. And also, again, AI with generative AI, especially, generative
AI is making new datas and producing new data
every day, every time, every hour, every second. So there's a cyclic
transformation of data. Data makes AI,
AI makes data. These AI, these data are all
residing in memory devices. So however, so this AI and
machine learning technology is a very memory intensive technology. It requires a lot of memory operation, memory data read and write activities. This AI algorithm
requires a large data sets to train this neural network models. And also GPU, when GPU inference
do inference operation, GPU requires high bandwidth, high capacity memory for this
memory intensive training and inferencing of this
large l
anguage model, right? Large language model has a
huge amount of motor parameters that those motor parameters
need to reside in memory, especially in HBM to be processed by GPU. So this is heavily memory
dependent technology. So now, in order to make
this AI system very costly, I mean, AI system means very
expensive system right now. So it requires, if you wanna do training, it requires like thousands and million, tens of hundreds of thousands of GPU, which is, you know, causing a lot, right? So
we need to find more cost effective way to process this AI
training and inferences. So to do that, memory
needs to play a key role to make it more high bandwidth and high capacity in more
cost effective manner. And then we have to, we even have to design
the whole system together with the GPU companies and CPU companies to better utilize our memory
rather than just focusing on producing more
capacity and more memory, more bandwidth into the
memory device itself. We have to work with our
customer
s and AI systems overall. Sometimes we have to work with
the software algorithm guys to innovate the way we
handle memory traffic in computer computation. So that's actually my
team is doing, right? My team is not working
directly on the DRAM itself. We are working on a software ecosystem and overall system level. So AI is still, but AI
as Jensen, Nvidia CEO, Jensen Huang said, AI is still at the beginning
of the workload stages. Actually, AI started to
boom last year, right? We talked about DNN
and computer vision and speech recognition about
like several years ago, but suddenly it boomed up because of this ChatGPT
announced last year, right? Because ChatGPT or everyone
is talking about AI system, right? And no more people like us
started to pay attention to AI until last year, we
didn't, we heard about AI, but we didn't really use AI, right? Itself. But nowadays, everyone is
using ChatGPT right now. So it's gonna go up, right? At least for 10 years, right? Jensen predicted that this
AI boom will go like more than 10 years and then this is actually
second year, right? Or this AI boom. So we need to continuously
invest hardware system that makes our AI service possible, right? That's why OpenAI has invested this, publicly announced that
they are gonna build a whole ecosystem and whole
hardware system themselves, and then it started to invest
no astronomic amount of money to their hardware system. However, I don't know, this
can be sustainable, right? This, it simply needs nuc
lear power plant to support these AI training data centers. And then we need like millions of GPUs, which is very expensive these days, and then a lot of power, real estate to build that
data center in this land. So we have to figure out how
to make this AI hardware system more cost effective, more
or less, consume less power. So in that innovation, GPU
guys also has to invest a lot. And then software algorithm, AI model, and model developer has
to also invest a lot. But we have a memory company
, memory technology also has a key role and responsibility to make
it more cost effective. And we have to lead the innovation. So only with that help of AI memory, advanced of AI memory, we can truly boost up the
efficiency of AI system. Otherwise, we can't sustain
this kind of system anymore. I don't think so. So to run a typical data center, they say about 500
megawatts of electricity. This is about half of
a nuclear power plant. Typical nuclear power
plant is one gigawatt hour, but 500 megawa
tt hour
is about half of them. But now advance of our LLA models, if we're talking about GPT-5,
GPT-6, then the power costs, the amount of power consumption
will go exponentially. So we probably cannot sustain and support. So that's where we as a
semiconductor engineers, including GPUs, CPUs and AI memory, we have to innovate and then
brainstorm out together, working with the software engineers. So with that investment into
AI memory solutions, AI, SK Hynix has a multiple variety of AI memory so
lution portfolio. The most important thing is HBM and then HBM is very essential
part in data center AI, but nowadays people start to talk about AI or only device AI. So if you want to use AI route
these days from your phone, your request actually goes to data center and cloud service providers and then the AI inferencing
training is done from the data center and then
retrieved back to your device. But that's, people start thinking that that
might be inefficient, right? Why don't you just
proces
s your AI process, AI inferencing at your phone? That's why we call, what
we call on device AI. To do that, we might need less bandwidth, but we definitely need a higher efficiency in power consumption because
smartphone cannot sustain that high power consumption. So that says low power
memory is important, that this low power PDL
memory is coming at. So this low power is a
more like a device level or your notebook, laptop
level AI solution. And then you also need
high capacity of memory to stor
e the large, huge amount of training data
rather than the computation. So that's why this high speed, high capacity storage device is important to store the train data, and then we have to also
innovate how we compute. So this is interesting point, I told you earlier about
Von Neumann architecture. Von Neumann architecture has
a distinct differentiation between those compute logic
and then memory logic, memory unit. But people start to realize that, oh, why don't we just do the
computation at th
e memory, inside the memory, right? Rather than moving data,
transfer back and forth between a compute and a memory, we can just do computation
inside the memory. Memory has a very structured and multi parallel design inside the chip. It has arrays of bank and cells,
you know, rows and columns. So it has, in here it is inherently
very parallel structures. If we can do little computation
inside that memory cells and banks level, then we can internal, we can utilize this heavy,
high internal bandw
idth. So that's where we start
to think about processing in memory. It's a, we have a first product. It's not in mass production, it's a first, something like a prototype
kind of concept product. But memory companies start to
think about this processing in memory devices. A couple of years ago,
we have announced a GDDL based memory processing in
memory, we call the AIM. AIM stands for accelerator in memory. This specific device is
designed for AI computation. So AI computation is, if you think a
bout this AI
computation theory a little bit, at the end of the day,
it boils down to metrics, multiplication, and accumulation. So it's all about metrics,
multiplication and addition, multiplication, addition. That multiplication, addition
is rather simple logic, but it requires a lot of
data computation in parallel. So that's a very good target example of processing that can
be better done in memory. So that's pin. Oh, we have time, right? So that's high bandwidth memory. Yeah, I'll move fast.
So this is, I talked about
high bandwidth memory. We achieved the high bandwidth
by adding more IO pins to memory and then stacking multiple dyes so that we can achieve higher bandwidths and higher capacity. So I'll move fast and then, even though we are expanding
bandwidths and capacity by stacking multiple dyes in 3D manners. So chips still need SOC and
GPU still needs more bandwidth and more capacity. So that's why we start to
integrate multiple HBM cube around this GPU SOC. And depending on
the application, training requires more
capacity and more bandwidth. So we, in training system,
typically we add more HBM cubes and then inferencing,
we put less HBM cubes. So HBM market forecast, you know, HBM market will continue to
grow even more in a higher rate than AI accelerator because AI
accelerator, the requirement, the amount and the number
of HBM cube required by AI accelerator is continuously growing. So we are expecting
even higher growth rate. So I also want to mention
about new
opportunity with CXL. I talked about HBM a lot, but CXL is a new opportunity
for memory companies and new opportunities for data centers. So basically between CPU and DRAM, they all require parallel
interface, DRAM interfaces. If you see, if you look at CPUs, there are around the CPU dye,
there are DRAM interface, which is parallel interface, which is very costly and
then takes a lot of spaces. So even though, even if
you want to add more DRAM, you cannot simply add more DRAM because it'll explo
de the chip
size, which is very costly. So, but however, so we need more, SOC are adding more computation
core inside the CPU and GPU. They require more
capacity and more memory, more memory bandwidths. But there's no way to add more memory. So the system architect, what system architecture do
is they simply add more GPU, they require more GPU not
because GPU competition needs more resource, just
because they need more memory. So more in order to achieve,
in order to get more memory, they have t
o replicate and
increase the number of GPUs to consume more memory, which is very costly and inefficient. So people have proposed a new standard using a new interface to add more memory besides the DRAM channel. So CXL is basically
utilizing PCI express lane, which is traditionally used
for IO devices, IO device, IO peripheral device extension. Then the industry actually propose an idea about the CXL interface
to add more memory. So because of this, there's
a new area of growth and opportunity f
or
memory companies to expand and then contribute to this system. So yeah, I will, yeah, I
have a little bit of time. So because of this CXL, it will create new opportunities
for computer system, overall computer system, which is beyond what is possible
today in server platform. Because of the new interface, we can have a way to expand
memory, bandwidth and capacity. And with the CXL technology, memory controllers comes from, move from a CPU site
to memory device site. So we can have a more room
for different memories. And then we can also, it can easily integrate
computational core, processing core with memory. So it'll make possible the, it'll make a computational
memory possible. And finally, it'll also
allow computer system to disagree memory from
compute note to memory note. So SK Hynix actually preparing a new kind of concept memory solution products to address each of these. And then working with data
centers and hyperscalers to test and do a proof of concept,
they work with our
customers. And then I will, I say
this will be a big change and big shift and change
in system architecture and data center
architecture in five years, from five to 10 years right now. So yeah, so memory now,
this is a very great time, great opportunity, it's
a good time for all of us to work on, working in the memory industry and then drive the innovation together in this data center world. So HBM and AI memory solution
will be the key enablers of AI technology. And then the CXL will create a
new way, whole new way to design
and use our memory system in data center structures. So this is a really exciting time to work in a memory company
and memory industries. So having said that, I want to
briefly talk about SK Hynix. We have four manufacturing
sites in Korea and China until yesterday. And then we have announced
today a big announcement and investment in the United States, in the state of Indiana. We are gonna have a big manufacturing site in the United States, and then four RD cent
ers located in US, Taiwan, Japan, and Korea, right? So there is a plenty of
opportunity for all of you, can contribute to this memory industry. And yes, this is a facility here, Professor Lee talked about today. We had a monumental announcement to that. We are gonna build a big
HBM packaging fabrication and also research chief
fab, and R and D fab in here in Purdue research park,
Purdue research science park, near in within West Lafayette. So it'll be about 90
acres in research park. And then st
arting the second half of 2028, we are gonna do mass
production of HBM here in the Indiana state. And then we are expecting
about thousand of engineers, R and D engineers to this
manufacturing facilities. So by having constructed, by producing new memory
manufacturing fab in United States, so we can better work with
our customers and partners, and then we can build the
ecosystem, whole industry, semiconductor ecosystem
in the United States, and then we can hire more
talented people like you, rig
ht? And then let contribute
to this industry together. So Hynix requires many engineers
from different backgrounds. About 30% of our engineers
are electrical engineers. But about 18 to 20 people
are chemical engineering, material science, mechanical engineering. And even these days,
computer science engineer, computer scientists are very important to work with the AI systems. So we have number of opportunity and from diverse backgrounds, specifically the SK Hynix in
California, San Jose office.
We are looking for these
kind of engineers right now. There's an immediate career opportunities. I know you guys are still, you know, undergrad and first year,
second year students, but we are expecting and
recruiting actively AI, ML system architect, SOC designers, system software designers, not just a DRAM and then device designers. We are actually working
in our LinuxCon community because if we want to,
if we divide device, if we produce a new type of memory devices that needs to be supported
by LinuxCon, right? Current LinuxCon is only
supporting DRAM and NAND. But let's say we design, we propose a new peel of
memory between DRAM and NAND, which is not being supported
by LinuxCon right now. So we have to divide, we have
to develop our own LinuxCon and then propose, and then contribute to
the LinuxCon community to support the that ecosystem
to support this new type of memory devices. So that's why we need systems of engineer, LinuxCon developers, those things. So that's it. So that'
ll be my lecture today. So please join us to change
the world with chips. So, thank you. (gentle music) (audience applauding) (upbeat music) (upbeat music) (upbeat music)
Comments