right so Jaime Teevan is our
next session uh Jaime is Chief scientist and Technical fellow at Microsoft
where she's responsible for driving research backed innovation in the company's
core products in this session on AI she'll look back at how the technology has
developed and she'll also be looking forward to consider how we shape preferred Futures
please give Jaime a huge round of applause hello and welcome I'm excited to talk a little bit
today about uh what science tells us about how we
can Thrive with AI and you know I'm particularly
excited to be doing that here at bet because we're in the middle of a a generational shift in
how people get things done and educators are at the foundation of ensuring that the world is ready
for that shift so uh you know as she mentioned I'm Chief scientist and Technical fellow at Microsoft
uh which means I'm responsible for driving research back to innovation in our core products
and it also means that I've had a pretty exciting year um y
ou know today I'm going to be talking a
little bit about Ai and what that looks like and you know sometimes it can feel like that's a new
topic it's been like headlines recently um but if you remember this shift is actually part of a
larger Continuum you know if you think back just a couple of years ago we were talking about remote
work and remote education and AI is really just a continuation of that it's enabled by the rapid
digital transformation that the pandemic and the shift to remote
work drove which created new data
for training our models it created new surfaces for us to provide AI support to people and you
know it matur it it drove a real maturation in the cloud infrastructure that we used to bring
these uh surfaces to people and they really required that we think differently about how we
use computers to get things done um and you can see this evolution in a lot of places one place it
shows up really clearly is actually in our mission statement so Microsoft came i
nto existence as a
document company essentially we we you know we started thinking about how to help people create
and share artifacts with each other and that shows up in our original mission statement which is to
have a computer on every desktop and in every home to enable that what's interesting about AI is that
it lets us unlock the knowledge embedded in those documents via natural language so essentially
we're transitioning from being a document company to being a conversational compan
y whether those
conversations are between uh sets of people or whether it's the conversations that we have with
our computers and that means that our focus is on helping people have great conversations like
how do we help people Express their intent how do we help people best share their knowledge and
understand things deeply you know it's no longer like about the computer and where you want to
put the computer it's about the people and so that shows up in our mission statement which is to
empower every person on the planet to achieve more and this is a really big shift like I don't call
it a generational shift lightly and one of the things that's kind of fun is these big changes
they often have like a a kind of a a moment a shared moment that we all remember and I kind of
suspect that the the shared moment that we have right now is our first experience with this sort
of current generation of large language models so I want you to think about the first time that
you interact
ed with a language model I certainly remember the first time I did uh it was about a
year and a half ago in September 2022 uh which was I have you remember it was before uh chat GPT came
out it was before um you know people had really knew what uh GPT 35 could do and I was asked to
um go meet with Sam Alman who's CEO of open Ai and Greg Brockman and a bunch of other folks from open
AI to get a sense of what gp4 was like and start thinking about how to integrate that into our uh
core product
s and I had been playing around with gbt 35 a little um I had spent a number of years
trying to integrate uh AI in ambitious ways into our products um and actually I've been doing
Decades of AI research my PhD is in Ai and so I went into this meeting actually deeply skeptical
you know I sat with like my arms crossed and I'm like okay show me what it can do AI researchers
tend to be some of the most skeptical people about uh about this advancement because it's
really a big surprise what's ha
ppened and so I'm sitting in this meeting and like Greg who was
um you know running the demo of GPT 4 you know he starts showing some of the things it could do and
I'm like oh that's a nice demo but we've all seen nice demos before uh and being an AI researcher I
knew the right kinds of questions to start probing and it wasn't until after like really pushing on
the model and like seeing how it could maintain context as you iterated with it and how it could
handle ambiguity and various diffe
rent conflicting constraints um and how I could even explain what
it was doing that I started to understand the real power underline that model honestly I was blown
away I was so blown away that actually um as I was driving home from that meeting and thinking
about what it would mean to bring this into the products and like I had been worried that it
was a bit of a Fool's eron and like all of a sudden I saw it was this huge opportunity I
couldn't even get home I had to pull my car over on t
he 2m Drive pull my car over and sit
in a parking lot and I sat in my car and I SC dreamed and I just I felt this huge sense of
responsibility you know a responsibility to bring this powerful technology into people's hands
and a responsibility to do it in the right way you know uh we get the opportunity in the course
of Our Lives to sit and observe a lot of trends like certainly that's one of the reasons studying
history is important is to see what's happened and learn from that moving forw
ard but it's not often
that we get to sit right in the middle of a trend and shape it and that's where we collectively are
right now and so you know following that meeting we began a Sprint to uh integrate gp4 into our
products and I've never seen the company uh move so fast but we were only able to move so fast
because of Decades of research and Decades of thought that went into this current moment and
I'm actually reminded of a quote that I really like from the hagga kuray um I don't know
if
you've read the hagga Kur if you haven't you should check it out uh it's this collection of
wisdom gathered from a samurai in the 1700s uh in the Years before he died so his clerk sort of
wrote down his wisdom it wasn't actually intended to be published uh but lucky for us it was and
there's this one quote in there that I really like which is in the words of the Ancients one
should make his decisions in the course of seven breaths and that doesn't mean like be rash and
just make willy-
nilly decisions it means that you you should spend your entire life meditating
and reflecting on the decisions that you need to make so that when the time comes and you need to
make a decision you're ready for it and that's and that's roughly where we are now we have spent
decades studying and trying to understand Ai and its impact on work and we're ready now at this
moment to make that happen so you can imagine there's been a ton of research at Microsoft and
in the academic community uh to
try and understand what what this will mean for the future of work
um you know and in in general digital technology has um has been amazing in shaping the way we get
things done but it also creates a lot of digital debt right the pace of work has increased from the
crush of data lot overload of information and the always on communication um in fact research shows
it takes 25 minutes to get up to full productivity and yet we're interrupted every three minutes
and even when we're not interru
pted by external sources we're like constantly distracted we self
interrupt to go like check email or check social media we actually only look at a given desktop
window for maybe 20 seconds and so it's not surprising in that people often feel frazzled
in a recent survey that we ran with over 30,000 people across a variety of different countries
and different job roles uh we found that 64% of people reported struggling with finding the time
and energy to get things done and those people who
were struggling reported being three and a half
times more likely to struggle with Innovation which is is particularly important right now
is that we are innovating and thinking about new ways to do things and so the question is is
AI going to make this worse kind of increase our digital debt or is it actually going to help us
can it actually unlock uh new ways of working and new ways of thinking about things it's quite
a different technology um and so there's been a lot of research startin
g to emerge in the past
year on this um early Studies have actually shown surprising impact on uh productivity so for
example there was a study just out of MIT earlier earlier last year uh by no and Jang looking
at using chat GPT which is GPT 35 uh to write essays and so they had a couple hundred people
write essays uh with chat GPT and without it and compared it to their performance prior um and what
they found was that people were a lot faster doing the writing Tas actually quite a lot fa
ster so
the writing test took 27 minutes without chat GPT it took 17 minutes with chat GPD that's 10 minutes
shorter and then what's more is the quality of the essays that they produced actually went up with a
people receiving higher grades and so that's with uh chat GPT GPT 35 early early on in this journey
and we've been doing a lot of studying as well of co-pilot which is Microsoft's version of uh
integrating large language models into our into our products um actually it's kind of inter
esting
you know everybody understands how deeply we are in the middle of actually like figuring all this
stuff out I've never seen our customers get so engaged around doing research with us as I have
uh recently so as our early adopter are trying out co-pilot they're like filling out surveys
they're helping us study and understand things um and they're certainly reporting being more
productive the reporting being faster 72% of all people said it was less mental effort on mundane
or uh repe
titive tasks but that's self-report data so as a scientist I actually really like to
look at the behavioral data as well and how we see actual Behavior changing um so we've run a bunch
of studies on a handful of common tasks trying to look at the um actual changes in speed quality and
effort underline that and we've looked at a bunch of different common tasks typically like so when
you think about what a language model is good for language models are good at generating text so
they're great
for the sort of blank slate kinds of problems starting something new um they're
also really good at um at sort of translating Concepts or ideas they're good at summarizing
taking a long document and translating it into something shorter actually like the blank slate
generating stuff that's sort of a translation problem as well how do you take uh a few notes
about what you want and translate that into more formal uh writing and uh what we see is that
people are 29% faster overall when runni
ng on these kinds of tasks uh that they're good for and
in fact you save a full 32 minutes summarizing a meeting with co-pilot I actually use that um I
use that a lot my uh and like these tasks and the thing that I would um want to note in particular
is that these tasks are the tasks that AI is good at um you might have seen this recent paper about
BCG Consultants um about the jagged technological Frontier uh it's talking about sort of those
tasks that research suggests um are going to see
really good early wins and in that study is
well well you see similar things uh people were 20% faster doing tasks within that Jagged Frontier
producing 40% higher quality uh results so one of the things I encourage you to do as Educators
is to think about which of the tasks in the work that you do fall in within that dag Jagged
technological Frontier that you can uh that you can start taking advantage of language models and
seeing wins like this the work that you're doing right away so Lin
kedIn ran a uh study where they
analyzed where and how generative AI was going to change a bunch of different specific roles
looking at what skills uh AI could help with and what skills really required um a human or a
person uh to do and in the context of Education they found that about half of the the skills that
teachers do uh require humans directly you know and that's stuff like classroom management
or uh differentiating and instruction but 45% of a teacher's job could benefit from help
from Ai and that's things like lesson plannings or curriculum development and I want you to
think if you could figure out how to get up with that how would you use that extra time
like what new opportunities would that open up because that's really the opportunity
right here we've got sort of the skills that language models are going to be very good
at and then we've got the opportunity that that creates and so um you know that's a little bit
about how the role of the educator is going to
change but obviously the you know the experience
of our students is going to change as well um and we need to be thinking about how and what uh
students are going to learn and what skills they need to sort of thrive in the coming world
I have um four boys at the end of the K12 Journey couple graduating colle high school this year and
entering in college one in college another in high school and so I think about this a lot like what
do they need to know uh to be successful in the world um a
nd I sometimes worry that maybe we're
teaching kids the wrong skills for their future and I think that in part because when you look
across these studies that I was talking about about um where AI really makes people more
productive one of the interesting things that you see is actually closes the skill Gap really
significantly I mean and this suggests to me that we're teaching the kids the skills that
AI can do versus the skills that are going to complement AI um so that no and Jen study t
hat
I mentioned for example one of the things they found was that chat GPT reduced grade disparity
by half and the BCG study that I mentioned when they looked at the impact of generative AI on
the lower half of uh participants they found that there was a 43% Improvement compared with
a 177% Improvement on the upper half you know so what does it mean when every student is a B+
student like that is great that means our Baseline is raised but we need to think about what the
skills are that pe
ople are going to need moving forward so um when we survey Business Leaders
a lot of the skills that they're asking for are ones that help people leverage AI you know you
have to spend the take the time to learn learn how to leverage AI how to write great prompts how
to evaluate the output how to evaluate creative work um how to check for bias and understand
the answers that you're getting you know these are the new core competencies there things like
analytical thinking complex problem thi
nking uh creativity and originality and you know what's
interesting about these skills it's not like they're they're they're not they're skills
that we've always known are important they're metacognitive skills they're the skills we need
to think about how to think and as AI gets good at like the production of content and making
things happen like that's what we need from people and generative AI increases the
metacognitive burden becomes increasingly important for people to do the planning
and
monitoring and evaluation of the work that happens as much as it is to actually do
the work and so like the task becomes one of critical integration we're moving from you
know thinking about things like searching and creating to actually analyzing and integrating
that work it becomes really important to think early on how do we express our intent what are
we looking at what are we thinking about and then later on you know how do we understand the
results we see and so these are like n
ew things to be thinking about um a topic that comes
up a lot that I'm sure you've heard about is over Reliance you know over Reliance happens
when somebody you know a human Place places too much trust in the results that an AI system uh
provides and if we think about our metacognitive processes we can actually start to address
over reliance um so for example it's really great to develop cognitive forcing functions
to force yourself to reflect on the results that the system gives you know y
ou should for
example ask yourself do you agree or disagree with the content that an AI system provides
or rate your confidence in the answer those sorts of things will help you do a better
job in working with AI and so we can teach these to people we can practice them ourselves
and we can also then go build them into our tools um and as we do that it actually requires
thinking about what our AI tools are a little bit up until now a lot of the conversation around how
to use AI is like thin
king about as an assistant how does it help me write better how does it
help me do this better um and actually there's this opportunity instead to think about AI as
a provocator something that asks us questions and makes us think about things more deeply and
Frameworks that structure critical thinking like Bloom's taxonomy inform the design for AI and
help us build that critical feedback cycle into our products um you know my favorite use uh for
a large language model is actually very speci
fic about thinking about the kinds of questions to ask
so when I have a um document that I'm trying to summarize in addition to like getting the bullet
points and summarizing the document I asked the language model what questions would a researcher
interested in an AI productivity have about this article or if I'm going to write an email like
a hard email that I'm worried about how it might land I actually asked the language model to help
me think about how different stakeholders will respo
nd to that email and we're building this
into our tools as well um so if you're using teams co-pilot uh it's pretty awesome and that
it will you know we talked about the 23 minutes that get saved uh summarizing meetings it's pretty
awesome in terms of summarizing meetings you join a meeting late you can get caught up you skip a
meeting and you know what you missed um but some of my favorite features there are actually if you
look at the prompt suggestions in the middle of a meeting they're
actually intended to spark good
conversation you know there's prompt suggestions like what are the points of disagreement in this
meeting what questions should we try to answer before we wrap up and like there that's not just
helping us do our work faster or more efficiently but it's helping us actually think about things
in a new way and so like these intentional things that we build into our tools are actually what
are going to help AI change the way that people learn and think about thin
gs I mean even think
about something as simple as spell check right you know like you can use spell checking as you're
typing and going along it can autocorrect uh what you're doing and and this happens like you know
with typos you're going along and like sort of on autopilot the system will just fix what you're
doing but you can actually learn If instead you get cues in line that help you understand the
the mistake that you made you know that you can actually see that you misspelled it see
the
corrections in in context um and learn for that and just and that's that's essentially what takes
the experience from being one where you're help where the system's helping you on autopilot
to something where it's your co-pilot and helping you learn and grow and just like with
spell check we can create opportunities that encourage learning and that improve learning
outcomes versus simply finding the easiest solution uh in one of the uh very first randomized
experiments on large langua
ge models and education this is a study out of um Microsoft research in
the University of Toronto uh they we found that large language modelbased explanations actually uh
positively impact learning over just giving people the correct answer so the study gives people a
bunch of multiple choice sat style question I think they're math questions taken directly from
the SAT actually um and there's sort of two phases to the study a um a training phase and then a test
phase and in the training pha
se people are given a question like this you know this is a standard
like somebody drives One Direction at a certain speed and drives back at a at another speed what
what's the average speed that they travel at and the answer here just to make it easy for you is
B and so like some students during the practice session would see this While others would get
llm generated explanations for how to think about uh and solve these problems um you know
and actually on the left hand side what you can
see is one that's generic generated by the
language model and on the right hand side is an explanation that's actually generated explicitly
with the intent of serving as a tutor or coach uh for the individual to help them learn and what
they found as they uh as um they had people take these practice tests and then looked at the
performance on a set of test questions uh was that um explanations significantly improve the outcomes
so when they were when people were given only the answers durin
g the training session they got like
slightly more than half of the answers in the test session correct on the other hand when they were
given these llm based uh uh explanations over two-thirds of the time they got the answer correct
there's a ton of detail in the paper that's worth digging into and in including like timing of
the interventions and stuff like that um but the important thing for us is that these improvements
are a result of the intention that's built into the system the llm
is essentially serving as a
coach or a tutor for individuals so that they can learn better and we're taking this emerging
research and we're building it into our products so that our products aren't just helping people on
autopilot but that are are truly serving uh as a co-pilot and I've talked a little bit about this
in terms of things like how it how it's how we're change how we're doing that in the context of
teams and our horizontals uh we also have a set of vertical that we're thinking
about explicitly in
in education the Microsoft learning accelerators which provide students with real-time coaching and
feedback and Educators with actionable analysis and insights um there's a couple examples here
that I really like um Reading Coach is a great example it actually like uh uses large language
models to generate Choose Your Own Adventure style stories for students to read um and then use his
AI to provide feedback on like reading fluency and vocabulary um and surface that up
to to their
teachers as well um another one that's super relevant to this conversation is actually the uh
search coach because the search coach actively teaches metacognitive skills like we were talking
about like it teaches people how to formulate a query and plan what they're going to be searching
for how to identify and find find trusted sites for information and how to understand the quality
of the search results um and what's particularly exciting is that technology like this is going
to allow the learning to scale in a way that we haven't seen before um and so there was a
recent study also a case study out of Harvard that you might have seen looking at the uh use of
AI in the classroom and um you know with the goal like it was for their intro CS course which
is a huge course and it's very hard to scale um there's actually a real uh dirth of sort
of comput computer scientists to teach those intros CS classes and um what they found was that
the students were felt as if
they had a personal tutor in this class and it's like it's our it's
our opportunity and it's Our obligation to figure out how to use these to these tools to bring
personal learning into the classroom and think about how we're going to use that extra time in
new ways to develop new approaches to education and hopefully I've provided a little bit of the
sort of scientific background that we need to understand Ai and future of work for for us to do
that um you know and I think that's actually
it's actually an important point where if you think
about it there's a bunch of unknowns there's a lot of ambiguity that we're headed into right
now and a lot that we need to figure out um but fortunately there is a model for that there's
a model for that that the folks in this room are experts at and that is the scientific process
we need to be teaching our students a scientific process and we need to be using it ourselves like
we need to be leading like scientists and that of course means
developing experiments and hypotheses
and and testing them out but there's so much more to the scientific process than that it means
building on the state-ofthe-art like not making everything up from scratch reading and learning
and and and uh you know building on what others know it means sharing what we discover so that
others can build on what we know and actually also so that others can disagree with what we're
doing and it means vigorous debate and validation like that's a big part of
the publication process
is actually validation of what you've done and it means considering the externalities and thinking
about the long-term impact of what we're going to do and you know I think together we can
use these scientific principles to create a new Equitable productive inclusive future with AI
and you all are on the front lines of helping the world you know deal with this generational
shift and you have the opportunity right now to shape education for the better and I'm
really
excited to see what you learn so thank you a huge thank you to Jamie for that
session uh we're going to race straight into our next session but we're still with the
theme of AI um and it'll be k f buttfield who is CEO of the good Tech advisory K will be
taken to the spage stage next the company
Comments