[MUSIC] Speaker 1:
We're witnessing the shift of generative AI from a
niche technology in the hands of a few to an accessible
technology businesses are harnessing to
benefit the many. Now, you can tap into this opportunity to
gain an AI advantage. Get to market fast, do it right, and do so responsibly,
safely, and securely. Copilots are becoming an AI category in
and of themselves. Today, you can build custom copilots to serve the exact needs of your
customers and employees. Bringing content gen
eration to all apps to supercharge
creativity, designing contextual
experiences that impress and captivate and freeing people to focus on their
highest value work. This is Azure AI Studio, the place to build your copilots where you can confidently apply the latest
state-of-the-art and open source models easily ground responses in your data knowing privacy is protected, deliver multimodal
interactions beyond text alone that feel more natural and build on a foundation of trust through every step
o
f app development. The possibilities with
AI are limitless. Build your own copilots
with Azure AI. Let's do generative
AI right together. John Montgomery:
Good morning. Welcome. I'm incredibly
excited to be here. Thank you for the warm welcome. I'm John Montgomery. I'm in the team at the Azure
AI team at Microsoft. I'm joined by Nabila Babar. We want to spend
the next 45 minutes going a little bit deeper into some of the
news that Satya and Scott Guthrie have
talked about today. Generative AI is
actually
real. It's happening. It's happening now we have
18,000 customers who are using Azure OpenAI Service today to build amazing
generative experiences. It's everything from the
simple chat experience to the most sophisticated
kinds of applications that you see in the
Microsoft copilots, companies like
Siemen's that you saw. I think in Scott's Keynote that's building an internal
application that does translation and can issue
tickets to ISVs like Grammarly and on into even more sophisticate
d examples like the Microsoft copilots. I want to spend a moment
right now highlighting one of those customers that
we're super happy to have with us,
which is Perplexity. Denis Yarats:
Perplexity tries to revolutionize the way people access and search information
on the internet, performance and speed is
at the core of what we do. We've been extensively using Azure AI Studio for this
type of workflow where you have an idea,
you try things out, and then once you're satisfied
we can take this mod
el, deploy it, and next
day its in productions. Lauren Yang:
With Azure OpenAI Service, we have the ability
to control and independently change which
model is serving prod traffic. Perplexity Ask started
out as a slack bot. You ask a question and the
bot replies with an answer. We hooked it up to web
browsing and added the capability to search the
Internet. It was like magic. We got overwhelmingly
positive feedback and from there created the official
Ask Perplexity product. Aravind Srinivas:
The
underlying models and Perplexity Ask couples on Azure and you want
to make sure that no matter what
requests people send, the service is safe, it's secure and reliably runs. That's what Azure
enables us to do. You send in a lot of requests to the large language model API and there are multiple times
you repeat the same thing. You don't want to keep paying again and again for
the same thing. What PTU has allowed is you can cash the repeated
requests, saving costs. Denis Yarats:
The power of larg
e language models is that you can skip
many of those steps. You can prototype with
tools like Azure AI Studio, and in a matter
of hours or days, you can deploy this
feature into your product. We've been working
with Azure OpenAI to migrate to a faster version
of GPT-4 models. That essentially led to increasing throughput
from roughly 300,000 tokens per minute
to 600,000 tokens. You getting twice as much as throughput through only
two percent of the cost. This not only allow you to release featur
es in a matter
of days instead of months, but also it allows
you to experiment much faster and get much
better user experience. Aravind Srinivas:
We didn't start off as trying to work on a search engine, but rather we were trying
to just do cool things with large language
models like GPT. We ended up building
something that would be useful for basically
the entire world. Lauren Yang:
Where the first generative AI answer engine out there, and we plan on
continuing to deliver a seamless and new ty
pe of search experience and
to continue growing our product using
Azure OpenAI Service to push us forward. [MUSIC] John Montgomery:
Perplexity has done some amazing things. I think that video
calls out some of the reasons that a tool like
Azure AI Studio is useful. Their ability to flight things, new experiences versus
old experiences to be able to do testing of one
model against another. The ability to scale the
solution from a proof of concept quickly out
to incredible load. This is the same
t
echnology that we use inside Microsoft to
build our copilots. We're not just
providers of copilots, we're not just providers of a
third party infrastructure. The same technology that Perplexity used and that these other customers have used, that's the technology
that we build our own copilots on
the exact same stuff. It's all Azure AI. That means that the
advantages that we get as we scale
out our copilots, the efficiency of the service, the accuracy of the models, the scale we're
able to operat
e at, we are able to pass all
of those along to you. A huge portion of that is about
making sure that there is commonality in the
infrastructure that we use across Microsoft, and increasingly
that we're seeing customers standardize
on in their accounts. We talk about this as
the Copilot Stack, and it has these common layers
that you build up from. Here we are actually in Level zero of the
conference center. I want to say that AI is in fact the foundation upon
which we're building all of Ignite b
ecause we're on the foundation of
it, maybe the Garage. I don't really think
we'd want to give this talk in the
Garage, but we could. But thinking about it
through that perspective, there are multiple layers and as we talk about
this in this show, we are going to build from
the bottom to the top. Talking a little bit about the foundation models
that we provide, the orchestration
engine we have, the AI toolchain that we provide and how we integrate
with data sources. Mostly in this talk,
we're go
ing to be talking about how you build a
completely custom copilot. We have other talks here to
talk about how you can extend the Microsoft copilots or to build copilots from within
a product like Fabric. But this is about building your own Copilot at
scale from scratch. The tooling we used to do
that is Azure AI Studio. Azure AI Studio is in
public preview today. We announced it at our
build event last spring, and it has been an amazing
journey to get to the place today where we are able to offe
r it to you
as a public preview. I encourage you to go to
Ai.azure.com and give it a try. We have brought together all of those practices from
that Copilot Stack. All of the knowledge
that we've gained from building and operating
our copilots at scale. All the customer
conversations together to give you one place to do AI right and to build applications quickly and offer them
out to your customers. AI Studio is a new thing for us. We are truly unifying multiple different AI
services into one exp
erience. We see this very commonly where customers want to be able to use multiple different AI models, they want to bring
their custom models, many different data sources, and we want to give you that
unified platform to do that. That brings in the best of our data and search
technologies. That brings in not only our own foundation
models that we have, that we partner
with open AI to do, but the best of the open
models that are out there and the best other commercial
models so that you can buil
d your copilots using
these technologies. To do so safely and
responsibly using the cutting-edge safety tools that we use within Microsoft, and to do all of that with a full end-to-end development life cycle because that is the next phase of what's
going to be happening here. We will see the increasing
merger of DevOps processes, MLOps processes and this
new area called LLMOps, which is about how you actually
build these systems and scale them and make sure that they version
correctly over time.
With that, enough
I think of me talking. Nabila, would you
actually introduce us to Azure AI Studio? Nabila Babar:
Awesome, thank you, John. We've all seen powerful
copilots like Bing Chat, M365 and GitHub and talked
about a few of these. All of these are built on Azure, and they're some of the world's
most complex workloads. What we're going to do today
is we're going to build our own copilot safely
using Azure AI Studio. Here you see that I'm
an Azure AI Studio, The first thing
I'm going to d
o is I'm going to create a project. Actually, that's not true. I've already created
the project. We're going to use this
project as you see here, and within the Settings tab, you can see my project
configuration. Here you can see I've created something called an
Azure AI Resource. This Azure AI Resource
helps me connect to, create, and manage all of the different AI-related assets
that go into my project. For example, in this project, I'm using Azure OpenAI, I'm using Azure
Machine Learning, I'm
using Azure AI Search. All of these are
already connected, and they're ready for me to use. You can also see the compute and infrastructure that my project
is dependent on in here, and you can manage
that here as well. I have access to different
API endpoints and keys, and I also have access to seeing who has access
to my project, and I can assign different roles and permissions here as well. That's the project set-up. Let's take a look
at our deployments. As a part of creating
this project, al
l of the deployments
that I needed, in this scenario I'm
using Azure OpenAI, they're already deployed and
they're ready for me to use. I can go ahead and manage these deployments
through here as well. We're going to go into
the project playground. This is a great place
to get started. The first thing I'm going
to do in here is I'm using a GPT-3.5 Turbo model. I'm going to ask this a question that is very specific
to my company data, and it's not going to work. As you can see, it's not able to gi
ve me
an answer because this model has not
been trained on my company data. Let's
go ahead and fix that now. I'm going to add our
data using Azure AI Search. What I've already done in
here is I've taken our data, and I've added it to OneLake. OneLake is supported throughout the platform in Azure AI Studio. Here you see me
ingesting my data from it and then adding that
data to my vector index. It's supported for
fine-tuning and also within our orchestration
flow as we'll see later. But I've alrea
dy grounded this model with Azure AI Search. I can see it's connected
to the playground here. Now what I'm going to do is I'm going to ask the
same exact question. This time the model
knows the answer. The hiking shoes cost $110. We also can view
exactly where in the product documentation
the information comes from. I've not only enabled this
but let's say we asked the same exact question in a different language.
I'm using Spanish here. (no audio) As we can see, it's giving
me a response in Span
ish, even though my product
documentation was in English. Through this, we've enabled
multilingual responses. Now that we have an idea
of how this model looks, let's take a look at some of the other tools that we
have within the playground. Up until now, my system
message has been pretty basic and system messages or prompts as we call them,
they're pretty important. They tell my application
how to behave. Through Azure AI Studio, you have a wide variety of
different prompts that you can use dire
ctly within
your application. If you want to
create your own, you can type that in here as well. Also, I've been having a one-on-one
conversation with this. I'm just trying to get a feel
for how this model works. But let's say now what
I want to do is I want to ask this many
different questions. Since the model has
grounded in my data, I want to compare all of the answers from
those questions. For that, we need to do
a manual evaluation. Here what I can do is
I can manually enter several differe
nt
questions one at a time or what could be quicker is if
I import an entire dataset. I've already gone
ahead and done that and let's take
a look at that. Here you can see I had a dataset that had around
10 questions in there. I've imported this in here, as well as a list of
expected responses that I'm expecting from
the LLM to give me. I ran this, and here you can see all of the different
outputs that I can now manually view and I can also add my preferences
to these as well. Here I can see the
majority
of them look good, with the exception
of this one. . . and all of this effort
doesn't go to waste. At the top, I see a summary of the insights
that I'm building. I'm building my dataset
as you see here. I can export this
and share this, and I can also save
these results to use as a dataset later for a more
comprehensive evaluation, and we'll see that
a little bit later. With that the tour for our
playground comes to an end, and I'm going to hand
it back to John. John Montgomery:
Awesom
e. Thank you. That's a tour around AI Studio. We're going to go to
each of these areas in a little more detail. But you saw the basics
here, connecting to data, working with a prompt, doing evaluation,
selecting a model. These are the cores of LLMOps, and you can do all of
these within AI Studio, both through that
user experience, through the SEKs and
through the command line. We have a full experience
for deployment. Let's start by
talking about data. I think Arun when
he was onstage, he said,
"data is the
fuel that powers AI." I would probably be a
little less formal. I would say garbage
in, garbage out. If you don't feed good
data into the LLM, you're going to get bad results. When I talk with customers, one of the most common things I see is for whatever reason, the data is not well formatted, it's not clean, the
chunking isn't right, the embedding isn't right, and you just get poor
quality results. We're trying to make it
much easier for you to connect to data and to get it into t
he right form to do that. Within Azure AI Studio, we have connectors to all
of the Azure data sources, as you could imagine, it's very, very easy to connect to
them and bring them in. But we can also talk to data
sources from on-premises, from other Cloud
providers and so on in order to
feed these models, the information you
need in order to get the best possible
output of them, whether it's structured
or unstructured data. One of the big hearts
of all of this is this idea of vector search and
b
eing able to use vectors. We announced the whole
bunch of stuff today about vectors and
vector availability. We can talk a little
bit more later about what goes into a vector and
why vectors are important. They're basically
just mathematical representations that it's very easy for a
system to compare, so you can get nearness
and relevance out of it. But it turns out not all
vectors are created the same. We're very proud of
Azure AI Search, which was formerly called
Azure Cognitive Search, and it
s support for vectors. Because it doesn't just
bring together vectors, it actually brings
together vectors with semantic search and
classical keyword search. We talk about this as a
vector hybrid search, and it actually has a
dramatically positive impact on the results that you're likely to get out
of a search index. As you think about
the vector support, which is a core for
all of this stuff, you also need to be aware of what those vectors are
doing and how they're being brought together with
o
ther technologies to make sure that the
information being fed into the large
language model is the best possible
information you can get in. Again, I think there are some very easy ways to show that. Maybe Nabila, would you
show us a little bit about the amazingness
of Azure AI Search? Nabila Babar:
Awesome, thank you, John. Before I show you, I want to get a little nerdy. Let's talk a little bit
about how search works. John Montgomery:
That sounds good. I like nerdy. Nabila Babar:
Perfect. In e
arlier you saw me grounding
my model with my data. It's a very simple example of me grounding the
model of my data. I want to talk a little bit about how we're
searching that data. Under the hood, we're running a sophisticated
ingestion pipeline. That pipeline does three things. The first thing it does is it
cracks open the documents. It then runs a
chunking strategy, and then on top of that
what it does is it embeds the data from that
and runs it through an LLM. I was using an LLM earlier. Then
it takes that embedded data and it searches Azure AI Search. When you're searching, you have three different
options in AI Studio. The first one is
called vector search. Vector search is really good at understanding user intent and relationships between words. For example, it can understand the relationship between tent and camping, waterproof
and jackets, for example, where it's not that great is getting exact matches. For example, if you're looking
for a product document, an e-mail address,
o
r a phone number. For that, keyword search works really well.
It's fast as well. What hybrid search does, it combines both of those based on what
the user is asking. It combines vector search, which is really good at understanding
relationships and intent, with keyword search based
on what the user needs. The last search type we have in here is hybrid plus semantic, and this is the one that
builds on top of both of them. It's the one that
rules them all. What this does is it
adds a semantic rank
er, and what that ranker
does is it ensures that the most relevant information is available at the
top of a search. This is the one we're
going to try right now. I'm going to go back
to the Playground and I'm going to ask
these two questions. The first question is, what gear do I need
for Seattle weather? What I did within my
product documentation is I have a lot of documents
about my products, and then I also
inserted some documents about just generally
about Seattle to see which document
this
is going to rank and search depending
on what I'm asking. Let's go ahead and ask
this, what care do I need for Seattle weather? This is great. As you can see, the first thing is
telling me that the retrieve documents do
not provide a direct answer. It's giving me
transparency that this was not in the
retrieve documents. Then it tells me that the closest information
available to this is for any weather. Actually, sorry, we're going
to try this one more time. I promise this worked before. John Mon
tgomery:
How do you know, it's real software? Nabila Babar:
What it should do and what it has done I promise is that it is able to reference Seattle
weather and able to say that it rains in Seattle and then it brings up my
waterproof jacket. John Montgomery:
What's the irony here right now is it's bright and clear and cold. Nabila Babar:
The next thing I'm going to ask, let's go back to
earlier one, maybe. No worries. What it
does is it pulls out the right document
within here and then it's able
to add to the record, it's able to pull up the
right document that has a waterproof jacket
within here that was within my product
documentation. I probably I am not using
the right dataset. But it does work, I promise.
John, now back to you. John Montgomery:
Thank you. That's a hybrid
search with semantic. There is a session on this that I would
invite you to go to, given by my compatriot, Pablo Castro, if you
are interested in it. He goes into quite a bit of depth into how this
technology work
s, when to apply it, and strategies for doing chunking
and embedding on your data. Now if it's a garbage in, garbage out with data, one of the other core things that's going to
determine whether a Copilot is successful or not is the foundation model
you build on top of. Within Azure AI Studio, obviously we have the
models from OpenAI, GPT-4, GPT-3.5, Turbo and so on. We're now expanding
that out for a while, we've had a model catalog that can bring in
open source models, and there's quite a bit
more
going on in this space. The key for us is to
ensuring that customers have choice because one size does
not fit all in this domain. Sometimes you need
a specific model, sometimes you need other models and sometimes you
need the models to be composed with each other so that you can get
the right result. That's actually most of the
copilots at Microsoft use a mixture of different models to achieve the results they have. One of the biggest announcements
we're making today is that we are now
bri
nging GPT-4 Turbo, the latest version of GPT-4 GPT-4 Turbo with Vision and DALL·E 3 into the
Azure OpenAI Service. If you were following the news, you know that about a week ago, OpenAI announced that they have these models and obviously
we are bringing them into Azure OpenAI as
quickly as we possibly can so that our Azure customers
can make use of them. GPT-4 Turbo, a tremendous model. It delivers the same level of accuracy that you would
expect from GPT-4. The interesting thing is it's much mo
re efficient,
much higher throughput, which is enabling us to be
able to lower the price on the token based
offering significantly. I suspect this is going to enable a lot more new scenarios since historically GPT-4 has been a somewhat
expensive model to use. The quality is worth it, but we're delivering
the same quality now at a significantly lower price. DALL·E 3 similarly, is
delivering a quality of image creation that is really blows the doors
off of DALL-E 2. I think you probably
saw some o
f that. It's an amazing model. But I want to spend a
moment here talking a little bit about
one of the trends that's happening within AI and that's called multimodality. I think a lot of us
have been talking about this for a while. Mostly today when you think
of large language models, they are language models. You type in language and
you get out language. But multimodal models go quite
a bit further than that. Microsoft has been at the
forefront of creating these multimodals for
a while and we'
re very pleased to be able to
say that we're bringing GPT-4 Turbo with Vision to
the Azure OpenAI Service. This is an amazing model, it can take in video, it can take in images, you can supply it with video or images and ask
questions about them, it can summarize videos. It's an incredibly powerful
model that extends the language to language to incorporate video
and images as well. There's a whole session on
this that my compatriot, Marco Casalaina, is giving. I encourage you to
go to that one.
He will go very deep on this model and what
it's capable of. I'll stop there. You should definitely go to
Marco's session. Somebody who was
reviewing it from outside Microsoft said it's probably going to be the best
session at the show, so you should go
to Marco's session. Now I want to actually reiterate something
that Scott talked about. There is a session we're
giving on this in detail, but a lot of the questions
that I get from customers have to do with
what happens with my data when it goes
into your model? Do you use it to train
the foundation model? Is it secure? How
are you using it? I just want to say
super clearly, the promises that Azure makes generally apply to the
Azure OpenAI Service. Your data is your data. We don't use it to train
our foundation models, we don't do anything with it. When you fine-tune a model using the Azure OpenAI Service, the weights of the
fine-tune model stay in your subscription,
we don't see them. We have some of the
strongest protections around d
ata and data sovereignty
that in the industry, data stays in the region it's in and we're very proud of that. We're very proud to
announce that today we are extending some of
the announcements we've made previously
about our copilots, where we are offering
copyright protection to applications built on top
of Azure OpenAI Service. We call this the Customer
Copyright Commitment. If you follow the published
guidelines we have for how to build a
copilot responsibly, that includes how to
design your
metaprompt, how to handle data, and you use the Azure
Content Safety Service with the switches
that we document, we will indemnify you if somebody makes a
copyright claim against you. That is a huge
announcement because it's one of the biggest fears that a lot of customers have. It's really about the control over what's coming
out of these models. A major announcement
that we made today. Another major announcement
has been about fine-tuning, so you can now fine-tune
GPT-3.5 Turbo, GPT-4. Now I'l
l say the GPT-4 Turbo,
that's a private preview, so it's a limited number of customers we're
going to bring in as we bring this service up. But I want to spend a
moment talking about fine-tuning versus
retrieval-augmented generation. Everything we've
shown so far has focused on retrieval
augmented generation. This is the idea that
instead of having to go and retrain a model
and become an expert in weights and biases
and things like that, you can grab some
data and you can feed it into the prompt
along with user questions and get
answers out of it. Retrieval augmented
generation is awesome in a lot of ways. It's relatively fast to do, building an index is
relatively inexpensive, you can reindex often. Retrieval-augmented generation
in our internal tests. It's really good at adding
knowledge to the experience. Fine-tuning is a very
different thing. You bring your data in,
you fine-tune the model. It takes some compute to
create those new weights. We then assemble the new weights with the
base model
when you call the model using a technology we have called
Low-rank Adaptation. But the interesting thing about fine-tuning is fine-tuning is not the best way to add
knowledge into a model. It's a really good way
of adjusting the tone of output or the format of output. When you think about
these two technologies, they are entirely complementary. It is not one or the other. A lot of applications won't
ever need to do fine-tuning. A lot of applications
may not need to use RAG. But we're
very pleased
to be able to offer both within Azure AI Studio. Now I've talked a lot about the OpenAI models. I'm also very happy that we have an amazing assortment of other models that you can
bring into the experience. We have the OpenAI models, but you can also deploy and fine-tune any model basically
through a partnership with Hugging Face as well
as partnerships that we have with a bunch of other companies to bring their models onto your
own infrastructure. This is basically an
infrastructu
re-based play. You can go to our model catalog, select a model, you can
fine-tune it, you can deploy it. It's a full experience of
bringing open source models into the Azure AI
experience so you can build them into your
generative AI application. Today, we also announced a technology called
Models as a Service. Now this goes the next level. We have a lot of
customers who don't want to deal with
infrastructure, they just want to call
the API endpoint, and that is where Models as
a Service comes i
nto play. These behave
basically the same as Azure OpenAI Service or any of the other
AzureAI Services. We operate the endpoint. You call the endpoint, it does what you ask. We have announced
partnerships with Meta around the Llama 2 model to operate Llama 2 this way, also Mistral, Jais, and Cohere. We have their models
now that we're bringing up as part of the Models
as a Service platform. This is going to make it a lot easier for customers that
don't want to deal with infrastructure just to us
e the model without having to
think about infrastructure, which is the direction
that we want to go. But it's not enough just
to have lots of models. I always say to customers, the proof is in the pudding
and you should test. Because it turns out
every model behaves differently and they behave differently with
different test data. Now, I would say GPT-4
is an amazing model. We have tested it
nine ways to Sunday, it is by far the best model we have tested on every measure of quality. Its inferenc
ing
performance is terrific, but you know what, it may not
be perfect for your task. We're going to stand
behind what we do. We will offer a model
benchmarking system within AI Studio. We're both going to
start by publishing benchmark results that we have
run of these open models, our models along with
common benchmarks. We're starting with a subset
and gradually adding out. In the fullness of time, what we'll enable you
to do is to benchmark any model against any other
model using this system,
this evaluation
system that we have. You can bring your own
data to that as well. This is a key part of answering the question about what model
do I choose for what task. We're going to give you the
tools to make that easy. Again, I would love
to actually ask Nabila to show off some
of what we have there. Nabila Babar:
I have a small surprise for you, John. John Montgomery:
What's the surprise? Nabila Babar:
I ran it again, and here is it. Here, it is work.
I didn't give up. (applause) Here we s
ee what gear do I need for Seattle
weather and you can see it referencing the
document that we need, which is about a
waterproofing jacket, even though I'm not mentioning
rain within that document. When I ask it a question about
where to eat in Seattle, even though I have documentation about Seattle that's
specific to food, it's referencing that and
not any product documents. Here we can see the
ranker pulling up the right information based
on which question I'm asking, and it's also able to bui
ld those relationships
between words. We were using the GPT-3.5
Turbo model the whole time and an ada
model for embeddings. But let's say, as
John was calling out, that we want to use different models within our application. Here within the model catalog, I have access to thousands
of open source models, including Meta's 70
billion parameter, Llama 2 model as well. But let's say I don't know
which model to choose. With our benchmarking tool based on the task that
we're trying to do, in this we'r
e doing
question answering, I can compare a set of models. Here I have the GPT-3.5 Turbo
model that I was using, getting compared
to a Llama 2 model against a standard dataset
and I see metrics right here. We will be adding
additional metrics to this, additional datasets, and you can also
replace this with your own dataset and
benchmark those. Now let's say we want to
use a Llama-2-70-b model. We're going to go back
to the model catalog. We're announcing two
services with this. The first one is
being able to deploy this model
through a mass service. This is a pay-as-you-go service. The benefit of that
is I don't have to manage any infrastructure,
any quotas. I pretty much call an API and this model is
ready for me to use. But let's say you want
to customize this model. You may want to
fine-tune this model if you want to introduce bias
into your application. Those could be for scenarios within certain
medical terminology. If you want to introduce
within your application or legal terms f
or that, you may want to
fine-tune models. For that, we're also announcing
a fine-tuning service, that's a managed service. All you need for this is your dataset and
you're calling an API, and we're going to take care of the quotas
infrastructure for you. To see this, I've already gone ahead
and fine-tuned the model. When I fine-tune
it, I can see it within my fine-tuned
models right here. Here you can deploy this model. But before you
deploy, you want to know what are the metrics like, is this
model ready
for me to deploy? Within the Metrics tab, you get a list of metrics
that you can view. You can always go ahead, change your parameters,
change your dataset, continue to iterate on this, and then when you're
ready you can select the best model and
fine-tune this. We've talked a lot up until now about a text-in-text-out
scenario. The future of
generative AI is all around building on top of that
and adding multimodality. What multimodality
means that I can add text images and even
video
s to my applications. With AI Studio, you can do that. We're going to take a look at
a few different scenarios. In the first one, what I've
done is I've created generated descriptions
that can be either for my website or
for my brands in here. I did this by going to a different mode
within the playground. I can then build on top of
that and I can also generate different images for my
product descriptions and for my websites as well. Here we're using DALL·E 3, and I've saved the
best for last. He
re you see with the
GPT-4 Turbo model, what I've done is
I'm asking this. I've uploaded a pretty bad video actually from a
vacation that we took. I'm not telling it where it is, and I'm saying based on
this generate a half-day itinerary for me
with a packing list. What we see here is it's able to pinpoint exactly where in Yellowstone National
Park this hike was. It's also able to create
a half-day itinerary for me and a packing
list for my hike. I think that deserves a
round. . . (applause) And
that's not all. What we've also done is we've enabled with Azure AI Speech, we've enabled text-in
in and text-out. Lastly, you can also create your own natural sounding
synthetic voices using Custom Neural Voice, and these can have different
speaking styles and can adapt to different
languages as well. Now back to you, John. John Montgomery:
Cool, thank you. Never give up, never surrender. The Neural Voice is very interesting
because of course, you can attach the output of
one of these language
models to a Neural Voice and it
can speak on your behalf. I'm going to take the
next couple of sections together and then Nabila will
bring us home for a demo. I want to talk about
safe and responsible AI. Microsoft has had these
principles for a while about how we approach and how we think about safe
and responsible AI. Things like fairness
and reliability, safety, privacy, security,
things like that. But the thing is, it's
not just principles. The principles infuse our
engineering processes, t
he ways we build things. Everything from how we source data to how we train models. Also the tools that
we create that we are able in turn
to offer to you, and they're the same tools
that we use internally. One of those tools is
Azure Content Safety, which is a major addition
to our safety product line. We use it internally with our Copilots, we're
offering it to you. It is the thing that
can identify things like hate speech
or sexual content, but it's also the thing that can identify jailbreak
attempts and so on. That's our Content
Safety Service. The other big service
that we offer for model creators is the
evaluation on our AI dashboard, which is built into
the AI Studio as well. It enables you to
see where the data may be biased or where you may be having issues with the application
actually in production. Think about these
as two parts of the equation for how you
build a safe system. Again, it's not just talk, there's real software
engineering behind actually making
these things s
afe. Now let's talk about the
full development life cycle. This is increasingly
a key thing. It's not just about calling
the API and walking away. There's quite a bit
of engineering that goes into shaping the prompt, that goes into the
model orchestration, the APIs that are
called, and so on. You need a complete
way to do that. At Microsoft that feature within AI Studio is
called Prompt Flow, which is generally
available today. It is our tool for
prompt orchestration, for evaluation, and for
pro
mpt engineering. It is deeply integrated
into AI Studio. It is also in our Azure
machine learning product. It gives you complete tools
to be able to understand the behavior of the graph of models and APIs
that you're calling. To be able to
evaluate the outputs. These models are very non
deterministic and so evaluation is at the heart of
how you actually make sure it's doing
the right thing. A little coda on this one. Everything I've
been talking about actually has application
on the Edge as well
. We are beginning the
journey to link together Azure AI Studio to be able to deploy and do things
with models on the Edge. We talk about this as
Windows AI Studio. It is a Visual
Studio Code plug in. We'll be showing a
little bit of it here, particularly about
how you might fine-tune something like the
fine-tune model to be able to deploy it onto the Edge using all the power of AI
Studio in the Cloud. But to show off the
rest of it, Nabila, would you please show us
some more of the product? Nab
ila Babar:
Awesome. Thank you, John. So as you saw, the reality is a lot
more complex than we were showing within a
playground experience, and sometimes things may
not go the way you want. How do you adapt to that? Customers may want to be able to add many different
data sources. They may want to use
many different models. They may want to use many
different meta-prompts and then they want
to compare these. These are all variables that we can use in our application. But what are the tools that y
ou need to be able to see what's the impact of these
variables on my application? For that, we need
evaluation tools. What I'm going to
do now is I'm going to click open in Prompt Flow. What this does is it
takes me to Prompt Flow. The entire UI that you saw is now broken
up into these nodes and an orchestration
engine that we can see here that was powering that UI. All of these nodes, they contain code that
developers can check in. If I ask the same
exact question, I'm going to get the
same ans
wer here. I can also see the files behind each of these nodes, right
here at the top. These are files that
I can download, I can check in my repo and
I can iterate on them. But let's say I want
to develop in a tool a developer tool of my choice. If I click on "Open" in VS Code, I'll be taken to VS Code Web. What's going to happen is a development container
is going to be created for me and
it's going to be my own development
workstation in the Cloud. Here I'm going to have access
to the same exa
ct files. Now I can go ahead and
I can iterate, encode, whatever changes I make here are going to be reflected
back in the UI. Let's go back there now. Developers face a
lot of challenges with LLMs because of their
non-deterministic nature. Minor changes in the
meta-prompts can have major impacts to
your application. To do that, we need
evaluation tools. What I'm going to
do here is I have a dataset here that has a
set of questions and answers. These are same exact dataset
that I had used earlie
r. I also have a Prompt
Flow here as well. What I'm going to do here is I have two prompts that I created. The first one is
pretty simple and the second one has instructions
for it to be safer. What I want to do is I
want to see the impact of both of these prompts
on my application. For that, I'm going
to run an evaluation. To save us some time, I'm going to go ahead right
here and here you can see all of the different evaluation
metrics that we have. We have both data
science metrics like F1 sc
ores and accuracy, as well as LLM-assisted metrics
like relevance, coherence, groundedness, and
GPT similarity. For example, GPT
similarity compares the ground truth from my dataset to the output of the LLM. When we run this, we're going to see
this evaluation within our evaluation runs. Here now we can see both
of those variations and I can see aggregate
metrics at the top as well as a breakdown for each
row within my dataset. I can see in general,
the safer prompt has slightly better
metrics i
n here. If I want to dig further, I can look at each
of these variants. I can look at each
question within here. Let's say actually further, I can trace every
single API call that this application runs. I can view the input and
output of that as well. If I want to know exactly how this 5 is calculated
for groundedness, for example, and where
it's coming from. What's amazing is the evaluation run itself is a Prompt Flow. When I ran that evaluation, a new Prompt Flow
is created for me. The benefit
of
that is now I have transparency into exactly how these metrics are calculated. Further, I can clone this and customize and tweak it
for my own scenario. Let's finish off
with Deployments. Here you see I've already
deployed my flow. I can consume this
in my application. Lastly, I can monitor
this. When I deploy this to an endpoint that's
consumed in my application. Here I'm seeing now the same exact metrics for my application
that I was while I was prototyping and developing. Now I'm able to
have
peace of mind when this application is
deployed to my customers. Lastly, on the same endpoint, I can enable content filtering. Here you can see
I can filter out both harmful content from the user input into
this application as well as the output. With that, our tour of AI
Studio comes to an end. We're incredibly excited for a public preview and
we cannot wait to see what you build with
it. Thank you.
Comments