[Music] this is louis xiv also known as louis the great louis the grand monarch and louis the sun king famous for supposedly declaring but in 1685 even the self-declared direct representative of god on earth had questions he could not answer on his own questions about the ruling qing dynasty in china how big is it how many people live in the capital what can they teach us about music culture astronomy so in the spring of that year louis sent members of france's order of royal mathematicians on a
voyage that would span three continents and three oceans their task gather information that would satisfy the king's curiosity [Music] it was a journey with numerous hardships and countless setbacks but five years four months and two days later louie's answers finally arrived in the grandest of human traditions he had become curious asked a question and learned a new piece of information just like billions of people who had come before him and billions who have come since [Music] people who had
access to cave walls clay tablets oracles scrolls books the printing press libraries semaphore towers telegraphs the radio the television the betamax tape and the short-lived french national internet system called minitel brings us to today fishermen looking up when tomorrow's tide comes in careful cooks wondering when anchovies expire travelers trying to figure out how to say chapstick in turkish friends settling a bet about which team won the 92 nba eastern conference finals job seekers looki
ng to make a move and a fourth grader looking at facts about the qing dynasty for a history paper that's due tomorrow billions of king louis asking trillions of questions in hundreds of languages expecting someone to give them an answer in under one second now who would sign up for a challenge like that interesting setup yeah this is ben gomes well the correct pronunciation is gomez this is ben gamez but i i say gomez this he knows a few things about search uh that search anyway he's kind of a b
ig deal even though he'd try to convince you otherwise ben worked on search for more than 20 years but that's not where his story started so i was born in darussalam in tanzania and but i at a very early age my parents moved back to india to bangalore and there was a few books at home from my elder siblings and that's the information that access to including i remember one torn encyclopedia that i think my grandfather had given my mom so it was really out of date in fifth grade i got two present
s a bike which my parents thought i'd be very excited about and a much better encyclopedia and i was actually much more excited about the encyclopedia this is where geeks come from uh than than the bicycle uh and you know my parents don't know what to do with this when i look back at how we found information it was so dramatically different from today when my mother was growing up where there was not even access to a good library you would have just accepted the fact that you didn't have the inf
ormation and that's the way it was going to be uh when i was growing up for some kinds of information there was a decent library but you had to take this bus it took about an hour you look things up in a card catalog that took time now today we measure in fractions of a second the time it takes for you to get information i think that reduction in friction is absolutely dramatic because it can enable people around the world to have equal access to information it's not just that people in some pla
ces who have access to the best libraries everybody should have access to the highest quality information so that combination of a deep technical problem and i think a fundamental human need to understand the world around us to know more about the world around us is the heart of google search and what keeps me coming to work still so excited 20 years later so in the early days i wondered whether the company had the infrastructure to be a real company because when i had come from my interview act
ually there was not even a sign indicating that this was google so i was not sure i'd come to the right place but halfway up the staircase there was a small neon sign that said google so that's when i knew and it generally felt completely chaotic and jeff was there jeff is also brilliant yeah we were a very small company we were maybe about 25 people we were all kind of wedged into this second floor area in downtown palo alto i was in an office with boris hutzler urs was in charge of all of engi
neering and at the time i don't think i knew how to pronounce his name but he put the three of us named ben in one office just so people could walk by and say hey ben yes we had the ben pen i think it was pure coincidence uh actually my first reaction to google was i have no idea what search is so it's probably not for me but then i was intrigued by the problem it was clear that there was some real value there because without sort of really good ranking all that growth of the web would be wasted
if nobody could actually find the things that were there so one of the core aspects of such is how do we rank results and how do we find the most relevant information so a lot of people work on that you'll get really good stuff on this from pandu actually yeah okay this is pandu nayak hi i'm fondue head of search ranking his personal motto no query left behind before working at google pandu worked at an artificial intelligence lab at nasa yeah we built an autonomous system that provided high le
vel control to a spacecraft called deep space one really the most exciting thing that has ever happened in my life my professional life i guess after doing that he wanted a new challenge i oversee the ranking team so ranking is important because if we simply return the million pages that match your search query that's not particularly helpful and so we need to rank the pages that you might find useful hopefully these are at the top of the results we're really trying to bring information to the w
orld at large and make it useful so people can improve their day-to-day lives and i feel really lucky to have the opportunity to work on this mission let's go back a bit summer 1999 room 300 and something in the gates building at stanford and these two guys larry and sergey who are about to announce something so big it merited matching polo shirts what is our mission so how's google different basically we want to organize the world's information and make it usually uh universally accessible and
useful 20 years later bigger stage same deal and today our mission feels as relevant as ever so what does this actually mean here are a few takes i think if we weigh up the various parts of the mission to me the most important piece is organizing there are hundreds of billions of web pages that are out there our job is to filter through that and to really give you what you are looking for at that moment in time and then the next part is the world's information so information means really anythin
g you know it started out for google with web pages but it's so much more than that whether it's physical books that we need to scan or maps that we build of every place on earth that's information too and it's not web pages it's the kind of stuff that we organize today and then i think that word universal is important because universal means for everyone whether someone that can't see whether someone that can't hear people that speak different languages really make it accessible to as broad a s
et of people as possible we might be goofy people who come to work in t-shirts and and desperately need haircuts and things like that and we may not look super serious but we know how much people rely on this and we take that mission really really seriously it sounds like the mission is pretty important to these folks but here's another important question so um how would you explain how search works right yeah so how does search work how search works um how search works in a nutshell [Music] thi
s is server rack 3349 it lives here in ballybane ireland along with cows a golf course and kavanaugh's auto accident repair center this is one of the places where search happens search is a big piece of software that takes the words you type in here and looks for them here on the world wide web it can do that because first it downloads a copy of the entire web scans it and makes a list of all the words and lists of all the pages each word appears on [Music] it's like the index of a book except 1
0 trillion times longer lasagna appears on 59 million of those pages when you search for lasagna the software puts these pages in order with what it hopes are the most useful at the top and less useful at the bottom most people searching for lasagna want a recipe for lasagna look at how delicious that looks some people want nutrition facts for lasagna and a few people want to learn about the life and research of lewis c lasagna md they call him the father of modern pharmacology the software livi
ng on server rack 3349b helps rank those pages depending on where you live whether the page was updated recently how many other pages are pictures of lasagna it does all this in less than one second billions of times a day every day mostly for things that are tougher to figure out than lasagna so behind the scenes of google search there are many kinds of engineers and many different teams that come together to bring you the search experience you see teams around the world in many other countries
europe london india japan so on we have teams that are working on the interface by which we present this information teams working on the evaluation process processes that make sure that the changes that are happening are good changes and then there are teams of engineers who work on ranking they might examine the kinds of queries where we're not doing well today and think about what are the kinds of techniques we could use to enable us to do better in the future like the team that's about to e
nter this meeting anything else we need to know don't look at the lines with the camera okay all right let's do it despite their lack of on-camera experience they're working on what could be the biggest change to search in over a decade but we'll get back to them later but we will actually see something so search is a pretty complex product it's a big effort to actually make these things work to take all of these different pieces of the system using a lot of mathematics and then trying to bring
them together into something more real into something that can actually be turned into an algorithm all right so behind the scenes people at google are working on algorithms let's dig into that for a minute at its most basic an algorithm is just a set of mathematical instructions that a computer follows kind of like a recipe just like there are different recipes for different dishes there are different algorithms for different jobs some make elevators go up and down some predict subway delays so
me help cars park themselves the google search algorithms exist to return high quality information based on a user's query stuff like all of the text pictures videos and ideas that people have taken the time to put on the open web stuff they want other people to find and read and watch and look at and learn from whoa hey guys lucas here in today's video i want to show you how to simplify a rational expression we're talking about the angle of the leaning tower of pisa how to his 710 split whateve
r this thing is this is the information that google tries to organize and make universally accessible and useful because this is the kind of information that people are out there looking for but you know what they're not looking for spam not the delicious kind the bad kind yeah so let me just talk about spam for a minute because spam is one of the biggest problems that that we face this is kathy edwards head of user trust for search which basically means she deals with a lot of crap so the rest
of us never have to broadly spam is what we consider a low quality page that is artificially boosted in our results she's talking about pages that use ai generated nonsense text hidden keywords and hijacked urls to trick their way into people's search results pages like fastcashonline.org topicalarticles.info the kind of websites that when you end up on them you hit the back button as quickly as possible because they're sh because they're spammed there's a wide variety of motivations why people
do this sometimes it's commercial interests spam where they're trying to sell things that are a little bit dubious right or you know sometimes it can just be to capture more of the user's clicks and that's not right that side is not getting those links organically it dilutes the value of that signal it makes it even harder for us and it makes it harder for users to find great information it's a very very hard problem because the people on the other side are very motivated to succeed and they're
smart too and they have resources and they're working on it also like we solve one part of it and they they adapt and they do something else and that's the reason that we keep google search algorithm a very closely guarded secret recipe for coke level guarded secret because if we talk about signals too much then people will manipulate them and that breaks search entirely fighting spam is a cat and mouse game it's not something that i think will ever be solvable as an example 40 of pages that we
crawled in the last year in europe were spam pictures this is a war that we're fighting basically so yeah people at google hate spam which is one of the reasons they're always making changes to search to keep spam out of your results and to keep high quality information in okay so you've got the search engine and it's working and by all accounts it's working better than any other search engine has worked before and every day you see millions of queries and clearly users are happy but as engineer
you ask yourself how can i make this better you see many ways in which we are still failing and you see a ton of opportunity for us to make it even better and over a period of time the developments we've made in the search engine have had a dramatic impact on how well it actually works for users no no i don't think we had that particular problem even though we've launched a whole series of changes over the years that have i think meaningfully and materially improved the search result sets i'm h
ere to tell you that search is far from a solved problem in fact there's actually no end in sight in terms of when this will actually be solved because the world keeps evolving we're coming up with new devices we're coming up with new ways of interacting with information we're coming up with new information sources like videos and so on that are adding in new opportunities as well as new challenges the content on the web has changed users have changed what they're searching for and how they sear
ch for example 15 of the queries 15 of the queries 15 of the queries we see every day we have never seen before that's just going to keep happening and we're going to need to constantly evolve to keep up it's a little bit like the red queen says to alice in alice in wonderland you know you need to run as fast as you can to stay where you are you know we're gonna add some friction we don't actually think we have good results the idea is to add friction for the worst of the worst results to start
with we change the search algorithm on average six times per day it's actually really frequent however to get to those six launches per day roughly you know a couple of thousand launches in a year we're doing 200 to 300 000 experiments so the vast majority of changes that we think about making that we might try actually fail imagine you have a smart engineer on the team right and they come to you and say i've got this great idea on how to improve search and you talk to the engineer then they com
e back a little while later and say okay i've got the change can i launch it and you're like no you can't launch it you've got to prove that this is actually good proof comes from data data comes from experiments side-by-side tests where results from the current version of google search are compared to the proposed version if the proposed version gives better quality results aka links to better quality websites and it gets closer to being put into production which is a fancy way of saying actual
ly in use by people around the world which brings up a question who decides what makes a better quality website those people that we ask the question of which is better a or b are known as search quality raters the people at google aren't deciding what's a good result from a bad result the people at google aren't determining what results to show for any given query but rather the raters are basically teaching our computers what's good and what's bad is this a high quality result is this low qual
ity result and they are trained on what are called our writer guidelines the search quality evaluator guidelines are a 168-page document establishing what makes a good search result good we're talking about websites exhibiting expertise authoritativeness and trustworthiness these words are given clear detailed definitions so the thousands of independent evaluators keeping an eye on search know what they're looking for want your website to show up higher in search read the guidelines seriously th
ey're publicly available and the more people that read them the better the web could be for everybody all right let's get back to ben making changes to search is a bit of a balancing act there are many different things you're trying to balance together quality freshness relevance but we also have to balance the performance some ideas may be really good but they may result in search that takes a lot longer so we have to be careful that we are not making search slower in the process of giving you
slightly better results in some ways the key innovation of bird yeah and what will latency does this introduce new latency or the distilled models pretty quickly yeah i think this in milliseconds or so seems like a reasonable trade-off for this level of win yeah right right from when we first started really we were focused on how can we make search run very fast so we respond more quickly with better results to more people every day every week i did a search a couple days ago complicated thing 3
00th of a second i mean it seems inconceivable you can do all that that quickly i guess we are about finding the world's information and bringing to your fingertips the second you ask it um in fact less than 0.5 seconds seems incredibly difficult and yet that's an area that works reliably 24 hours a day 365 days a year around the world but how are you going to look up an index that goes to the moon and back several times in a fraction of a second i don't know ben maybe we should ask the expert t
his guy you saw earlier this is auras hotslay he manages the technical infrastructure at google this is ours holdslay in 1999 when he also managed the technical infrastructure at google my first business card actually said search engine mechanic because my job was fixing things that were broken and the problem was hard because really everything was broken and it was just about fixing the thing that's most broken to people sometimes the internet seems kind of like it's nowhere you know i'm using
my phone and then here's wireless and i don't really see anything but when it comes to search engine when it comes to the data center these are really physical big machines so to speak a data center actually is conceptually very simple it's a building with lots and lots of servers and that's really it [Music] so in dublin we have one of the data center campuses uh it's actually one of the smaller ones i think it's the smallest data center we have we're considered the baby data center of the flee
twood quite small quite the snowflake actually this is this is quite big for any other company this is bewildering this is not a thing this is philip kevin james daniel petra and the crew we hired to film and this is where they and all their co-workers work the google data center in dublin ireland the scale of what we do here can be kind of crazy millions of searches a day goes through those machines that's why they're very loud and they produce loads of heat that means they're constantly workin
g constantly answering your queries and so how do we really store the web so dispute the way to think about it is we take the internet download it index it and chop it into small pieces and then each server has a small piece all of these servers for that data center work together to each search their little part of the internet and it literally takes millions of servers and hard drives to be able to support the world's website so each of these data centers has a complete copy of the web so if yo
u're in france or if you're in south africa you're not sending a query that goes through the wires underwater cables and comes to mountain view ask that question then we send it back and that's just not possible that's never going to work as a solution that's fast how it actually works is you're going to google and you type in a search then we direct your query to the data center that is closest and so that's actually the reason why we have data centers everywhere because we want to be close to
the users that we're serving because that's the only way to get you the most accurate response as fast as possible [Music] so there's a lot of expensive equipment here huh yeah how does that all get paid for i have absolutely no idea i guess it's from advertising ads keeps the lights on and probably puts gas in my car at the end of the day all right yeah i think we might have to talk about ads a bit here any last thoughts before we cut keep it sweet all right ads why are there ads two reasons on
e ads keep search universally accessible no paywalls no subscriptions no you've used your last credit want to buy a 50 pack just search that's free for everyone and two ads help people want to buy a thing find people who sell that thing like bart here hi and his employees at car hardware in pittsfield massachusetts yep we sell 38 000 items like weed whackers tack hammers wrenches and m10 metric castle nuts i think the only thing we don't sell is milk and bread bart buys ads on google that only g
ets shown when someone near their town pittsfield searches for instance lawn mower dealers near me and google only gets paid if the person doing the search maybe your neighbor or your brother-in-law clicks on bart's ad which is always labeled at it helps people find mowers to buy and it helps bart in the store get business and it helps pay for all the stuff that keeps search and maps and docs working and free [Music] that's why there are ads since i've been in google and worked on search for the
last 14 years i have to say that no one absolutely no one comes to me and says you know i did this search and the results were great nobody says this they only called to complain that they did something and it didn't work and the name of the man who's been collecting google's dumbest search mistakes for the last 14 years [Music] senior software engineer eric lehmann eric take one parker over the years i've been gathering some of my favorite uh bloopers sort of walk you through some of those so
how far from the coast is cambridge massachusetts um it's actually a little over 3 000 miles from the west coast how many calories in 330 tons of butter so this caused an overflow error and we said about minus two billion [Music] what color is green that's a tough one um blue sure search meat nutrition facts we brought up all kinds of detailed information i think it's quite good it's the query's a little ambiguous because it doesn't say what kind of meat and so the system chose roasted muskrat y
eah avogadro's number is a sort of important uh constant in chemistry um it's also the apparently the name of a restaurant and so we've given a lot of chemistry students their phone number is that what you're shooting for or okay since you started watching people have done over 100 million searches enough results to fill 27 libraries but none as cool as this one this is the westin library on oxford's campus two buildings down you'll find the office of dr john paul gabriel a professor of early mo
dern history he specializes in the history of information and archives suffice to say he's an expert on this stuff it used to be before say the 16th or 17th century that if you were reading a manuscript copied by someone perhaps someone you knew perhaps someone who you didn't know but they were recommended to you by someone else you could have a certain trust that the text you were reading was stable was authoritative was right and printing changes all of this sure printed word can flow everywhe
re but that worried lots of people because for example if we don't know who printed it well then what should we think about this information if there's an error in the printed word then everyone will get it wrong so we look now actually the print revolution which we used to think about almost in a celebratory way and we think now that actually the anxieties that people had about print in many ways parallel the anxieties that people have today about fake news about origins of information google s
earches an index of what exists and so if that content is out there sometimes we can surface it that can present results that are accurate when it comes to the content of the web out there but not accurate in terms of what the truth actually is but that can result in some what i would consider to be reprehensible or really offensive results a few years ago people were pointing out that for some queries like did the holocaust happen we were giving people results that had the words and were on the
topic but were from low quality sites and we viewed this as a pretty profound failure this is clearly bad because this is clearly a case of misinformation because you know the holocaust did actually occur and so then we wanted to understand why it is that this was happening so we take a very algorithmic approach we did not go in and say oh for this query we're going to change the results the fundamental reason for that is every problem that is reported to us like this is usually the tip of the
iceberg and it's usually just a representation of a whole class of problems in this case problems of misinformation and just solving the specific problem that was reported to us does not solve the large iceberg of problems that were not reported to us part of the reason why we were all in search is because we want to give you know good results to users we want to make their lives better by giving them good information this was contrary to everything that we wanted as employees in search right in
a very egregious sense it wasn't just you know a misspelling or something like that and i have access to it every query is going to have some notion of relevance and each one's going to have some notion of quality and we're constantly trying to trade off which set of results balances those two the best but if you type in the query did the holocaust happen higher quality web pages may not really bother to explicitly say that the holocaust did happen right they're talking about the holocaust and
sort of taking for granted the fact that like we as informed citizens are aware that the holocaust happened because we learned about it in school and so on and so the only kinds of websites that are actually going to have the combination of terms that seem to closely match a query like that might be ones which in fact say no the holocaust didn't actually happen it's all a big hoax right those results are not the high quality results they tend to be lower quality even though they're more relevant
and so what was happening on the did the holocaust happen to queries is that the relevant signals were overpowering the quality signals to a degree that was resulting in low quality results for users we've long recognized that there's a certain class of queries like medical queries like finance queries in all of these cases authoritative sources are incredibly important and so we emphasize expertise over relevance in those cases so we try to get you results from authoritative sources in a more
significant way and by authoritative we mean that it comes from trustworthy sources that the sources themselves are reputable but they are upfront about who they are where the information has come from that they themselves are citing sources and so the change we've made in the case of misinformation is to change the ranking function to emphasize authority a lot more and this has made all the difference misinformation is one of the challenges that comes with helping people find what they're looki
ng for but it's not the only one launched in 2010 the autocomplete feature has saved millions of hours of people's time by guessing what they're searching for before they finish typing but when those guesses have been wrong it's led to some pretty disturbing predictions [Music] a few years back we started hearing from people that sometimes folks were typing things into autocomplete and they would be sort of shocked by some of the predictions that they were getting you know autocomplete was desig
ned to help people complete their searches faster instead we were actually returning them information that they weren't searching for what we provide you with something that's shocking that's not relevant we really at that point not stood up to our core principles i think i and all the members of the team felt a deep personal responsibility to try and develop the systems to minimize these kinds of occurrences as much as possible okay first we developed a set of policies that say what kind of pre
dictions that we would not want to offer to users things like violent content sexually explicit content hate speech but we also publish those policies that way people can see where we stand and then that gives us some accountability with these auto complete algorithms we try not to surface predictions that violate the policies now these algorithms are very good at what they do but they're not perfect and every so often we'll get some predictions that in fact violate them so you can report if you
've seen a prediction that violates those policies and every day we get flags from our users out there to tell us where we might be seeing problems in the product we use those reports to improve our algorithms to try and see whether we can address the whole class of problems that the report might be just pointing towards but one thing that i would like to emphasize is that this in no way prevents users from searching for whatever it is that they want they're absolutely free to do that think abou
t it this way search is like a door that leads to the web with autocomplete it's the kind of door that senses you walking towards it and opens for you but if you're typing a query that violates its policies the automatic part stops the content of the web is still behind the door but you won't see any results until you complete the query yourself search isn't perfect we do make mistakes and we make more mistakes than we would like to make but we need to learn from them we need to get better and w
e need to continue to improve to avoid those cases in the future each time that something happens where we become aware of that result we use that as learning we use all that feedback to continue to improve and make sure that google you know one day from now five days from now 10 days from now 10 years from now is continuing to get better many people tend to think that search is really easy you type in a few words you get a few documents and the process feels very easy and in many ways that's th
at's what we want to achieve we want search to be very easy for people but behind that is an extremely hard technical problem of actually understanding what people mean when they type in a query not just matching words but actually understanding language much better over time so that we can match the thing you asked to the concept that you're really looking for in the documents and we can bring these two things together it's an absolutely fascinating problem to work on because it lies at the fro
ntiers of what computers and computer science can do and our understanding of basic aspects of how we wish to interact with computers as human beings as long as there have been machines humans have tried to get those machines to do more of course for most of history machines couldn't speak human so humans had to come up with new ways to tell machines what to do joseph jacquard used cards with holes punched in them to tell his loom put the thread here and here and here it made weaving complex pat
terns easier punch cards were a big idea they're how early computers took instruction did math solved equations holes punched in the cards represent data to be placed in the computer then computers got screens and keyboards but you still couldn't talk to it like you talked to a human you had to write in code see colon slash carrot smartderf.exe once search came along things got a little easier you just put in the words you were looking for and google came back with websites but you were still wr
iting in code ice cream shop27705 when real you meant where can i get some ice cream around here as we understand language better you should be able to ask a question in a much more natural way what time is tonight's match on who do i call for a tow truck around here does anyone make a nail polish that's safe for dogs so rather than you having to craft keyword ease that the search engine can understand we want to be able to understand what you had in mind in the most natural way you can express
it so that we can satisfy that information need with information that that we have available we call this problem natural language processing so where are we in the space of solving this problem [Music] i think we've come a long ways but the journey is so long it's very hard to see where it ends right i mean we began to work on this problem 19 years ago with the system that i worked on called spelling correction we got beyond that to understanding synonyms and how words are related to each other
but to go deeper we needed a different approach google has been doing research in something called machine learning for almost a decade and jeff hinton was the forefront of that please welcome jeffrey hinton the engineering fellow at google when jeffrey hinton began work in the 1970s people said artificial intelligence was the stuff of science fiction today he's revolutionizing how we live jeff hinton combined forces with jeff dean at some point and we began to see these huge breakthroughs in m
achine learning if you look at the last say eight or ten years machine learning has gone from a small part of overall computer science research to something that is now affecting many many fields of endeavor and we realized this could pay off in a big way in helping us do search better what kind of impact do you hope deep learning has on our future i hope that it allows google to read documents and understand what they say and so return much better search results a few years later a new developm
ent in natural language processing was announced they called it bi-directional encoder representations from transformers it's a bit of a mouthful so we just call it dirt research like this gets us closer to technology that can truly understand language so it's a big deal for search at least it could be which brings us back to this team from earlier it's going to be up to them elizabeth jin cow sundeep eric and a few other folks to figure out how to get bert working in search they name their proj
ect deep rank after the deep learning methods used by bert and the ranking aspect of search and also because it sounds cool it's cool so i think we're finally getting going here um one of the things that we can do today is talk through some of the new evals when i first joined the project i got really really excited thinking this system is doing something pretty special that most of our other systems in search probably can't do we are still at the very early stage of building a search system whi
ch truly understands human beings but this project is very unique in the sense that this is the first time for such we have a signal which understands the relationship between different terms that's why we are very excited about deep rank because we are hoping that this could help us make google search more intuitive to use and make it feel like google search actually understands our users that is like the most ambiguous so people use language every day we don't even really think about how we yo
u know put sentences together uh this is tremendously subtle things slight changes of wording can change the the meaning of what we're saying and it's very hard to write a computer program that captures all of that that subtlety so it's actually sort of interesting early on in information retrieval which is kind of the science behind search people would tend to just sort of give up on these things so like a lot of little connector words they simply ignore them they call them stock words you just
throw them out i think we've learned over time that that those words often have uh sort of an important role in communicating what we're trying to say communicating an idea and so through machine learning systems deep rank we hope to sort of pick up on these sort of subtleties of language that humans get so naturally but are so difficult to program so hopefully people will be able to phrase search queries in a sort of a more natural way for humans and not suffer from this problem that machines
don't get the subtleties eric makes it all sound pretty straightforward but actually getting burnt to play nicely with search it's not going to be easy i mean these all look like the queries where we would expect to see winds from deep i would have guessed that the team starts by testing their theories months go by progress is slow and it's not trying to make a distinction in that ring so i'm just not that thrilled with this part of it you know the change that is so positive and so powerful uh t
here is a tendency to feel like oh we should just get it out there as soon as possible and so you have to temper that with some pragmatism there if this is where your is win is coming from that's not super thrilling let's put it that way for each result that gets better others are getting worse single term queries are also way more negative when we don't know what we're doing we're doing great each failure requires a new test each test requires rewriting big chunks of code they don't have all th
e time in the world even just experimenting with a system based on bert takes thousands of servers crunching quadrillions of numbers so dprank needs an enormous amount of computing power google has tremendous resources but even by google standards this is a lot we have enough tpus to launch deep rank but barely [Music] if they don't show progress soon the resources will go to some other team with a more promising idea it'll all hinge on getting a strong quality bill let's put it back yes because
if we don't get that then we're not getting the resources time is running out i would say in general on many of the examples i see when we have optionalization on both sides this is actually some place where or deep frank typically does better but once we mix in the localness so we have these high-level measurements that we do to say whether something is good or not because if something's not good for people searching for google we are not going to launch it period i mean no matter how great th
e technology is so this was the week where we saw some really nice experimental results and that was that was so reassuring i would like us to go through some wins so one of my favorites is what temperature should you preheat your oven to when cooking fish okay i was kind of fascinated with this one it is a tough query holy cow that's really really nice here's what they're so excited about without deep rank the google search algorithms were surfacing some good information about cooking fish but
they were also getting confused showcasing a recipe for baking cookies when deep rank was tested on this query it understood that the result was about cookies reducing the prominence of the incorrect recipe and is dead elevating useful relevant information about cooking fish these are the kinds of wins the team will need to see more of if they want their project to launch and start improving search results for billions of people around the world however before we can launch we need to get launch
approval it's a formal process where any change to search gets a lot of scrutiny so i'm feeling a little pressure to like yeah i don't know yeah launch committee um so launch committee is essentially the final review before you actually choose to launch a project i mean i feel like that we've seen that so when you go to launch committee you're essentially saying hey we have a project that we've built we have all this data that we think shows that it's a good thing and now we're getting approval
to actually put it into production there's always a little bit of anxiety because you know the outcomes of these meetings are really important to people people a lot of work into them and i had to have a change rejected is is uh pretty dispiriting before the meeting i always feel like there are things that i forgot to catch so i was going over the launch report again trying to see if there is anything i'm missing there's lots of stress but also the hope right like okay no matter what we will ha
ve some reasonable feedback from the launch discussion it may be over or it may be like approved and then we can launch it it's regardless it's a big milestone jin cow has every right to be nervous around here launch committee is known for killing experiments because despite their best intention despite the months of work that went into them most experiments never make it out of the building if you talk to the average engineer they will have their share of war stories of of moments that have bee
n incredibly frustrating for them but the flip side of that is you know there's not many products that are more impactful than google search so when you can ship something that's really great it's it's really an amazing feeling [Music] all right are we ready so we are here to get launch approval for dprank okay launch committee is the meeting where we all get together look at them at the metrics and argue with each other that's not what this is saying this is saying when site diversity increases
generally speaking the engineers don't present their own work so let's take a look at the live experiment they're there often for contacts to answer questions but your work is presented by an analyst because we want the analyst to be an impartial third party because it can be a little tough there is a slight issue in the way the metrics are calculated it's important to realize that most of the changes we make in search are not ones that are 100 good right there are always wins and losses there'
s only one thing which is positive but it's not out of the noise actually the one that i think is particularly worth looking at is the long tail set right yeah let's look at that and so one of the things that the launch committee is doing is to weigh these wins and losses wow it's pretty clear from the wins and losses there are some interesting relationship understandings going on in here however deep trying illustrates some really nice wins we get from understanding language and the nuance of l
anguage this is my favorite when can you get medicine for someone pharmacy it's a very beautiful natural language when you see yeah it's an important question right can you pick up medicine for somebody else um and this is wonderful brings up this uh very relevant very specific result you can imagine why this happened yeah because before all those words like for and maybe you and get they're all stoppers largely ignored and now like because of bert you'd actually understand that those are very i
mportant to me yeah yeah yeah yeah yeah but for someone is a really hard concept yeah yeah yeah we saw some winds that were really really beautiful in various ways so point two is i think one of the biggest changes we've seen in a long time because you're getting towards more semantics and all over here when you that's all you have you don't have other signals right and so this is where it can uh can excel yeah all right this seems like a great launch really excited about this yes when it's all
done the coordinator of the launch meeting just changes a field in a spreadsheet changed it from blank to yes it's a very momentous occasion this was a very positive launch meeting uh the decision is to launch deep rank i thought i wasn't feeling nervous but when the moment came it felt so good to get that approval thanks guys awesome hey pretty darn cool yeah so after a launch you might imagine there's some great big celebration more typically uh people stand around the uh the meeting room a li
ttle awkwardly for a few minutes and say hey good job and then kind of generously shuffle back to their desks and try to catch up on on on life um and probably that'll happen here maybe we'll do something a little bit a little bit more in this case it was a pretty remarkable project congratulations in the moment this approval feels big it feels significant but in the grand scheme of things it's just another step forward an improvement just like all the others that came before it that helps make
search a little bit more useful than it was yesterday i think there was a promise there of [Music] something solving the search problem is not easy uh that's for sure i mean we've been at it for uh 20 years and i think there's still a lot to be done humans have more access to information than at any other time in history and i really feel like it's our job to make sure that they're connecting with the highest quality the most authoritative the most relevant information for them and that they're
really able to access the information that makes a difference in their lives this is sort of a core value and we feel deeply responsible to our users to make this happen what is google in 20 years you know it's very hard to predict the future i would never have predicted 20 years ago how google looks today the mission will still be there like making information accessible to people and i think the thirst will still be there that people really want to find the things that they're looking for info
rmation really releases things that are in people's potential it enables them to make decisions that they couldn't make before enables them to to to know about things that they couldn't know about before to know about things in the world to know about the people around them and i hope it also improves their understanding of the world around them as they do that and i believe that our role in search is to actually help serve that curiosity in people to help them find that information that they're
looking for that takes them on the next step of that journey of curiosity [Music] all kinds of people on all kinds of journeys curious about the thing holding them back curious about the thing pushing them forward people searching for themselves and their families just like people always have and always will and while that curiosity lives on in us i think our job here in search is never done you
Comments