The Future of Generative AI: Transforming Education, Work, and Society

CASSIE MCGRATH: Hello, and welcome. This VPAL Signature Event is brought to you by the Office of the Vice Provost for Advances in Learning in partnership with the Harvard Alumni Association. Thank you all for joining this afternoon. Join our panel of esteemed experts for an exploration of the future of generative AI and its implications for education, work, and society. Our panelists will share their expertise on some of the latest technological advancements and explore their implications. They

will also examine the ethical considerations of AI and how it can be used for good. And now without further ado, Bharat, the virtual floor is yours. BHARAT N. ANAND: Thanks, Cassie. Thanks so much. Good afternoon, everyone. Thank you for joining us. We're going to jump right into this topic. So I will say rarely is there a topic which attracts so much attention and about which we know so little in terms of how it's going to play out. But we're very fortunate to have colleagues today who join us

who will dare to speculate about the future, but also bring relevant expertise to the table. All four of my colleagues are folks who've worked with AI tools and algorithms in different though related areas. All are colleagues at Harvard. All have been gracious enough to spend this afternoon with us on the topic of generative AI and its implications for a whole variety of things. And we'll touch on four. Research, education, work, and society. So thank you, Rebecca, Dustin, Latanya, Vijay. Let me

just start by trying to level set around terminology. And I don't know who wants to take up this one. Just to set the language straight, there's generated AI, there's ChatGPT, there's GPT-3, 4, there's open AI. What's the difference between all these things? Vijay, do you want to take a crack? VIJAY JANAPA REDDI: I'd say that generative AI is the key word, right? ChatGPT is an instance of generative AI specializing in natural language processing as an example where you type and you're able to a

ctually converse with this agent. But talking about even just ChatGPT as an example, there are multiple versions of that sort of an agent. ChatGPT-4, which just came if you haven't played with it, is the ability to understand images, text, and so forth. And there are multiple models of information that it can actually handle, whereas the old version understands just raw text and does NLP. I'll stop there and let others chime in too. BHARAT N. ANAND: Anyone would like to add anything to that? DUS

TIN TINGLEY: Yeah, I also think it's important to understand that there are a lot of companies getting involved in this space and have been involved for a long time. OpenAI is the one that's been in the headlines recently, but Google's been a player in this generative context for a while, and in ways that you have probably already benefited from, but just didn't know. So sometimes you'll see if you're using Gmail, it will suggest how to complete your sentence, which I love, because I hate typing

. So it's been around. There are lots of companies that are using and building out these types of capacities. OpenAI's just one of them. And obviously, they've been partnering with Microsoft and whatnot. So it's a big and exploding space and a lot of companies that are trying to figure out their business models. BHARAT N. ANAND: Just to summarize what both of you said, the distinction is between companies of which OpenAI is one example. The software model, so to speak, AI-driven, which is GPT-3,

GPT-4, different versions. And then ChatGPT is sort of this dynamic chat bot, right? It's like the Google search bar and search as opposed to search more generally. OK, that's great. What I'd like to do is actually, there's three questions as part of a short poll that Cassie, if you can just post in the chat, we'd love for all the participants to fill out, just to get started while we engage in conversation. All right. So for the panelists-- and by the way, this link in chat will take you to so

mething called Poll Everywhere. And there's three questions in that poll if you can fill it out. So back to the panelists. Each of you has been doing work in AI and related areas for a while. And yet, there seemed to be sort of a big deal moment last summer. And I'd love to hear from you, what was that big deal moment, and why was this such a surprise? Dustin, I remember you came to me in the corridor and said, there's something different here. Can you just share with us? And then I'll turn to L

atanya and Rebecca and Vijay. DUSTIN TINGLEY: Yeah. I mean, I think my something different even started a little bit before last summer when something called GitHub Copilot came on the scene. And it was probably a year before. So GitHub is a service where people like me and Latanya and others can write code and put it up, and we can have people collaborate. People can make recommendations. And it's been a really helpful service. So they come along and release something called Copilot that was a

generative AI, and it was useful where you could tell it the code that you wanted to write, and it would write the code for you. And that was like an aha moment, A, because I'm not the best coder. I always put the comma in the wrong place and things don't run, and then I lose my hair. But it also made me think, jeez. Like, what did they exactly train their algorithms on? You have to train these algorithms in order for them to generate content. And it was a real puzzler about how they did it. And

they got in-- they started to get into a little bit of heat about, were they using code that people like myself had put up on GitHub to make available to the world in order for them to build a product? And that was like, huh. Where is this all going? But then yeah, last summer we started to see the release of some of these GPT-3.5, et cetera, models. And the breadth of things that it was able to do was just sort of stunning. And I actually pulled together a little mini working group that Vijay

was part of where we were like, what are the implications of this going to be for education? And then Lo and behold, around December The New York Times and others started to pick up the advances, and it's really unfolded fast BHARAT N. ANAND: Thanks, Dustin. Latanya? LATANYA SWEENEY: Yeah, I would-- well, first of all, I would add to the notion of generative I think is what really creates the excitement around it. So AI algorithms were doing decision making. That was the way before. And people w

ondered what it was trained on and whether or not that training data had introduced bias. And there were lots of great examples of that and lots of stories around it and lots of questions around the use of this sort of automated tool that's making these statistical inferences and our use of it. But what made generative different is it creates-- it seems to create something. Something that didn't exist before, all of a sudden we get a new one. So generative photographs, generative videos. All of

these have been pretty stunning because in some ways as humans, we think ourselves are the only ones creating. And it's not creating in the way humans are. It's a kind of mosaic regurgitation, but in a coherent manner. And that's why it really made the moment-- I think is making this feel really different. It's around creativity, and it seems to be creative, but it's really building off the creations of humans that came before it. BHARAT N. ANAND: Hmm. Thanks. Rebecca? REBECCA NANCY NESSON: Hi.

Yeah, I'd just start by echoing what Latanya said. I think one of the things that was really striking about it is in the coherence and the amount that that gave me and lots of us pause about the ways that we normally evaluate authenticity or reliability of things that we're interacting with. And that when you-- it's hard to tell whether something is generated. It suddenly calls into question all of our systems for that. But the thing I actually wanted to share was for me, my surprise moment came

when my daughter was playing around with it, and it was funny. And I was really surprised that ChatGPT was able to be that funny. She asked ChatGPT to generate an ad for emotions, an ad jingle for emotions. And it generated, attention, all humans. Are you tired of feeling the same old emotions day after day? Are you ready to add some excitement and variety to your emotional repertoire? And then later on says, we've got everything from tried and true favorites like happiness and love to the more

complex and nuanced emotions like envy and guilt. And it goes on in this vein. And I just thought it was kind of making fun of humans and all of their emotions throughout and was surprised by that kind of cleverness or contextual understanding that I had never experienced before with a generative agent. BHARAT N. ANAND: That's so interesting. Vijay, you've been working with Google also for a while, right? And was there a surprise moment for you, or how did you react? VIJAY JANAPA REDDI: So I ta

ught a course, which doesn't help me with, and Bharat, you know all about, which is like a tiny amount. So that was a very exciting thing that I was doing during the COVID time. And then the course was quite popular. And then I was like, oh, I have to write-- I should write a short little book on this. And then the time is so limited, I was feeling very strapped, and I kind of could feel like, OK, I have to write this stuff. But I don't want to write all the boilerplate stuff like what is AI and

all the basics. So I started looking around for services. This was pre-announcement of ChatGPT. There are other services like Jasper AI and so forth where I kid you not, I wrote 90 pages of a book in less than eight hours. And I was using already-- like, to Dustin's point, there are other services that exist like this that are basically-- they had-- GPT's sort of the access point. It's when I started doing that I was like, wow it's actually generating fairly good content. I gave it to my wife.

I gave one page to my wife and I was like, can you tell-- what do you think about this? And she kept asking me, why are you asking me to read this one page? And I was like, oh, can you just take a look at it? Because I was wondering if she can tell whether it was human-generated or machine-generated. And she could not tell obviously, right? And of course, at that point, once it started generating the volume of content, that's when I was like, Oh my gosh. I think this is something big. It was rea

lly the first time I started thinking, oh. You know that experience that you feel when something transformative-- like when the iPhone shows up at your door or when you figure out that, wow, there is this notion of the internet or that the world wide web is accessible to everybody? When I started using that service, that's when I was like, this is that next big breakthrough that's going to change it. And you know why? It's not about-- honestly, I don't think it's about what it generates. It's mu

ch more that suddenly there is AI that's accessible for everybody in a way that they can interact with. Instead of just hearing about all this AI, AI, AI and its enterprises that are doing stuff, you can actually do stuff. I think that is when I was like, oh my gosh. This is going to be incredible because the world is going to use this in ways that we cannot imagine today. BHARAT N. ANAND: That's fascinating. So if you're using it to write your books, I guess this has big implications for whethe

r we allow our students to use it to write their essays, which we're going to turn to in a few minutes. By the way, my own story was-- actually had to do with a mistake that ChatGPT made. And this was back in December. And this is going to sound a little nerdy. I asked it about whether there are externalities associated with the Pigouvian tax, OK? This is like econ jargon. And it gave me an answer which was wrong. And I prompted it back and I said, your answer is wrong because of x. And instanta

neously it said, you're right. Here's the correct answer. And it gave a much more detailed description, which was far more elaborate than what I had given it. And that was just like, wait. What's going on here? I mean, the ability to actually change on the fly and correct itself was pretty incredible. OK. I'm just going to show the poll screen. So it seems like Poll Everywhere is overloaded already with responses, but we have a good distribution here. So roughly that's what we are seeing, which

is about 80% of folks who've tuned in have spent somewhere between zero and 10 hours on ChatGPT. And then we have-- and then we have some on the other end. All right. Let me turn to-- I want to go through three or four major topics here. And obviously, this has implications for virtually everything we're doing, but I want to focus on a few areas. Let me start with the implications for the research enterprise. We spent a lot of time talking about teaching, but I want to start with research itself

. So Dustin, I'll start with you. So GPT-3 and GPT-4 are basically state-of-the-art-- what we call natural language processing models with something like 175 billion parameters. You actually spent the early part of your career using natural language processing models in your research. Can you just tell us what it is simply, and why was this such a big deal in terms of the evolution? DUSTIN TINGLEY: Sure. So let me put this slide up just to work people through something. So I started working on n

atural language processing with a goal to be able to detect patterns in text. So for example, if people explained why they voted for one candidate versus someone else, what were the sort of general themes that might come out of this? And so something like topic models was one area that I worked, and that's just asking this question, like, what words simply tend to co-occur with each other? So for example, if we were to take each speaker, what they're saying, and ask, all right, what are the diff

erent topics that they're talking about, I might talk a little bit more about some of the algorithms, maybe some of the business models. Vijay might talk a little bit about some of the more technical things of how they work, et cetera. But it was really about how words just tend to co-occur together. Or something like sentiment analysis, which was saying, hey, I know which words are negative and which words are positive, and let's just get a general sense of the room. Like, are people feeling go

od or not, right? And I should say for us, I got a lot of citations for developing software packages that did this sort of stuff. My colleague Gary King has been a leader in this area as well, just providing these sorts of tools. So then you started to see more advanced things come up. So people started to do research on something called word embeddings. And this is this idea that maybe we can figure out words that share a semantic meaning even though they're different from each other. So for ex

ample, how does an algorithm know that Massachusetts is similar to New York? Well, they're similar because they're both a state. And so companies and researchers started to do work training models to figure that out. And then imagine a version of that where we're able to do the same thing not for words, but for whole sentences or whole documents. So there's a lot of really great work that happened there. And again, this was stuff that as a researcher, a white collar job, I guess, I was able to h

ave some impact. But now we're in a whole other world, right? We're in this world of multipurpose, generative AI. Here we're talking about text, as Latanya and others illustrated. And here, we're doing things a lot differently. So sentence completion. Now, we had a version of that earlier. We're style editing, all right? Like, Dustin, don't write in the passive voice, right? But the flexibility is just transformed. So now it's, hi, write code in language X to do Y. Or better yet, here's some cod

e in some language. Now translate it for me to some other language. Or here's 50,000 pages of documents that I've collected in some, say, sort of legal discovery process. I want you, the algorithm, to summarize coherently or even be able to have conversations. And the sort of-- where this has taken research is going in all sorts of new and fascinating directions. I'll give you an example that just came out yesterday. So a group of researchers-- again, I'm a political scientist, so I'm going to u

se an example from politics. What they did is they asked ChatGPT to compare pairs of legislators and say, which one is more liberal and which one is more conservative? And then they did that with lots and lots of pairs of legislators in the US Senate. And by doing those sort of pairwise comparisons a bunch of times, it came up with a scale of who's the most liberal, who's sort of in the center, and who's sort of extreme on the right, OK? They then correlated that scale that it came up with just

by querying this tool with other scales that people have developed using how people actually vote. And the algorithm did not have access to that data, and the correlation was stunning, right? And so it's just an example of how the research process is not only in a sense replacing, if you will, or augmenting earlier work that had happened, but now just opening up new ways to be able to extract information and make meaningful measurements. So the research phase, we're just tapping the surface of t

his right now. It's going to move fast. And that's literally just in a very small domain of me talking about politics. But you know, Latanya has domains that she's exploring. Vijay-- I mean, on the technical side it's moving fast, so really exciting. BHARAT N. ANAND: Well, thanks for that. That's really interesting. Just to summarize, so there's a few things in what you're saying. It's in effect a super smart sentence completion software. It's not just pre-programmed, right? It's dynamic. It's o

n the fly. And now as Vijay was saying, it's open to the public, and at least for now for free, right? That's essentially what we're describing. There's a question in the-- and by the way, folks, please chime in questions in the Q&A. There's a question about, what are the limits to this? Which I want to get to in a second. But Dustin, you were sort of, I guess, half joking that in the early part of your career, you got a lot of citations partly because you had built up comparative advantage with

using these kinds of tools. Is that comparative advantage now basically commoditized? Am I getting that right? Or what are you implying? DUSTIN TINGLEY: It's getting awfully close. I mean, I don't feel like I'm going to be able to compete in that area, and that's OK. Put me out to pasture, I guess. But now it's going to be incumbent on me to be creative in coming up with new ways to apply these tools that these large companies are coming up with. That is going to be where I will have a research

advantage. And the example I just gave from some colleagues at NYU is just one such example. But it's going to put pressure on me to bring their creativity of how to apply these tools, whereas before I was the person that was kind of making the tools available. Like, here's a knife. Here's a spoon. Now they've given us a giant tool kit loaded with lots of utility knives that can do all sorts of different things. BHARAT N. ANAND: Just to complete that thought, Vijay, where is most of this innova

tion now or the advances coming from? I mean, in some-- by the way, I have to say what Dustin said is quite sobering, because we often talk about AI and blue collar jobs. This is very sophisticated skills like the ones he and others have that essentially now look quite different. But one of the things you were talking about is most of this is not coming from the research enterprise within universities. It's coming from companies. Can you just share more on that with us? VIJAY JANAPA REDDI: Yeah,

it's a little disturbing. If you look at some of the fundamental building blocks of what makes, for example, things like ChatGPT really work, those fundamental building blocks are really things that-- like the "Attention is all you need" paper or like the BERT masking methods, which are all kind of the critical pieces that kind give rise to the underlying architecture that does it, you look at those innovations, those innovations-- I hate to say this-- really came from industry labs, right? And

I think it's getting really hard for us academics being really good and being really passionate about this topic to have real technical impact in here. And in part, that reason is because you need massive-- I'm talking about massive computing infrastructure to be able to do the kinds of experimentation that you need to do to be able to figure out, what are those nuances and knobs you want to turn here and there? I think that is something that is going to really hold us back. Can hold us back. I

think it's a concern that definitely keeps me up at night. I'm like, how am I supposed to do this innovation? You already kind of mentioned, Bharat, like, oh, I tend to work with the companies and so forth. I'll tell you why. Because if you don't, you will be left in the dust. I guarantee you. It's a really critical thing. And there's so many of my colleagues just in the computing field will ask, oh, how are you able to know this and that and that? I'm like, well, it's because for better or wor

se, I do have to work very closely with them because they do have access and knowledge to things that we are still not aware. And by the time they come out, especially now I can tell you, given the competitiveness that's in the field right now, doors are closing really fast. They don't want to necessarily publish all this stuff. Now suddenly, there's this thing like, OK. My gosh. This is going to be make or break for us. So that's the big concern that I have is that academia is struggling in thi

s sort of space. BHARAT N. ANAND: Fascinating. I mean, it's sort of somewhat-- Gary King, who Dustin was referring to, has also been talking about the major trend, let's say, with data analysis for the last 20, 30 years where 30 years ago, we were basically buying data sets, right? Or using government data sets. I mean, that was the norm. And now most of the data that we want to analyze sits within companies. And what are the implications of that for research? There's several questions already s

tarting to get at issues around plagiarism, cheating. Rebecca, I want to start drawing you in into what the implications are for education, which are potentially profound. In fact, somewhere near-- and this is my rough estimate-- 50% of the articles written on ChatGPT are talking about the implications for what's happening with teaching and learning. There was an Atlantic story titled "Will ChatGPT end high school English?" which talked about this as basically the combination of the printing pre

ss, steam drill, and the light bulb having a baby in terms of the innovation. You were part of the team overseeing undergraduate education at Harvard College for several years. You actually taught a course in Second Life, if I understand, more than a decade ago. And now you're Dean of Academic Programs at the Engineering School, so you've seen the spectrum from writing to coding. What are the areas of teaching and learning most likely to be impacted? And if I can start with a concrete question r

elated to ones in Q&A, which is, what's your view on cheating versus teaching? REBECCA NANCY NESSON: OK, that's a lot of questions. I think maybe I'd like to start by saying that what Dustin said about what he needs in order to be competitive in his field is to be able to creatively use these tools is, I think, the fundamental challenge for all of us. If we don't transform education in a way that it helps our students be ready to do that, then we're not succeeding as educators. And so at base, I

would say we're making a mistake if our attempt is to focus on how to ensure that students are still doing the things that they used to do and not using these tools when the tools can be helpful. You know, but here's a cynic's view about the cheating side of things. I teach a large undergraduate class. I have 180 students. I spend a lot of hours every week on this class. And in the cynic's view of things, if the students can do the work with ChatGPT, I might look at it as that I am essentially

teaching GPT 180 times to do the work of this class. And then I would do that same thing again next year, and that is really accomplishing nothing. I think the minimal view of it is that we already operate with an honor system for the most part for our students where we say, you cannot use outside resources, whether they be ChatGPT or a person who took the class last year and saved all of the problem sets that are somewhat similar to the new ones. And we feel OK about that because we know that t

he students for the most part have a strong intrinsic motivation to learn what's being taught in the class if you've done a good job setting up what they're there for. And that's OK. I think that's a totally reasonable policy for a faculty member to take. But what's really exciting is to think about, what can you be doing with the new landscape that we're in? What should you be doing with the new landscape? So if you think about the question of, what is the difference between a student armed wit

h ChatGPT who has taken your class and a student armed with ChatGPT who hasn't taken your class, at the end of the semester there should be something that's different about the one who did take the class. And where I come down on it is that first of all, you're much better off if you're teaching your class in some kind of active learning, highly interactive format. That is something that the students are going to be highly engaged in doing and that they, just as a matter of course in the activit

y, can't substitute a software for. And it will engage them in the process of thinking about the ways in which they can effectively collaborate with the software, not just substitute the software for themselves. And secondly, it will help us to learn where the real learning is in our classes. So I certainly would encourage faculty to think about how to incorporate it rather than how to stamp it out. BHARAT N. ANAND: Yeah. I mean, just to pick up on the last set of points you made, Rebecca, it's

really, really interesting that in some sense, this process of adaptation to technology-- and how can we lean in? I mean, we saw this partly during the pandemic where the notion of take-home, open book assessments became essentially a necessity, [LAUGHS] right? I mean, one question is, what happens to that with ChatGPT? But the broader point is, how do we think about, A, the nature of assessments, and B, as you're hinting at, how do we think about what we teach and how we teach, right? I mean, t

hose are pretty fundamental-- those are pretty fundamental questions because they start getting at what we're doing in our classrooms. Latanya, you've been using this, I understand, in your classrooms with your students? LATANYA SWEENEY: Oh, definitely. From everything-- one student took my final paper assignment where they had to write a research paper and asked ChatGPT to write a research paper for them, and literally just gave it the entire assignment. And it literally wrote a-- it produced a

final report that was really pretty good. And then another student asked ChatGPT to write a research paper written by Latanya Sweeney on data privacy. And it literally wrote a research paper that sounded great by me on data privacy, only I never-- it's an experiment I never did. As far as I can tell, no one else ever did. It had citations that don't exist. It was pretty amazing. So it's amazing on two levels. One, that it could produce something that was such a great imitation of something. And

it's that notion of that imitation that allows us to use it as a tool, as a starting place. But it makes up stuff all over the place, right? It just adds in, I need something that looks like this, and just puts it there. And so that's-- so it creates a lot of false information. And so even if someone were trying to provide the output of ChatGPT in an academic setting, the problem with it is you've got to go through and clean up all the things that are inaccurate, right? And sometimes it's a bit

repetitive. On the other side, the imitations are so good that when we did lots of experiments where given a paper, a research paper to a person, one actually that came from science and one that didn't, that ChatGPT wrote up and we asked it to write it in the specs of science, and people couldn't really tell the difference. That's pretty amazing. Even educated people in their field couldn't necessarily at first glance tell the difference. And then in other writings, people can tell the differen

ce. But where it really seems to be almost impossible is when it puts on a personality in chat, because then as humans we personify it. We believe it's often will-- people will believe that the ChatGPT, for example, is the person, and the person who is speaking is not. So I found that was a really stunning example. BHARAT N. ANAND: Fascinating. I'll share one thing that I tried, which is I gave it a prompt asking, what should Harvard's strategy in online education be in a way that preserves the

magic of the campus experience? And you know, it typed out a fine essay, but it was pretty generic. I then cut and paste the essay and gave it back to ChatGPT and I said, if this was the answer to an essay prompt like this, how would you grade it? And it said, I would it a grade of C plus because it's not Harvard-specific enough. LATANYA SWEENEY: [LAUGHS] BHARAT N. ANAND: Which was absolutely fascinating because I was like, actually, this is not just for giving answers. It's for grading answers.

And there's a certain degree of self-awareness, or dare I say, humility here. LATANYA SWEENEY: [LAUGHS] BHARAT N. ANAND: Rebecca, let me just cycle back to you. So you have instructors who are basically saying sort of the equivalent of laptops down in the classrooms. You can't use ChatGPT. You have others like Latanya basically saying, embrace it. Use it. There are imperfections. You will figure it out. As an academic leader, how do you set policy? REBECCA NANCY NESSON: Well, we do leave it lar

gely up to the faculty in their classes to determine what their policies will be. We ask all the faculty to be really clear about it on their syllabus exactly what they expect. But we do find that it's confusing for students that the policies are so different across different classes. And I think that's-- I think that's the better option in the trade-off than to try to have one fixed way that we expect all students to do it. I also think the challenges are really different for different classes.

So just back to what Dustin said at the beginning about GitHub Copilot, early on with that tool David Malan was using it in CS50. And rapidly, as soon as it came out, tried it out with the assignments. And not only was it generating correct code, it was generating the solutions to the problem set, because I mean, there was a question in the Q&A about, when is it plagiarism? I mean, in that case I think it was plagiarism. The AI was actually copying something-- solutions that it had found online

and spitting them back out. And so you need in that case to be able to have a different policy in that class and what you're working on in that class with a particular tool than, say, in a different one where, for instance, you're working on essay writing. And even a content-based class with essay writing might have a different policy than a class that's really trying to train students to write. So I just don't think we're in a one-size-fits-all situation. What I would strongly suggest to facul

ty is that they incorporate some forms of assessments that are a little bit like laptops down assessments, things that actually happen in class that are more interactive and can't just be done by a software. BHARAT N. ANAND: I mean, you're highlighting a fundamental trade-off here, which is in some sense you want local innovation and local efforts by faculty to try and see what works and doesn't work in their classes. The trade-off, of course, is for students, that can be confusing. And how do w

e balance that, at least in the short term? Let me just say, there's a whole bunch of questions here around, how might we address things like false information, misinformation, the notion that most of the train data for ChatGPT has been in English? How do we incorporate other languages? Latanya, I'm just flagging those questions because we'll come back to you with that in a few minutes. I want to pose this question to all of you before we leave the teaching side of things, because I mean, this i

s really an important set of issues for all of us. I'm thinking of the London cab drivers, right? The black cab drivers, the taxi cab drivers in London, who-- and you all might be familiar with the story, right? I mean, they go through three years of training to basically acquire knowledge on over 25,000 streets in London, and it's considered the hardest test in the world by some people. And then GPS comes along. [LAUGHS] OK? And it's like, wait. What just happened here, right? And over the last

decade, it's sort of been an interesting struggle just to see how they're trying to adapt. If we can fast forward 10 years and if you were to speculate on, what is the most different aspect of the content and the teaching in your courses, what would that be? Whoever wants to-- DUSTIN TINGLEY: I'll go first, and just very briefly because I know we've got a lot to cover. Relationship-driven rather than content-driven. BHARAT N. ANAND: OK. Thanks, Dustin. Great. VIJAY JANAPA REDDI: I'm just going

to say that I don't know about 10 years, people. We didn't know this thing existed one year ago. And I was talking to a couple of researchers at Google who work on these things, and they were like-- they were saying, we didn't know we could do this kind of stuff. How are you going to plan for 10 years from now? We don't even know what's going to happen in the next six months at the rate of which this is actually happening right now. BHARAT N. ANAND: Yeah. LATANYA SWEENEY: Yeah-- VIJAY JANAPA RED

DI: That is an important question. Sorry, Latanya. LATANYA SWEENEY: No, no, no. I was actually thinking even in a shorter timetable. I've been teaching a version of these courses around technology society classes for over two decades now as we've been in the middle of this technological revolution. And this will be the first time where we're asking the students to start with a paper that actually is written by ChatGPT on their particular research area, and then edit from there. That's pretty ama

zing. I never could have predicted that. But everything about this sort of revolution that we're a part of has already in just so many-- so few years changed everything about our lives, even how we're giving this presentation today. Online, how we work, how we play, and so forth. And we don't know when the end is going to come. Many historians call it the third Industrial Revolution, and it's just not clear when the end is in sight. It's hard to know-- I mean, if you went 10 years earlier from h

ere, you know, where the new technology was Facebook, you could not have foreseen where we would be today. REBECCA NANCY NESSON: I think I would say for me, it feels like a comparison to the kind of back and forth in baseball between the pitchers and the batters. And this is a strong shift for our students towards needing to be able to read and to understand really carefully and critically more so than to be able to generate. And so I think the assignments are going to push in the way Latanya's

saying. It will start from something that's generated and they need to be able to make sense of it. BHARAT N. ANAND: Yeah. I think-- I mean, in some sense, I think one of the things that you are all pointing to is, what is it we're really trying to assess now? Rebecca, you and I were chatting yesterday-- what's that? VIJAY JANAPA REDDI: What you're trying to assess, Bharat, is the ability to generate prompts. [LAUGHS] Theoretically, it's called prompt engineering, right? But in fact, I think som

eone in the chat actually raised that. But it comes down to that, the ability to understand how to play with this. And one thing to keep in mind, it's not about just prompt generation, right? It's also understanding the context. To your point, Bharat, like where you give something and you go back to it. And so it's this interplay between the system. It's not just a one-shot, oh, I put in something and I get something out. And I think that's that new generation that students will all be looking a

t, is like, what is prompt engineering as a science? LATANYA SWEENEY: I have a great example-- BHARAT N. ANAND: Sorry. Go ahead, Latanya. LATANYA SWEENEY: I was just going to say, I have a great example of prompt engineering. My son-- he's 14 years old-- he wants to build these card games that have these amazing images on them. So he uses a generative AI platform to generate these images. The theme of them are all chess-oriented. So he became this amazing prompt engineer to create exactly the ki

nd of image he really wanted. And the images are absolutely stunning. And so there is something to-- but that's us learning how to get what we want out of this machine. Right? BHARAT N. ANAND: There's-- yeah, no, that's a great point. By the way, just to make sure, I mean, the flipside of this is where Khan Academy is going now, right? And Sal Khan announced two days ago the partnership with OpenAI where in effect, they're using it as a tutor. So it's not giving answers, but if I want the answer

to something, it's basically going to work with me like a personalized tutor to keep prompting me with the questions so I can figure out what the answer is, which is equally interesting. There's a question around, what are the skills ultimately that we'll value in terms of writing and coding? Or what are the markers of intelligence, I guess, more broadly? And Rebecca, we were chatting about this yesterday and you used the analogy of math addition, simple math addition and neat handwriting [LAUG

HS] are things we sort of don't care about anymore, or at least they're not valued the way they used to be. Do you have any sense of what are the kinds of skills we will-- REBECCA NANCY NESSON: I don't want to shortchange ChatGPT or generative AI, so I don't know exactly what things it's going to be able to do, or even is potentially able to do now. But I think that some of the things that seem to me to be valuable are to be able to decide what problems to work on. Maybe ChatGPT can do that. I'm

not sure. Maybe these models can. I wonder that about Latanya's example about generating a whole paper written by her. And how interesting was the experiment that it came up with for you to do? And I also wonder about the ability to do more transfer-like activities, connecting of knowledge across domains in a way that's really creative and interesting. BHARAT N. ANAND: That's really interesting. By the way, there's one question about, is there a risk for ChatGPT replacing interactions with the

on-campus experience from a professor? And I'll just share my thoughts on this. One of the things that was fascinating to me over the last three years with remote teaching was the extent to which a number of faculty started thinking about once they experienced online tools, it's like, wait. This is actually pretty good for content transfer. And then it forces you to think about, why do we actually need to meet in person, right? What's that synchronous-asynchronous mix? What's that online versus

in-person mix? And you start appreciating the importance, as Dustin was saying, of relationships and connection and community, and those kinds of things. And I think this is going to force us to do exactly the same thing, in effect. I want to turn to thinking about the broader applications of this in industry. And Vijay, you've been working on AI integration in a whole range of applications for many years, and you lead a lab at Harvard. In some sense beyond just individuals like us interacting w

ith generative AI tools like ChatGPT, can you say something about the broader range of applications where we're likely to see this in business and work? How do you think about that? VIJAY JANAPA REDDI: I think the big thing is we're all talking about ChatGPT because it's the hub of the world, it's the poster child. But in reality, from what I've gathered from many companies, they'll say that, oh, we're not going to build a chat bot. That's not what this technology's really going to unlock. That

is just the thing that people can engage with, and it showcases the possibilities of what is really possible. And the thing you have to realize is these are-- the reason they're exciting is because they're able to synthesize things that are well across domains, right? So there's interactions that you and I might not be able to pick up because we can handle a limited sort of domain knowledge. But this thing can just suck in whatever connections it sees, and sometimes those unexpected connections

are what gives rise to all sorts of new information that you extract from the raw data. So I think ChatGPT is just the tip of the iceberg in being able to do that. And I think a lot of research is showing that it's not the kind of service that people really care about. Video content and online media, talking about blogs and those kinds of things, apparently it turns out it's like a mass market. Law, I don't know. Latanya, you'd know much more about this. But I've been thinking, oh, this is prett

y interesting. I wonder-- I was going to ask you about this in person, in fact. I was like, oh, if I wanted something drafted up, would I really need a lawyer or someone really experienced? I mean, especially with law I feel like it is kind of so, in a way, rule-driven, perhaps. And I was kind of wondering, in such industries maybe it's transformative because it's like suddenly, I don't need to pay, I don't know, thousands of dollars or whatever it costs to get a lawyer. Now suddenly, the best,

talented lawyer is probably accessible for me for $0.99 per request. So that's the way I'm thinking about it. Latanya, it'd be nice to hear your thoughts on those. LATANYA SWEENEY: Yeah. You know, all of us are very-- certainly spoken to the positive advantages and excitement around these generative tools. There are these incredible flip sides. And the question that Vijay's asking starts to begin to think about what some of the flipsides look like. This idea of false information and just making

up stuff, how do you deal with that if you wanted to use it in a setting like law, right? If I wanted to understand the law, like, suppose it was primarily being learned off of the actual legal texts, it's phenomenal just to help me understand. We have 2,167 privacy laws in the United States, the last time I counted them. And if I wanted to know, is a certain act OK, it would be fantastic to be able to just ask that in a generative form like that, just in a prompt. But I can't trust the answer t

hat I might get right now, or even for a while. So solving that problem is real. The other thing too, when we think about the future of the internet, within three years I would guess 90% of the content, maybe even 95% of the content is no longer going to be generated by humans. It's going to be generated by bots. And these bots are basically statistically regurgitating variations of themes of things that humans said and then regurgitating variations of things that other bots have said or written

. And what does that mean? What does that mean about our sense of what is real, what is truth, what is knowledge? And these bots are being taught primarily on data from the internet, in the case of ChatGPT. And it learned on the internet the internet has all kinds of biases, all kinds of issues, all kinds of problems. And ChatGPT shares those issues and problems, as it will often regurgitate things that it really shouldn't be saying. And right now they have a filter on it to try to not let those

things out. So there are a lot of issues. And then I think even in the chat was the intellectual property question that in the GitHub or in the images, we as humans, we create them. There's a kind of economics involved in what we think was the production of this work, whether it's a particular way of writing poems or a particular art form, or what have you. And then all of a sudden, ChatGPT appropriates it and uses it and is generating it en masse. How does that work? So it brings on intellectu

al property questions that we haven't examined before. BHARAT N. ANAND: Latanya, you've been, I mean, in some sense working for over a decade on some of these issues, right? Algorithmic bias, privacy, re-identification of data. As you think about generative AI models and tools, I'm just curious, how does the scope of these issues change? Is it incrementally, or is it a matter of-- is there a discontinuous change? And I guess the more fundamental question is based on your experience, what can we

do about this? Take just the example of algorithmic bias. And it'd be great if you could just explain simply what that means to the audience. LATANYA SWEENEY: Yeah. Well, the way-- I would go back just one step further. When I started this-- I'm a computer scientist by training, and when I was a graduate student you could just sort of see this revolution coming. It promised to transform our lives. And we just-- many of us just believed there were no downsides to it. It was all going to be upside

. But right away, one of the first things that I was early to see was the problems with data privacy, how easy it was to re-identify people, how hard it was to keep your data safe. And so in many ways, that became the first major technology-society clash. Then as we go forward, the next big technology-society clash was when AI algorithms were used for decision making. They would train-- they still are. They would train on data and then become-- make decisions for us or advise our decisions like

in criminal justice or credit worthiness, and so forth and so on. And these algorithms are used widely, but most people don't even know they're part of making decisions. They can't explain their decisions. And if they're trained on biased data, they just produce biased results because they're just replicating statistically what they were trained to do, even though they look like there's going to be impartial. So that's sort of the next wave. The third wave has been around misinformation and disi

nformation and the use of that information, the spread of it on social media. We don't really know how to establish truthful content at scale or what trust means at scale or moderation at scale. We haven't figured any of those out around the issue of moderation. So those are three major tidal waves, and now comes generative AI. The time between them is shorter. Between privacy and algorithmic fairness was one time, but only a couple of years later before we got to the issues around democracy, an

d only a few years shorter than that before we get to these issues around generative AI. So these clashes are coming faster. And the truth is, even though data privacy goes back to 1997, we still don't know the answer. We still don't have the solution. So these things are transforming our society. We can't even enforce many of our laws and rules. It's upending others of our laws and rules, and we are moving at a snail's pace in comparison. It is definitely an issue for our society. BHARAT N. ANA

ND: Is there-- well, [LAUGHS] is there a silver lining, or what-- is there any cause for optimism on any of the fronts you described? LATANYA SWEENEY: [LAUGHS] Yeah, well, you know, I'm still that graduate student. I'm still the graduate student who wants society to reap the benefits of these technologies without the harms. And many of us-- over time, we are definitely growing a workforce of technologists who want to work in the public interest, who are trying to shore up government, trying to s

hore up sort of consumer, civil society, and so forth. All of the people who would normally help society maintain itself need to sort of be-- need to have technologists who can help them do their job in this sort of new area at the speed at which these issues are coming. And so there's a lot of movement towards public interest technology. We think that'll make a big difference. Whether that'll save us, we'll see. BHARAT N. ANAND: Yeah. No, and obviously, that's one dimension. I just want to clos

e with one question that just came up. Are there steps that you believe tech CEOs can be taking now to roll out safer AI-enabled products? I guess to any of you. LATANYA SWEENEY: I'll just give a quick one-liner. Two weeks ago, I did have a great meeting with venture capital firms in Silicon Valley trying to change the way this cycle is working to help them as part of when they're developing new companies, investing in new companies actually think about technology-society clashes, because a lot

of times they can be avoided simply in the design stage, and it's really easy. It's much harder once these products come to market. BHARAT N. ANAND: Hmm. Vijay, anything to add there? VIJAY JANAPA REDDI: Yeah. So around safer AI, one of the issues is that as people are building these models right now, everybody's really focused on what new capability they can really unlock, because they really want to push the state of the art. It is a very competitive sport. And so that's what people are really

doing. But then we have this notion. We're working with a community of AI researchers spread across Google, Meta, and a whole bunch of other companies and academia to build this notion of safe AI is better AI for everyone. And the intent is that as we use these generative methods, what we need are mechanisms to collect those adversarial examples, the cases where these things are breaking down. For example, like ChatGPT or any of the generative ones, you want to be able to say, OK, well, this on

e is considered a harmful image or chunk of text. And everybody has to do that. You and I have to do that, because the way you think and the way I think are different. So what I might think is totally fine is something that my wife will totally disagree with. That's totally valid because there are two different perspectives going into what we're looking at. But if we can capture those examples with a little bit of context and then we can build these large corpuses of adversarial use-- adversaria

l examples where you have a prompt and then something is generated, you have that new data set, what we're trying to do is get the companies and everybody to kind of give us access to the model. And in exchange we say, hey, we can build these data sets and we can give you these data sets back into the ecosystem so you can lift the baseline of safe AI for all of your models, because every model when it's going up should have some minimum sort of thresholds that it needs so it just works. How well

it works at the top, that's a different story. But to Latanya's point, I think that base has to be lifted. And I think the way we've got to shift to big tech and so forth is to think, OK, everybody should be able to lift it, which means, how can we all work together to get researchers and academics and so forth to help us do that? So that's something we're all kind of collectively working on as a community. That's just one example of how to address that issue. BHARAT N. ANAND: Very interesting.

We're near the end of the-- by the way, thank you all. This was a fascinating discussion. Let me just show the other two poll questions and where the model response was. This was a question about, have you started to think about the implications of ChatGPT and generative AI models for your own career/role/college major? So most are thinking about it. By the way, I don't know if you have any suggestions for how undergrads should be thinking about what to major or their own career choices. And I'

ll come back to that in 30 seconds. And then the last question was, do you believe the widespread use of language models like ChatGPT will lead to a loss of human creativity in fields such as what we were talking about-- writing, journalism, content marketing-- or will they enhance our ability to generate and express new ideas? So some optimism here. Let me maybe give you each the floor for 15, 30 seconds each. What is the one thought, question, or advice you want to share with folks as we start

to engage with these kinds of models going forward? Dustin, let me start with you. DUSTIN TINGLEY: Yeah. I think we in higher education need to think about new types of curricula. So Latanya Sweeney started a track within the Government Department, Political Science Department on technology science that immerses in these issues. Here in our office, we've launched a series on data and digital. And it's not about training people how to code. That's great. We should be doing that sort of stuff. Bu

t critical thinking around these technologies. Latanya happens to be featured in the class that I taught, right? But the curricular reform can complement the pedagogical reform that Rebecca was talking about. And I think that's a huge opportunity. The other piece real quickly, we haven't talked about this. What are the business models that are going to sustain these new operations? This takes enormous energy, enormous amounts of servers to work. And I don't think a lot of them really have a grea

t idea yet, and the advances that will be made in being able to support the business models will be something to look at that we haven't talked about today, but it's a fascinating topic. BHARAT N. ANAND: Rebecca? One thought, question, piece of advice. REBECCA NANCY NESSON: I would like to see this technology revolution be different than the other ones in terms of it helping us to address some of the biggest challenges facing humanity. And I think it has the capacity to potentially be deployed t

hat way, to help us overcome-- manage climate adaptation or any one of these big challenges facing us. So that's what I would encourage people to think about, is how to put it to work to do something important. BHARAT N. ANAND: Great. Thank you. Latanya? LATANYA SWEENEY: I think the question is, how do we maintain our control of our lives, our society? All of these changes by technology are changing the way we live and work and play, but they're also making the rules that we live by. And the que

stion is, we no longer-- it's not rule makers increasingly or the laws that we have that determine our rules, but the design of these technologies. And how do we restore sort of control over that? BHARAT N. ANAND: Thank you. Vijay? VIJAY JANAPA REDDI: One thing I'd focused on, it's like across the entire planet we teach everybody doing any sort of computer science or engineering-related thing how to program code. And we have heavy debates about, should we use high-level languages, low-level lang

uage, and blah, blah, blah. I'd say the course, for instance, I'm actually trying to put together and scratching my head is, how do we teach our next generation of students how to program with data? So imagine this course on data programming, because these systems are all going to be "coded," quote, unquote, with data. And today, we have absolutely no tools. We don't have a framework. We don't have a syllabus. We don't have a criteria for how to systematically teach people to be able to build th

is. At the end of the day, there are also engineers and scientists building these things, and the building is really going to be with the data. They're going to code it with the data. So how do we systematically think about that? That's one thing that really keeps me up at night around this. BHARAT N. ANAND: Well, let me just say thank you all. This was incredibly thoughtful and such a delight to just hear from all four of you. I'll close with what seems to be a narrative that is almost sort of

captured through this discussion. I think there's three sets of questions. One, what can these technologies do today, right? Even just getting our heads around this is hard. There's a second-level question of, what does it mean for our enterprise, for our work, for our jobs, for our roles, for what we do day to day, how we do it? But I think as all of you are saying, it sort of leads ultimately into the third question, which is, what does it mean for us as individuals, as humans? What does it me

an to be human, as Michael Sandel, our colleague, often reminds us? Where does human agency come in? And those are hard questions. There's obviously some issues that we haven't been able to talk about, like Dustin, you were saying the business model implications, which have huge, huge consequences, as Latanya was alluding to in the social media context. Most of those outcomes, I think, obviously were a reflection of business model choices. Perhaps that's a discussion for next time. Thank you all

. DUSTIN TINGLEY: Thanks, everyone. CASSIE MCGRATH: Thank you all. And thanks to all of you who have joined us today. We hope you'll join us for more Signature Events soon. Thank you for joining.