AI at Roblox: Revolutionizing Game Creation | Morgan McGuire from Roblox

Yeah, that's exactly right, Matt. So Roblox is a social platform. It's online for 3D interaction between people. As you know, we have about 70 million daily active users, so it's a lot of people, and they're interacting in 3D worlds that are 100% created by the Roblox community. So Roblox, the company, provides the platform. We provide the safety infrastructure and the communication infrastructure, but the content is all created by the community for itself. And within those experiences, some of them are indeed game-like, some of them are more social, some of them are educational, people have all the kinds of interactions that you have in the real world.

And that can vary from hanging out with your friends to socializing, meeting new people, again, the educational aspect. And we're starting to see things like shopping and brands engaging on the platform as well. So the whole space of physical human experience brought into a virtual world with all the benefits that digital technology can bring there.

Essentially everyone. Historically, Roblox had a lot of game-like content, and it had younger users. And it's one of the reasons that we've really excelled, is we have strong safety. It's really important for the younger users. We've since seen the platform grow and generalize, and we're now at the point where we are active in countries around the world. We have something like 40 different languages people are using to communicate on the platform. And our fastest-growing segment is actually 17- to 24-year-olds.

And so we're sort of an all-ages, all-people platform at this point.

So it's an incredibly complicated in-house technical stack in order to present a really robust and simple interface to the user. So we have our core data centers. We have about 100,000 servers that we maintain with our own infrastructure, running our own software stack on those. And that's the heavy-lifting backend. That's how we persist all of the data. You have an avatar in the virtual world. It's how we keep track of what you're wearing, what the social graph is.

All of the items in the world, all the 3D worlds. We have edge data centers around the world. I think we're up to 17 of those. And so wherever you are, there's an edge data center geographically located near you so that we can have low latency, so that the experiences can be really responsive because there's not a lot of telecommunications delay. And then we, of course, have the clients, which run on every device. So you could have a game console, you could have a VR headset. A lot of our users are on mobile devices, especially phones and tablets, laptop, desktop.

And so whatever device you're on, wherever you are, we present a window into this 3D world where you can interact with real people.

So the company as a whole is about 3,000 people at this point. We're based in Northern California, and it's heavily engineering-focused. So almost all of that is engineers or engineer-associated professions. A subset of those associated professions are, say, the research scientists, the machine learning scientists, the data analysts as well. And essentially, everybody is working on the entire stack. Obviously, there are specializations for different areas, but I specifically want to call out the machine learning and AI because I think that's been the fastest-growing area of our investment in our people, has been in taking what's been a core adoption of AI tech for about six years now at the company, originally starting in safety and now growing into all aspects of the business, especially our generative AI, which is where most of our recent product releases have been focused.

So because, as we discussed, all the content on the platform is created by the users, we have a suite of creation tools. So on desktop and laptop platforms, we've released our free tool called Roblox Studio, which is available to everyone. And I think one of the most important and unique things about Roblox, which really differentiates it from all other 3D creation throughout the entertainment industry, is that on Roblox, it's not just that the tool suite is free, it's that publishing is instantaneous and free.

And so it's the only platform where you can do for 3D what today you can do for microblogging, like text or images, which is one person, without some sort of app store approval, without becoming a publisher, can go and create something that you want to share with the world, push a button, and make it live in minutes around the world on all these different platforms. And you can even make money on it. So you can monetize your content directly, control who sees it, and then you can price objects within that.

And there's no other toolchain in the world that allows you to do that. And so that's sort of the core of the creation engine. Technology is super important, but the business and the community are really where the magic is of that process, which is that they have instant access to the world. So they get feedback, they see what's happening, they have motivation, they can build a business and a lifestyle on this, and then they can iterate based on that feedback with that motivation.

So the tools themselves, you have Studio, which is sort of a best-in-breed, easy-to-learn 3D creation tool. That includes making 3D models, it includes augmenting those models with physics, with animations for avatars. You can act out motions, you can draw curves in the Animation Editor, you can write source code in our Luau programming language, and you can access our suite of creator analytics tools. And that's part of that feedback loop. You can see who's entering your experiences, what they're doing, how much time they're spending, how you stack up compared to similar experiences.

So that's a relatively conventional tool suite. It mirrors what you would see for professional tools for making movies or AAA games or other 3D content, but we've brought it down to the point where it's accessible to people who might have never programmed or done 3D before. And then beside that, we have sort of our prosumer set of tools, which is that you can upload through Studio, you can use more professional-style tools if you're already an experienced 3D developer. A lot of our creators use, for example, Visual Studio or Visual Studio Code for their software development side, and they'll use tools like Git for version control, and they might use a tool like Blender for the 3D modeling.

So we don't lock people into our toolchain. We offer a free toolchain, and if you have your own tools, maybe some open-source ones that you want to bring, especially for a small studio, we accept that. We can ingest content from a variety of forms. So there's sort of an easy on-ramp for new creators, and then for pro and semi-pro creators, there's a way to plug directly into the tools workflow they have. And the direction we're increasingly pushing, and you're starting to see this with some of the things we've been teasing in our demos, is creation within the experience.

And so that would mean that you don't see anything that looks like traditional 3D tools, but actually, within a 3D world, you're able to leverage the power of most of our creation engine. And we're starting to make that available to our developers, and the developers can expose it to all the players on the platform.

Thank you. Thank you. I'm really proud, as a company and as a team, of the iteration pace and the fact that we've embraced two concepts that I think are somewhat unique in this space, which traditionally is more controlling because you have rights holders and you're trying to be careful about how your IP is portrayed. One is this sort of fearless transparency and iteration in public. We talk, we publish our roadmap for the product for our creators. We bring many of them to our Roblox Developers Conference and give them previews and show them what we're doing.

And we've done that explicitly without fear of saying, we're going to try things. We're going to ship these features. We're telling you now because we want your feedback. And after we've shipped them, we are not afraid of getting it wrong the first time. And there are many cases where we've released things, we've gotten community feedback, creator feedback, iterated on it quickly. And by doing that in public, doing it with a published roadmap, doing it with our open APIs and our embrace of open source, it's allowed us to move really fast because we're not trying to hold back and control it.

We're just trying to partner directly with our community. And I think they understand that we're doing all of this for their benefit. And so they're willing to meet us halfway and iterate on that rather than expecting something that's perfect and locked down the first time it ships. They'd rather get it a year earlier and provide feedback. The second thing that's really impressive that Roblox does in terms of development, and again, I don't know anyone in the space who does this, is that we ship weekly.

And most products in the space, if you think of 3D software, it'll be every four years there's a release of a 3D modeler or a point update, or for a game engine, maybe every two years. People sometimes release patches every quarter. Every single Thursday, we ship a new client to every single player around the world, releasing new features, new access, new security mechanisms. We are continuously updating the backend. And this is something where we are really inspired by web development, where the notion is if everything's on your server, you can iterate continuously.

Because there aren't discrete releases. We've brought that kind of rapid release to the much more complicated world of 3D software development and pushed it all the way out to the client devices and said, we are going to ship frequently. So it makes us incredibly responsive. Whenever there's that kind of community feedback and we want to adopt it and change the feature, we don't have to wait for a release six months from now. We can push out a change in a few days.

And that kind of iteration and that kind of embracing the community, I think, has allowed us to be a lot more nimble, and that as AI technology has gotten more powerful within our space, specifically for safety and for creation, we've been able to embrace it really quickly.

Yeah, that's a great question. And I think, let me define some terms because I think being knee-deep in the technology every day, you have a different perspective. And the first thing is that it is very early days of AI. This is, I think, not just for Roblox as a company and our general industry, but I think for the world as a whole. This is going to be something that affects everybody. It's a really powerful technology, but it's not about next year, the AI boom is over.

The markets and other stuff will do whatever it does, but I think the core technology has decades before the full impact is felt. It's going to be a long-term transformative effect on our industry and one that everybody has to really be thoughtful of at every level. So in that context, I think a lot of this hit the public consciousness with some of the products that you talked about. But the reality is this has been a very long—some of this technology stretches back into the '70s and '80s, and it was just waiting for the right time and the right scale.

And we needed all of the pieces. We needed GPUs for some of this. We needed internet-scale data. There were just all these pieces that needed to be in place, but it wasn't something that came out of the blue in terms of the core technology. It just wasn't relevant to the average person and the consumer until fairly recently. So for Roblox, our embrace of what I would say we would classify as machine learning, which is sort of the box within which AI sits.

And so this is all sort of data-driven techniques because modern AI, we think of it as a robot or a computer thinking, and that's nice packaging. But what it really means in terms of the underlying technology is that it's computation that, instead of being driven by a person writing code explicitly saying what the behavior of the program should be, instead it's a program whose behavior is controlled by the data that we feed it. So it's not any sort of consciousness or thinking for itself.

We're a long way from that kind of technology, but it is a fundamental shift from: do you drive your program with data or do you drive it with code? And that creates an entire industry, an entire set of technical challenges around how do you curate the data, how do you prepare the data, how do you compress the data, and so forth. Machine learning is something we've long embraced because we use it deeply for all of our analytics inside the company. There are a lot of techniques, many of which have then led to modern deep learning and AI techniques that we've been aware of and using inside the company for how we run our operations.

The next step for us was a predecessor of what you know as large language models, such as GPT or LLaMA or these things, which are based on what's called a transformer architecture. And this is a keyword you might hear a bit in AI coverage today. Without going into the technical details, the idea of a transformer architecture is it allows an AI system that's processing a lot of data to be selective about which parts of the data it's using to answer any given query.

It's called an attention model. The word transformer has—there's no intuition behind that. It's just the name. But this attention model is the key idea, that it can filter and say, oh, I only need this set of the data to answer it. I shouldn't try to aggregate everything. I should focus on this is the important part of the question, this is the important part of the answer. We were an early adopter for transformer models because we've always been really aggressive on safety on the platform.

One of the ways that we monitor safety is every communication between humans is monitored by technology. So we make sure that everybody is acting within the terms of service, that they're being civil and positive, and that they're constructive interactions. This is very different from most online platforms that might have some level of keyword filtering or something, but really aren't trying to control the tone of interaction. We strictly wanted this to be a positive, optimistic environment and not sink to the lowest common denominator.

So we had things like keywords, and then we had more sophisticated natural language processing models. When the transformer architectures, these attention-based models, first came out, they rapidly started outperforming what any of the classical parsing, grammar-based ways of monitoring human communication were. And so we said, we have to get on this. So we adopted a technology called BERT and DistilBERT. And then our contribution there—this was really sort of our first step into doing AI R&D at Roblox—was taking those existing models for doing text moderation and optimizing them so they ran substantially, an order of magnitude more efficiently, because we need to deploy these in real time.

So as you're typing, we are checking what you're typing, and we just don't let it through. We won't allow exposure if you're doing something inappropriate. So we made them much more accurate, and then we extended them, in that case, originally to 16 languages. So these were originally English-only models. And for us, it was really powerful. And that was the first example of using AI for the consumer-facing part of the business and seeing how much more effective we could be using data.

Because as a company with a large number of users, I mentioned that there's 70 million a day, we also have 15 million experiences, millions and millions of objects in our asset store. We have a tremendous volume of data from which we can train our own AI systems and optimize them. And so something that had been part of the burden of running a large platform, of having—we have to process all these events, all this data, we have to make it cost-effective—that suddenly became a tremendous asset for us as well.

And in the modern world of AI, whoever has the data has a real opportunity in order to use that data for the advancement of their users. And so we've been able to leverage that in many cases. Fast forward to today, every couple of weeks we're releasing a new AI-powered feature. They're all throughout the safety system. We're doing voice moderation. So, short-lag, real time, as you're speaking in voice chat in multiple languages, we are monitoring what you're saying and we give you feedback as a user.

We can say, hey, that was, that was not the best way to phrase that. You have a 30-second warning. You're not allowed to talk, and then we'll back off. We really don't want to ban you. We don't want to exacerbate the situation, but we do want to find a way to make a positive intervention. And eventually, if someone is acting really unconstructive, there are various levels of abuse reporting.

Yeah, but we honor every audio, every human interaction, the 3D experience, all of this. We're able to apply AI to safety and then now the creation pipeline. And we have a huge number of tools in alpha and beta for creation, all the way up to writing computer code for making 3D experiences that we can help semi-automate.

Yes, which is now out of beta. So that's now a full-blown feature.

So, AI Code Assist is primarily targeted at helping people who are new to programming on the platform to quickly become better programmers. So, for this specific feature, setting aside the long-term vision, this specific feature is not enabling no-code creation of sophisticated new algorithmic development. That's not our intent. It's to say a human being is going to be the software developer, but we want to remove a lot of the learning curve for them, and we want to automate a lot of the tasks that might be more boilerplate.

So with AI Code Assist, you're sitting in your text editor for writing software code inside of our Studio tool. You start to write what you want a piece of code to do, or you start to write the actual code itself, and it will recognize and, sort of like in email when it auto-suggests, “Do you want to say yes, absolutely, or no, I can't make it tonight?” it auto-suggests. But instead of auto-suggesting a sentence, it auto-suggests 10 or 15 lines of code that perfectly fit into your program.

So it might be the equivalent of, you're writing a leaderboard for a game. Maybe you're making virtual field hockey, and if you're starting to look through the players and assign scores and numbers, it will recognize what you're doing and say, “Hey, here's an example of how you iterate through all the players. Here's how you find the players in the world. Here's how you make it robust to this kind of cheating,” and it'll suggest code, and then you can accept the code.

And then you can edit the code if you want, or you can ignore it completely and just keep going. And what's amazing, so we released this very early in beta because, like I said, we were just experimenting in public. We show people what we're doing. We've tried many different algorithms, backends, user interfaces for it. And we're now at the point where, I think, it produces character by character, like each letter of the code, and you kind of see it typing. It's sort of fun.

300 million characters of code have been suggested by this tool. And so that's sort of getting to the level of large volumes of software having been written by our AI to help creators. And it's fantastic because it allows people to get moving much more quickly without looking at the reference documentation, without having to watch a tutorial video.

So, yeah, everything we do uses whatever the best tool for the job is to present features to our community. And a lot of the time, if we make something new, such as the optimizing Lua compiler and type inference, we release it back to the community as open source. Or we've published, and we can get into this a little more later, StarCoder, for example, was some of our research on this: how do you make software that programs? And so we shared our LLM with the world that we had built, as fully open source, that anybody could build on.

And we also, more importantly, shared the methodology for how we trained it, how we taught it based on not just our own data of Lua code, but we trained on other code. And so it was able to learn from looking at Python or Java code how to write Lua code for Roblox. And that was called domain transfer. You learn one thing that's nearby, and then you transfer that knowledge to a different domain. And we developed specific techniques for how you do domain transfer for programming languages.

And we've also done a lot of analysis with seeing how programmers use the tool, and we've published all of this as open, peer-reviewed scientific research. So, the actual tool today is our own custom frontend, and then we're using a mixture of third-party and proprietary solutions on the backend for powering it. And part of the value of that architecture is we periodically switch out for all of our AI services. So we provide a Roblox interface, and then we might start with, like, for an experimental service, something that's a third-party hosted API service.

And then once we start to understand the needs of the feature, we might specialize it for ourselves, and then at some point, we usually release a completely proprietary backend that's exactly designed for our community and optimized.

Yeah, so AI Image Generator. You've probably seen tools for making sort of clip-art stock photos where you type in a description like, "cathedral with Socrates," or something, and it produces some image, right? And they're a lot of fun to play with, and there's a lot of services like this. So, what we were targeting instead was, because our environments are 3D, one of the building blocks of making a 3D scene is that you separately define the shape of objects, the geometry, usually by a polygon mesh.

And at this point, I think everybody has seen the sort of white-outline wireframe kind of behind-the-scenes stuff. So, you just define the geometry. Then you essentially paint the geometry with materials, and a lot of the detail is not actually in the shape; it's just sort of painted onto the surface as a texture, so to speak. There's a couple levels of that. One is it's literally the color that each pixel will be when the object appears on the screen.

The more sophisticated level, which is what most of Roblox's technology is based on, is called physically based materials. And the idea is, instead of painting the colors of objects, you're painting the chemical and physical properties of the material: how it's going to reflect light. Is it smooth and shiny? Is it rough like dirt or rubber? How will it respond to light passing through it with translucency? And so, creating the geometry is something that's a little bit more accessible. And then creating the materials, because they have to have these physically based properties, means you need so many of them.

You might reuse the same car, but you might have it with lots of different colors and materials. The materials are sort of higher up the value chain if you want to automate that more quickly. And then there are other aspects which we're also working on automating, such as how objects move. And in a traditional studio, these are separate skills. There's sort of this assembly line of people who make the geometry, then people paint it, and then people animate it.

So Material Generator was attacking that middle of that pipeline, and we brought that kind of technology of: type in a sentence describing what the material looks like, and it will paint that onto the object. And for the original Material Generator, which we've since released a more sophisticated variant of, actually just last week, the Material Generator was traditional 3D modeling kinds of materials where it's essentially just a square, and it tiles that. So if you think of one tile in your kitchen of linoleum or something, and then they sort of go out.

And you can see this in many 3D environments sometimes, or in the distance, you might realize the grass is repeating or something like that. The new version that we just released is the Texture Generator, we call it, which isn't about tiling; it's specialized. So, we have a lovely example of this backpack. And instead of just saying, "I want a red leather backpack," and having a square of leather that gets repeated, it actually specializes it so that it paints the buckle metal instead of making that red leather as well.

And it paints the creases in where there's been wear and some dirt, and then the little studs on the rivets. And so, that's what the most sophisticated 3D modelers will do, is that kind of specialization to this object. And that's obviously a really tedious, really expensive step of creation. If you're making an entire world, having to make a unique texture for every backpack would be very expensive. And so we are automating and helping human artists to take something that's been a really slow part of their pipeline and accelerating it in much the way we accelerate our own release iteration schedule.

So everything we've done for generative AI in the past, and certainly our current vision for where we're going in the future, is that we want to take the tasks where the creativity is separable from the execution of the creativity and give you automation of the execution without taking away the creator's intent and agency. And we love cases, which I think both the Texture Generator and the Material Generator do, where the output is something that, if you're satisfied with it, you can ship that. And that's good, especially if you're new to creation, where you might not have the skill set to go beyond that and you're limited by your skill set.

But as with the code editor, everything we do has the vision of wanting you to always be able to, if you have additional skills, take that output and edit it and customize it. And so it's really about getting you off of the blank page and sort of springboarding creation. But at no point do we ever want to take away your ability to customize it down to the individual pixel if you want.

So this is in the category of—I would break down our generative AI work across four categories. There are things for communication, like we do natural language translation. You can type something in Spanish and I can see it in English in real time. There's safety, which is things like voice moderation or image moderation. Creation tools, which are—I would say avatar auto-setup is probably the most powerful, but a great example is the material generator we talked about. And then there is research, where it's not product, it's fundamental work we're doing that'll probably affect multiple product lines inside the company, and usually where we're doing it in collaboration with universities, sometimes with other companies, and we're sharing the results with the world.

So we're trying to advance the state of the art aggressively, and it's not about a specific product, it's about changing the way we think about AI and how powerful it is. I would say three of our recent contributions in that space have been StarCoder and StarCoder 2, which are large language models, and there's a large collaboration with a lot of participants; ControlNet; and AdaptNet. And AdaptNet is a version of ControlNet. So let me talk about ControlNet specifically. The idea of ControlNet was when we had something like—and this is part of what's powering the Material Generator.

A lot of generative AI tools that have come out in research and things like that are able to, from a small amount of information, generate a complete result for you. And then it might be a result that wasn't really what you had in mind, and you don't have any recourse. Previously, most image generators, for example, you would type a sentence, you would get back an image, and if you didn't like the image, the only way you could express an edit was to go and change the sentence and rerun the entire process.

This is called prompt engineering, and it's led to some pretty ridiculous prompts. So you end up with—the dream is you're typing a sentence in natural language like English, and the practice is it looks like you are a hacker. You're typing "photorealistic" and "church" and "Hudson River School." And then you say "Hudson River School" five times, and somehow that makes it—it's nothing that you would actually tell another human. And so, to me, that's not generative AI. That's a really strange programming language that's undocumented.

And so I think the big challenge that the field is facing, probably in the next two or three years, that we've been focusing on is: how do you control generative AI? How do you make it so that you can iterate in the way you would with a human? And so with ControlNet, and this is in collaboration with Stanford University, Professor Manish Agrawala and his students who work with us, we took the Stable Diffusion algorithm, which is one of these image generators, and said it's a whole complicated neural network.

It has many stages. It's this big, big thing. And usually when you try to make any changes to a large network, you run this on a supercomputer for a week. And so in terms of the carbon impact of doing that, in terms of the financial cost, in terms of the time cost, it's going to be a week before it does anything. That's not a great way to control AI, is retrain it from scratch. And we don't think having the creator control AI by completely changing their input is a great way.

So instead, the way that, for a human, if I was working with an artist, I described an image and they gave me back an image and I didn't like it, I would draw on it to communicate something. And so I might say, "I want the main character here, and I want the secondary character here, and I want a skyscraper in the background." And I would draw those lines, and then they would go and adjust their composition. And prior to ControlNet, there was no way to give this kind of input.

To an AI image generator. What ControlNet does is we create a second neural network. It's the control, and it's very small and lightweight, and it latches onto kind of the brain of the AI. And so it's very fast to train because it's a little thing, but it pokes into all different levels and it controls it. So it's kind of mind-controlling the original AI. And for the examples that we started with, you can control it by saying, "I want the edges of the image to match the edges of this other image."

I want to preserve this aspect. So you would express things that were sort of visual image concepts directly in the space of the image. And you could do things like get back an image of a deer and say, "I like the deer, I like the composition, but I was thinking of winter instead of a deer in summer." And it can lock all the edges of the image and say it has to be that same exact composition, but now I'll add winter and rerun it with that constraint.

With that control. And what we've since been doing—and this was incredibly powerful because it meant that, for the first time, you could control a generative AI in a natural way. And you could even start to mix and match. Maybe you would 3D render a blocky version and say, "I want that setup, but I want this sort of style or this amount of detail." And so, once you had that control, we've started applying it to other things. So we did this with animation. AdaptNet is our example of: can you make characters where you have a stock animation, but we say, now the character is twice as tall, or the character is struggling with a heavy weight, or the character's a knight who came out of battle and their arm was injured—they're going to hold themselves differently.

So we're able to inject that lightweight, fast, cheap, ecologically friendly-to-train neural network onto a large pre-trained network and get really controllable animation out. And as you would imagine, we're expanding this and just looking at every place we could possibly do this. One of the aspects that makes the Material Generator so effective is that it's using this sort of technology in order to adapt to the exact objects that you're working on instead of making that tiled version. And so that was a real breakthrough that we're using throughout our pipeline.

Yeah. So StarCoder was a multi-site collaboration between academia and multiple companies. One of the leads is Arjun Guha, who works with us at Northeastern University. And there was this problem that there was this arms race, essentially, between—you were hearing about these large language models, things like GPT. All of these companies were producing them and claiming theirs were better. They were all closed source, so you can't really get in there and use it. You don't know what it is.

You're restricted by a company's terms of use, and you can't really deeply evaluate it. You can only go through their interface. You can't sort of get into the actual technology. And so we felt like we have a need for a language-based AI model that we can have control of, but a lot of people have this need, and it would benefit the world and the advancement of science and technology if, instead of each company making their own, we got together with several groups and said, let's just make one and then share it publicly and share what we've learned.

And we focused initially on the source code generation aspect because it was a key problem for us, of course, because we wanted to do code assist. But it's also something where it's easy to get licensable, legally, ethically reusable source code to train on, versus arbitrary text, which runs into all kinds of issues with copyright and permissions and personal information. And so we wanted to kind of steer clear of that and be really careful with this public model.

So we trained StarCoder. We trained it on many programming languages. And we were able to, at the time of the initial publication, beat them on quality. So essentially, we'd give it a programmer interview, right, that you would give to a new graduate applying for a programmer job. You would give those kinds of problems to StarCoder, and it was doing extremely well in the benchmarks against all these other large language models, which were not open. We then worked on making smaller versions of it so it'd be more efficient to run and responsive.

And there's a trade-off of how much quality are you willing to give up for more performance, right, and tuning at different points. And then StarCoder 2 was advancing the science behind that: new ways of training, new ways of doing that domain transfer, of validating the output. And we've just released the model, we're showing benchmarks, and these are completely open. Anybody can use them, no strings attached. But for us, the most important part was what we learned and the learnings we shared of the process, because there's always going to be another model every month from someone.

And the idea is you can take those learnings and you can change your model out every month, or at least if you have an architecture like we do. But those learnings are how you make the new model run better and more effectively.

Yeah, absolutely. And a lot of, as I mentioned right at the beginning, a lot of the challenge of AI is we have decades of experience in software engineering and how to do traditional software development in a way that's maintainable, in a way that's efficient, that avoids bugs, that has security best practices. For AI, because you're essentially programming by feeding data into the process, we're not starting from scratch. There is work on this, but it's not the level of sophistication that we have for software engineering.

As a field as a whole. And so a lot of the learnings in AI are not on the technology side. It's not about creating a new AI model from scratch. It's about learning how to prepare data, how to augment the data so that you don't need quite as much of it, learning how to normalize or regularize the data so it all sort of fits into a standardized format, learning how to prevent bias so that if there's some bias in the underlying training set—maybe it preferred, there are more red images than blue images or something—you don't want that artifact to come out in the net result.

And how to prevent things for generative AI like hallucination, where it's sort of extrapolating meaninglessly beyond the training data. So most of our AI papers are about either applying a model in a new way, like applying ControlNet to animation and what do you need to do to customize that, or about ways of preparing data and training more effectively. That's where a lot of the advances are coming right now in the field. That's what the StarCoder papers are primarily about. The fact that we gave away a free large language model is really a side effect of the process.

The goal was actually understanding and publicly sharing how to build a large language model that's high quality.

So everyone, we have radically different backgrounds for people and sort of ways that they found their way to Roblox. Part of that is that Roblox is really a unique business in terms of the space we've carved out. We've been doing this since about 2004. I think we might have incorporated in 2005. We've been doing it for a long time, but we were the lone UGC 3D social platform. There aren't people who come to Roblox having worked at other social 3D UGC platforms, right?

Everybody has to have something adjacent they did. And so we have people who came from academia, and many of our researchers, for example, were professors at prestigious universities before working on these kinds of problems, working on AI or 3D or distributed systems or networks before they came to Roblox. We have people who come from video streaming companies, from traditional text- or image-based social media, people who come from video game companies, from film companies. And so we get this real mix of people.

And also, we've grown so much, especially in the last five years as a company. I mean, we went from a few hundred to a few thousand very quickly as our user base grew. And one of the perspectives I have is, if you look at a company, you sort of divide the number of users by the number of developers, and you think, like, I am personally supporting a million people, right?

It's quite intimidating. And our ratio there is kind of shocking if you compare it to some of the Fortune 500 companies and think about how many in-house people do they need per thousand developers or something. So everybody has a different background. My personal background is, I've always been working on both sides, and this is a burning candle at both ends; it's not straddling a fence in terms of metaphors. I've always been working in industry, usually something adjacent to 3D creation.

It's been compilers, video games, it's been distributed systems technology, but it's all been sort of in service of allowing people to create at massive scale online. So the original kind of science-fiction virtual reality and metaverse vision is what personally motivates me in the work that I've done. And I've gotten to work at lots of just absolutely fantastic companies doing that: NVIDIA, Unity, Activision, just lots of terrific places, and learned so much and seen so many problems. As well as, I love teaching, I love science, I love working on the unknown and really pushing that frontier back, and I love the external engagement, right?

So, not just working with people within whatever institution you're at, but reaching out and meeting people in different companies, different universities, different countries, and learning from them. So, I taught through—I was a full professor at Williams College in the computer science department. So it was like lovely classic New England wooded, stone Hogwarts-looking buildings kind of place. And that was fantastic, and just had some of the absolute best colleagues and best students in the world and loved that for that sort of curated niche.

It's the John Irving. It's what you think of higher education, and it was fantastic, rewarding. But then at the same time, I was working with all these great entertainment and tech industry companies as well and collaborating for my research. And so I've always been driven by this vision of: I want to be able to create at scale. I want everybody to create at scale. I want to live inside of these fantasy worlds. And I love video games and movies and all of that.

So for me, I've never changed what I'm doing. I mean, since I was a little kid, my brother and I taught ourselves to program because we weren't allowed to have video games. And we've both ended up spending a career in the video game industry, is the fantastic thing. So I guess you shouldn't let your kids play video games if you want to motivate them to be a game developer. But I think it was inevitable. My whole family's in the entertainment industry as well.

So from that, to me, it was just, where is the place where the most exciting thing is happening on this problem that I want to work on, of making virtual worlds come to life, making them more accessible? And so I've been really fortunate and just had lots of opportunities at various places, including Roblox, which I actually worked at twice, of, hey, right now, the thing I want to build is exactly what this company needs. And then I show up and get to work there.

And then in some cases it'll be, well, now this place has the thing I want, so I'm going to go over there. And right now, because we've built the social 3D app, it's at scale, but we have the data and we have the servers and we have the experts from across different domains in the company. This is the most exciting place in the world to do that kind of research. And it's the place where I write a paper, I get to share it with the world, but that's not the end where it really changes the world.

And Roblox is the place that's able to productize the ideas and really push the envelope forward. And so that's both on the scalability side and the distributed systems, like all of that server infrastructure we've built. But increasingly, generative AI is the opportunity of this academic generation and these problems, and especially in 3D. So text is hard, video, images, but 3D is the ultimate. Making interactive 3D—there is no thing past that. That's the ultimate media.

And so by tackling the hardest problem and doing it with sort of the best resources, for me, it's just the perfect place to be right now.

Thank you, Matt. That was terrific. Great questions.