I’ve been interested in the intersection of AI and crypto for a while (see AI & Blockchain: An introduction), and Numerai is one of the most exciting companies I came across in that world. Numerai is a new kind of crowdsourced quant hedge fund, which provides data for free and enables any data scientist around the planet to contribute models they believe will beat the stock market. Numerai offers its own token, called Numeraire, to incentivize participants.
As it turns out, this model delivers exciting results, and Numerai announced a few months ago that it had outperformed market neutral hedge funds by 29%.
It was a real pleasure welcoming Richard Craib, founder of Numerai, to Data Driven NYC to talk about the very exciting work Numerai has been doing.
Below is the video and full transcript.
(As always, Data Driven NYC is a team effort – many thanks to my FirstMark colleagues Jack Cohen, Karissa Domondon Diego Guttierez)
VIDEO:
TRANSCRIPT
[Matt Turck] I’ve been very much looking forward to this conversation because Numerai is so fascinating to me. It combines all the cool things I like – it’s like a hedge fund, it has AI, and it has network effects, and it has crypto and token economics. So all the cool things in just one company. Let’s start from the top, what is Numerai?
[Richard Craib] Well, it’s definitely all those things.
The thing we’re trying to solve is predicting the stock market and getting really good intelligence about that particular problem. But why it’s interesting is that a lot of people think maybe that’s impossible and so how would you go about it if it was considered this very, very hard problem?
The way we go about it is by first of all, giving away all of our data so that anyone can build any model that they want on our dataset using any machine learning algorithm or any approach.
The next thing you need is incentives. You need incentives not to overfit, right? You don’t want to make a back test that looks good on historical data, but doesn’t look good on live data. And so we make our users stake cryptocurrency on their models and so we have the best modeling talent because we have much more, many more data scientists working on Numerai than any other hedge fund in the world and we have the best incentives because everyone who’s working on the models that we use for trading are staking them, so they’re fully aligned. And so putting those things together over time, we’re just getting better and better and better and better at this problem and it’s starting to really work at the moment.
This is really working. You published a few months ago, the first numbers that you talked about publicly and as it turns out you beat some of the very best quant hedge funds in the world by quite a wide margin.
Yeah. So obviously this disclaimer is past performance doesn’t mean future results and things like that. But we have done quite well and what we’re doing is running the whole strategy in a market neutral context so it only makes sense to compare it with other market neutral hedge funds and there aren’t many-
What does “market neutral” mean?
It means you don’t take any net exposure to the market. Meaning every $100 of long positions you have in different companies, you also have to have $100 of short positions. We are never taking any directional exposure to the market and for example in March 2020 when the market fell 30% in about 20 days, we were down one and a half percent. So whatever’s happening in the market would not give you an indication of how well we are doing. Whereas most funds, that you heard of like a Warren Buffet kind of fund, or Cathie Wood of Ark, these types of funds are very exposed to basically risk factors in the market. So we’ve built a market neutral strategy and it’s a sophisticated strategy that’s really suitable for institutional investors who already have tons of market exposure. They don’t need us to buy stocks for them, they want a sophisticated uncorrelated strategy and so that’s what we’ve built.
To recap: you’re an AI powered sort of crowdsourced hedge fund where a bunch of data scientists around the world can contribute models based on data that you provide and sort of compete, and also put skin in the game via cryptocurrency. So assuming that’s correct, let’s break down those different components. So the data part, so obviously as everybody that attends those events knows the AI and machine learning really depends on the quality of data, in the hedge fund world that’s particularly important. Where does the data come from? What kind of data do you provide? And how do you anonymize it as well? So I guess three questions in one.
That is the one unique thing. When I say we give out all of our data, it’s all in this obfuscated way. If you look at the features of our data, you have no idea what they correspond to. They might correspond to the PE ratios of stocks, or the momentum or anything, and there are thousands of them but you don’t know what they are. And so all you have is the data and you also don’t know what the rows are, you don’t know what stocks correspond to what features.
So it’s pure numbers?
It’s just pure numbers. It’s literally like millions and millions of numbers between zero and one and that’s what you get when you download our data. If you are a quant or someone who works in finance, you’re typically like, “Whoa, I can’t even use this website.”
This isn’t a finance problem or a quant problem, it is purely a data science problem and so we’ve sort of massaged all the data into this pure data science problem. People who have expertise with machine learning are extremely used to this. The people at Google Translate are not really good at languages, they’re good at machine learning and in a Kaggle competition, you can model healthcare data, you can model whale sounds data, you can model financial data, it’s all the same problem.
And that’s what people didn’t get and so even though they’ve maybe been attempts to kind of crowdsource stock tips from the internet, this is not what Numerai does. Numerai is about crowdsourcing machine learning models built on our very kind of custom data set.
Where do you get the data from?
We get it from a lot of sources. Some of this data would be quite expensive for an individual to buy it. We probably spend more than a million dollars a year on data but what we’re giving out is so different from what the raw data is that it’s very, very, very far removed from any of the source data. But yeah, it is important to know I think there’s some discussion around alternative data in finance.
“Alternative data”, for folks that don’t spend time in that space. is data (like traffic on parking lots, etc) that is not typically financial but that hedge funds have been mining to extract signal from over the last few years.
Exactly. That’s quite a popular thing to discuss but in a weird way, the more alternative the data, the shorter it usually is like it doesn’t… for example, if you had some data on social media, maybe it only goes back to 2008, sentiment data on social media so it only goes back to 2008. So then that’s not very long because that’s pretty much one gigantic boom cycle since then. So you might not be able to learn very much that will generalize out of sample on short data sets like that. So I would say in general, we have some alternative data, lots of alternative data, but we are mainly focusing on data that’s very, very long. And so the more data you have, the better your model’s going to be and the more robust it will be to future events.
What are the participants that contribute the models? Are they young aspiring people that are trying to build a resume? Are they people that do that as a side job?
They’re all kinds of people and some of them are anonymous and some of them we don’t know.
In fact, some of the best ones, we kind of have no idea but some we have with one really good user who works at NASA Jet Propulsion Lab and who’s really strong at data science and he’s actually working on a mission to one of Jupiter’s moons and then on his other tab, he’s got Numerai open and he’s monitoring stock market.
And I actually went to visit him at NASA and that’s crazy that, this is real. And there actually others, another guy at CERN who’s really amazing, one of the top users but yeah, you don’t have to reveal who you are. And I will say there are a couple of people who maybe were interested in finance but didn’t know machine learning and then they’ve kind of self taught. So we had one PhD or professor of finance who’s a big, big data, a scientist on Numerai but it could be anyone, but
Is Numerai a good place to learn? It is is described as the hardest data science competition in the world. If I’m a student and I’m looking to learn, is that a good place or is that something I should do later in my career?
It is a good place to learn, but it is not for the faint of heart and that’s why we do say it’s the hardest data science tournament, because it’s a very difficult problem. And some of the normal machine learning problems that you’re trying to classify spam or something, you can get like 89% accurate. On Numerai, to get a one or two per percent correlation with the future returns of the stock market, it’s very difficult. So a lot of the best people are using… yeah, very new techniques and you might struggle just messing around but it’s definitely worth trying and it’s instant. You can instantly download our data and create a model within a few minutes to get started, but it’s hard to be good.
You mentioned that concept of tournaments, I saw that you have two running, you have a Numerai Tournament and Numerai Signals, if I read correctly. What are those and why do you have two?
Good question. So Numerai is… we give you this data, you model it. Signals is – go find your own data, use whatever data you want and then model your own data and then submit that. So this idea is… Numerai’s mission is to kind of monopolize intelligence, and monopolize data and then monopolize money and then decentralize the monopoly.
That’s what you call the “Master Plan> That’s awesome. I wish I had a master plan that sounds pretty awesome to have one.
It has been pretty good to direct our energies and we’ve never changed it, so yeah. On Numerai, we’ve got this data and so it’s like, “What about this?” There are people who are very good at modeling but don’t have any access to data. Let’s make sure we can assimilate their intelligence into our hedge fund. And then there are people who already have models, but they have no way of trading because they don’t want to be… they need $10 million to use a prime broker or whatever. So then let’s make sure they can submit and that’s what Numerai Signals is. So we are trying to just get all these different sources of intelligence into our fund so that we can be the best and ultimately manage all the money in the world.
Do you have a concept of meta model, to combine all the models from the community?
We do. So everyone’s providing signals on all 5,000 stocks and global equities and we take the stake weighted average of all those signals. So sort of true, if a lot of people have staked on Google then with Google as a high rank in their Signal, then that stock would be something the meta model would want to buy next.
Do you want to explain what staking is for people that don’t necessarily spend time in crypto?
Staking is a very simple but important thing you can do with a blockchain. And the idea is you can lock up some cryptocurrency in a smart contract and no one… you’re not trusting anyone to keep it so it’s a bit like an escrow. You’re not trusting anyone to not steal it and so Numerai, you come in with your NMR cryptocurrency, that you can buy on Coinbase or something, and-
And that’s your currency, right? NMR stands for Numeraire?
Exactly. That’s our own one and you can take it and stake it. And if your model performs well on Numerai, we’ll give you more NMR tokens and if your model performs badly, we can burn your stake. So it’s this very strong incentive to make sure that your model performs well on live data.
So it’s having skin in the game.
Exactly, exactly. And it’s a critical thing. In some ways, that’s almost like the problem of the internet. You don’t have anything at stake and therefore, you can do things that are bad like you can make 10,000 accounts on Numerai and just hope that one of them gets lucky. If you’re told you have to stake, well, then you’re only at a stake on the ones that are good, that are actually likely to work. So you’re basically policing yourself more because you have something at stake.
And you said that your model is more likely to make it to the final calculation if you stake more?
Yeah, we actually only trade the models that are staked.
What if you take more in NMR, you’re more likely to be taken into account?
Yes. If you are 5% of the total amount staked, which would be high like half a million dollars, then we would waive you 5% in the meta model. So the more you believe in yourself, the more we believe in your model.
Presumably you believe in yourself because you tested your model on past data and you got a high performance therefore you’re willing to put your money where your mouth is, okay.
There’s all kinds of things you can do to cross validate a model really well. It’s not really like we’re trying to test whether the person believes in, has a thesis about stocks or something. It’s much more like how many different kinds of tests have you done with this more model that you’re willing to ship it, to be part of the meta model and their stake on?
And how does that work putting all those models together? Is there like an assembling technique that’s well known or is that something that you guys build?
We do, it’s literally as simple right now as the stake weighted average. So it’s just the average with the stake weights and we’ve actually tried many times to beat that but it’s hard to beat because if we try something else like, “Oh, why don’t we just up weight the ones that have done well recently?” Then suddenly they don’t work and they start doing badly and so you always want to trust the users to be the ones who know the most and they’re expressing how confident they are with the stake so we can never really beat the stake weighted model. However, as it comes to trading time in this process, we still have to choose what portfolio to construct. So when we have the meta model, we have predictions on 5,000 stocks and we have to decide which ones to hold in our current portfolio and that is an optimization process that we are in charge of. So we’ve made our own custom optimizer that minimizes all kinds of market risks and ends up with a final portfolio.
People are making real substantial money with this, right? Did I read correctly? I think I saw a 42 million number of payout.
Yeah, we have. Yeah, we have paid something like that. We’ve even… I think even just last year we paid 10 million. So it’s so much higher than… the rewards are so much higher than the other data science competitions online by orders of magnitude. And there’re some users that have made more than a million dollars and the reason is kind of because of crypto, right? Our cryptocurrency is what we use to pay people. So the thing you do have to get your head around is, do I really want to have the risk of holding this cryptocurrency which is quite volatile? But some users buy in for a small amount and their stakes end up doing well. And it’s also important to note, when you do badly on Numerai, it’s not like Numerai does well, it’s not like we take your stake, it’s not like we’re the house or something. All the staking is happening on the blockchain and if you do badly, it’s just that the NMR gets burned. So it’s very user aligned and that’s why I think we have… yeah, about right now over $15 million at stake.
Fascinating. A question from Tony in the group asking about the business model or revenue model, if the data science is outsourced.
So the business model is simple, it’s a hedge fund. So we sell the fund to big investors, basically, so a big investor could come and give us $100,000,000 and we would charge 2 and 20 on the investment and if the performance is really good, that is a very, very good business model. So that’s always been the idea, just have a simple business model of charging fees for how well we can do but making sure that we have something that…. because what often happens with these quant funds is they have a good year, and then they have a bad year and then they kind of die. But we wanted to build something that would credibly get better and better and better. Just a year ago, we had three times fewer users and the year before that it was three times. So it’s like 10 times more state than than two years ago and over a hundred times more at stake. So it’s growing in a way that only an internet company can grow. And not the way that a quantum traditional quant fund can grow.
Yeah. And then people keep their models, right? Or they just contribute the results of the model.
Yes, exactly. They never give us their code, they’re never assigning us IP around their models, they can train them in any way they want, keep them on any server they want. The only thing they have to do is provide us the predictions every week, the freshest predictions. So for the users, it’s kind of cool that way that they’re not trusting us, you can always quit.
Question from Philip, “How much of the performance is function of the feature engineering you expose rather than model themselves?” Could crowdsource features have a bigger impact than the models.
It’s a good question because if you know data science, which you probably do, a lot of this is about setting up the problem well and having not just good data, but also setting up the data and the problem well and a lot of the performance comes from that. But if we do say a linear model just on our data, it sort of performs okay. If we do our own internal machine learning model on the data, it performs better than that and if we use everyone’s model that’s being submitted to Numerai, it’s way better than even that. So it’s always going to be like, “Yes, there’s some edge that’s coming directly from data but it’s very much worth it to get the extra edge.” Especially in the finance domain where let’s say you’re currently 52% right. If you can go to 52 and a half percent right, it’s like a whole different level of performance for the end investor, a whole different level of sharpe ratio or returns and risk. So it’s very helpful. However, Numerai Signals is in some ways the crowdsourcing of features, people can submit whatever data they want on that. So any gaps in Numerai can kind of be filled by the crowdsourcing on Signals.
There’s a related question from Adam, “If the data is all just numbers that are unlabeled, how can you model individual assets to invest in?”
Your kind of not modeling individual assets. It’s sort of set up like a cross sectional panel data problem where you’re looking for the relationship between assets. So every model on Numerai the target variable is residual return. So it’s a return with all the factors taken out, the sectors taken out and you’re looking at it in a cross sectional way. So you’re trying to find a signal that has correlation with residual return. So no one on Numerai has any particular insight into Tesla or something but they do have insight into the relationships between stocks and those relationships can be used to build portfolios.
A question from Alex, clearly you are getting people excited about the prospects because Alex is asking, “Why not open to individuals to invest or allow users to passively stake an NMR or profits or residual your value from the fund?
It’s a good question, I mean we get that quite a lot. It is sort of unfortunate that at the end of this process, which is so driven by modern ideas, like anyone can submit, it’s a completely open system, anyone can join Numerai, anyone can stake. But at the end of this process, we have a hedge fund that is basically closed to everybody except for like seven investors in the world. And the reason it’s done that way though, is that there are regulations about selling hedge funds to people with… it’s kind of sad I think. If you have less than a million dollars or whatever it is, you’re not considered an accredited investor and therefore certain investment products are not available to you as if money is the total determinant of whether you’re sophisticated.
Yep. Hotly debated topic, yes.
Yeah. So yeah, it would be quite cool if we could have on our website, “Here’s an Ethereum address or all you have to do is send money to this address and you’ll be in our fund.” If we did that, we’d be breaking so many different rules including KYC rules, accredited investor rules and all these rules that are supposed to protect investors and secure the financial system. So that’s one of the reasons but it doesn’t preclude us from thinking about other products. So for example, mutual funds are investable by normal people and then same with ETFs. So it’s definitely crossed my mind that at some point we might be able to make an investment product that is much more suitable for the average person. There are even tax reasons why hedge funds are not the right things for people.
But yeah, it’s a good question. And in terms of NMR staking on other people, it might not help us that much. It kind of might hurt us. So the idea is, if you are just a speculator holding NMR and you want to just help Numerai but you’re not a data scientist, why can’t you stake on someone else? And the problem is why would you know anything about their future performance? The whole point of staking is to get an assessment of how much they believe their model will generalize, not just to get the person who’s the best at marketing at stake to a speculator. But I also think there might be some things in the future where we can do something like that because Numerai has been quite a… I would say kind of a speculator hostile project where we’ve basically put the data science first, put the hedge fund first. And we just happened to make a cryptocurrency right in the beginning of the whole crypto game, like back in 2017 is when we made NMR. But in the long run, there’s some exciting things happening and I’m sure we’ll be able to take NMR to the next level with ideas like that.
One last question from the group and then maybe one question from me and we’ll wrap up. A question from Patricia and my spin on the question is Patricia mentioned it normally takes a few years to know if a model is working. How do you know… how many years do you need to feel that, “Okay, well, we had a good couple of years but this thing is really working long term?
Yeah, that’s a good question and it really does depend. So there are often the heuristics in the industry of like, “Well, I’m not going to trust anyone’s track record unless it’s three years long or five years long.” And maybe that’s why people can trust like a Warren Buffet or something because he’s kind of one of the only people with 50 years long or whatever it is. But it’s not true in reality that you can’t tell things sooner than that and it just depends on how the trading happens. So for example, if I flip a coin a thousand times and I get 600 heads and I manage to flip a thousand coins in just one month, it just took me one month to do the process, let’s say. So I flip a couple of coins a day and I get to a thousand, and I get 600 heads, that is very statistically significant.
You can use a formula called the cumulative binomial distribution and you can say the odds of that happening without that coin being biased towards heads is close to 0%. Now, in a market neutral quant fund, there’re all kinds of things that are different about this coin flipping scenario. In particular, stocks are correlated with each other so they’re not like independent trials, but there are things you can do that can help the assessment. So we trade about a thousand stocks at a time, 500 long, 500 short and if every month it’s 550 long, 550 being right out of 1000 and you do that for 18 months in a row, it does get to this very credible point that you do have an edge. And so for sophisticated investors, they can be quite bright about this type of thing. If you just put all your money in Bitcoin and for three years it went up, there’s something not as good about that as if you invest in a thousand things, you and you got on average 55% of them right and so that’s the type of argument we would make about our track record.
Well, this is absolutely fascinating and actually, I would’ve had another 30 questions for you but we’re out of time. We really appreciate your coming and like telling the story to the group and congratulations on everything you’ve built it’s one of the most intellectually interesting things out there that I’ve seen in a while and on top of it all it seems to be working amazingly from a financial standpoint. So congrats and thank you.
Thanks so much, great questions too.