Pinterest is near and dear to our hearts at FirstMark because we had the good fortune of being the first institutional investor back in 2009 when the company was just getting started (fun fact: the founders were in New York for a brief moment in time before moving to the Bay Area). Pinterest has had a remarkable ride ever since, and it’s a $49B market cap public company at the time of writing.
So it was a particular pleasure to welcome Dave Burgess, Head of Data Engineering, to come and talk to the Data Driven NYC audience about all things data at Pinterest.
We covered a bunch of interesting topics, including:
Pinterest’s newly open sourced project, QueryBook
The stack Pinterest uses to manage is 400 petabytes of data
The use cases for data analytics and machine learning at Pinterest
Below is the video and below that, the transcript.
Real-time data streaming is an increasingly crucial part of the data ecosystem. While financial services (trading) initially represented the bulk of the demand for streaming, the emergence of more mature technology in the space has unlocked more use cases, which in turn created more demand for better technology.
At a recent Data Driven NYC, we had a very interesting conversation with Arjun Narayan, CEO of Materialize, “the only true SQL streaming database for building internal tools, interactive dashboards, and customer-facing experiences”. Materialize is headquartered in New York and has raised $40M in venture capital money (with a new round rumored to be announced soon, at the time of writing).
This was a very educational discussion, where we covered the following topics:
What is streaming? What is Kakfa?
Why is there a need for a streaming database for analytics?
Why is SQL underrated?
What is Materialize?
Partnering with DBT to make streaming ubiquitous
Materializes’s roadmap
Below is the video and below that, the transcript.
At our most recent Data Driven, we had the great pleasure of hosting Chip Huyen, a writer and computer scientist who also teaches machine learning design at Stanford, for a fascinating and fun conversation.
We covered a range of topics, including:
What is machine learning design?
The MLOps landscape, and how it’s both overdeveloped and under-developed
What is online machine learning?
The divergence between East and West for machine learning and data infrastructure
A couple of book recommendations
Below is the video and below that, the transcript.
While it’s been around for 15+ years, Reddit has been on a tear lately: a $367M Series E round announced a few weeks ago, rumors of an IPO, and plenty of Internet action with r/wallstreetbets in particular.
Interestingly, there was a major gap for many years between the central role Reddit has been playing on the Internet and its relatively small team size. While companies like Facebook are largely AI companies (see our conversation with Jerome Pesenti, Head of AI, Facebook), Reddit’s data team was tiny.
Enter Jack Hanlon, VP Data at Reddit and our guest at our most recent Data Driven NYC event. Jack has been tasked with leading the data team into rapid growth, and we had a really interesting conversations, in particular around the following points:
How is the data team at Reddit organized? (preview: data science, data platform, machine learning, search)
What’s the data stack? (preview: switch from AWS to GCP, Kafka, Airflow, Colab, Amundsen, Great Expectations, Druid/Imply…)
What are the key use cases for data science and machine learning at Reddit?
A book recommendation: “Invisible Women: Data Bias in a World Designed for Men”
Anecdotally, Jack is our second speaker in recent memory who was a regular attendee in the early years of Data Driven NYC, before ascending to leadership responsibilities in a major Internet company! (the other being Alok Gupta, who spoke about leading data at DoorDash).
Below is the video and below that, the transcript.
In just a few years of hyper growth, Snyk has become a $2.7B unicorn, most recently raising $200M in September 2020. A developer-first security company, it has also helped usher the “DevSecOps” category.
At our most recent Data Driven NYC, we had the pleasure of hosting its Founder & President, Guy Podjarny, zooming in late at night from Israel.
We covered many interesting topics, including:
What does DevSecOps mean?
How did Snyk initially get developers to care, and how did they expand horizontally from there?
What is infrastructure as code?
Thoughts Snyk Code and Snyk’s vulnerability database
The nuances of combining a bottoms-up, freemium motion focused on developers, with an enterprise motion focused on economic buyers of Snyk’s products.
Below is the video and below that, the transcript.
If you follow the various talks at Data Driven NYC, and the data ecosystem on general, it’s plenty apparent that the overall tooling for data, data science and machine learning is still in its infancy, particularly compared to the software stack.
While this may feel ironic (yes, I really do think) given the billions in venture capital money that have been poured in the space, it’s worth remembering that the data stack (at least in its “big data” phase) is relatively recent (10-15 years), while the software stack has had several decades of evolution.
In many organizations, the data science and machine learning stack looks a collection of various tools, some open source, some proprietary, glued together with one-off scripts. Teams started experimenting with one tool, then another, then created ad hoc pathways to make it all work together over time, and before you knew it, you ended up with complex environments that are painful to manage.
In response to this situation, various machine learning frameworks have emerged to make abstract away the complexity. Several of those frameworks were developed internally at large tech comapanies to solve their own problems, and then open sourced.
Kedro is one such example. It was developed and maintained by QuantumBlack, an analytics consultancy acquired by McKinsey in 2015. It’s McKinsey’s first open-source product.
Kedro is somewhat hard to categorize. If it had its own category, it might be considered a Machine Learning Engineering Framework. What React did for front-end engineering code is what Kedro does for machine learning code. It allows you to build “design systems” of reusable machine learning code.
At our most recent Data Driven NYC, we had the great pleasure of hosting Yetunde Dada, a Principal Product Manager at QuantumBlack, who has been the key driving force behind Kedro.
Below is the video and below that, the transcript.
Although this was never publicly announced, for the last 2+ years, FirstMark was the lead investor in an exciting seed-stage company called Timber, alongside a great group of folks and firms including NextView, Notation, Addition, Lux Capital, Nat Turner, Zach Weinberg and Zach Perret.
Over time, Timber developed Vector, a very interesting open source project focused on observability data. Vector is effectively a “routing layer”, a high-performance observability data pipeline that enables customers to collect, transform, and route all their logs, metrics, and traces.
Billions of dollars have been invested in the rise of data science and machine learning as mainstream disciplines in the world of business, one of the most exciting tech trends of the last (and next) decade.
In the enterprise, many of the applications of data science and machine learning ultimately produce a prediction: which customers are the most likely to buy? Or churn? Which transactions are most likely to be fraudulent? What part of town is likely to place the most food deliveries tomorrow afternoon?
However, powerful though it may be, there is one thing machine learning generally doesn’t tell you: once you have a prediction, what do you do with it? For example, once you have predicted high demand for food delivery in a certain part of town, how do you decide which delivery team member to dispatch where and when, to optimize for efficiency and maximize revenue and customer satisfaction?
Enter decision science. While the term has not crossed over to mainstream consciousness like its data science cousin, decision science has been around for decades. Also often known as Operations Research, it encompasses a variety of advanced analytical methods and quantitative models to help with decision-making and efficiency, including simulation, mathematical optimization, queuing theory, etc.
For anyone in the data analysis community, Wes McKinney a very well known figure. In addition to literally writing the book on the topic (“Python for Data Analysis”), he’s played a leading role in several key open source projects: he created Python Pandas, he’s a PMC member for Apache Parquet, and he’s also the co-creator of Apache Arrow, his current development focus.
He’s also a serial entrepreneur, having co-founded DataPad (acquired by Cloudera) and now Ursa.
So it was a real pleasure hosting Wes for a chat at our most recent Data Driven NYC. As always, we tried to position the conversation to be approachable by everyone (with high level definitions) while being interesting for technical folks and industry experts.
Watch the video below (or read the transcript copied below the video) to learn:
What are pandas? What is a dataframe?
What is Arrow? What is its history and why is it a big deal?
Hosting Alok Gupta at our most recent Data Driven NYC was special for a couple of reasons.
First, because Alok is the very talented head of data science and machine learning in a company that has all sorts of really interesting use cases for AI and just had a phenomenal IPO, valuing it at $60B at the time of writing.
Second, because it was a homecoming of sorts for Alok, whose journey in the field of data science was inspired in part by Data Driven NYC – as he puts it:
This also feels like it nicely completes my journey starting 8 years ago when I was working on Wall Street in 2013 and started coming to your monthly evening talks at the Bloomberg building to learn more about ‘Data Science’. That was really a launching point for me to switch from trading to DS, and I’m grateful to be able to give back in a small way :).
One of those stories that brings joy to the heart of the organizers of this community!
Here are the video, as well as a full transcript for easy perusal:
There’s an inherent tension at the heart of modern data infrastructure. On the one hand, it’s becoming more mission-critical every day, as companies around the world rely on it to run their business. On the other hand, it’s more complex, and potentially brittle, than ever, an “assembly chain” involving multiple tools and repositories.
This tension has led to the emergence of DataOps as a distinct and very active segment. One particularly important area is known as “data lineage“. The concept is basically to monitor data pipelines and understand the journey of data through its various transformations and usages. This makes it possible to fix any issues that happen along the way, and go to the root of data quality, and potentially fairness, issues.
Because data lineage involves many different tools, platforms and companies, it makes sense for those different parts of the ecosystem to collaborate around standard definitions. This is the concept behind OpenLineage, a cross-industry effort involving creators and contributors from key data projects (DBT, Spark, Pandas, etc.), gathered together at the initiative of the founders of Datakin, an SF startup beyond the open source data lineage project Marquez (originally started at WeWork).
At our most recent Data Driven NYC, we had the pleasure of hosting Julien Le Dem, CTO of Datakin. His talk (video below) is very approachable and educational.
The tech industry has a rich history of startups that started with a pretty awkward name, and rebranded over time to the big brands we have come to know. Some of those changes are plain fun to remember.
A few days ago, I tweeted this, and it led to a cool thread with plenty of examples and suggestions (Twitter at its best), so I thought I’d compile the results here for easy reference.
As we close an incredibly active year in the world of data infrastructure, it was a particular treat to host at Data Driven NYC two of the most thoughtful founders in the space, for an in-depth conversation about key trends.
Tristan Handy, is the Founder & CEO of Fishtown Analytics, makers of DBT. DBT is one of the most popular, open-source, command-line tools that enable data analysts and engineers to transform data in their warehouse more effectively. Based in Philadelphia, the company raised both a $12.9M Series A and a $29.5M Series B, back to back in 2020. Tristan also does a great weekly newsletter, The Data Science Roundup.
Jeremiah Lowin, Founder & CEO of Prefect. Prefect is the new standard in dataflow automation, trusted to build, run, and monitor millions of data workflows and pipelines. As another leader in the open-source world, Prefect powers data management for some of the most influential companies in the world.
We had a wide ranging conversation, covering lots of topics: the modern data stack, data lake vs data warehouse, empowering data analysts, workflow automation etc.
Business planning is, of course, one of the vital functions in the enterprise: hard to run a successful company beyond a certain size without a clear sense for objectives and resources.
Yet, to this day, business planning is a often a cumbersome, rigid and time-intensive process. Typically led by the finance team, it is largely done through email, excel spreadsheets and meetings. In large companies involving multiple business units and geographies, the process can take several months. As a result, business planning tends to effectively happen once a year.
“It wasn’t a walk in the park. Today, Gong is a super hot company. But at that time, we got a lot of no’s, by not stupid people. There were a lot of objections, like salespeople are going to hate it as a big brother, and Google and Amazon will compete with you“, says Amit Bendov, the CEO of Gong.
From those early days of facing skepticism, Gong has indeed become a hot startup loved by customers and ushering its own category, revenue intelligence. It’s also had tremendous fundraising success with VCs, raising $305M in less than 18 months, including a $200M round on a $2.2B valuation, announced in August 2020.
We were thrilled to welcome back Amit at Data Driven NYC, where he had spoken a few years ago, when he was CEO of SiSense.
