For anyone interested in a quick overview of our long-form 2021 Machine Learning, AI and Data (MAD) Landscape, here are the Cliffs Notes! My co-author John and I did a presentation at our most recent Data Driven NYC, focused on top 10 trends in this year’s landscape.
As a preview, here they are:
Every company is a data company
The big unlock: data warehouses and lakehouses
Consolidation vs data mesh: the future is hybrid
An explosive funding environment
A busy year in DataOps
It’s time for real time
The action moves to the right side of the warehouse
The rise of AI generated content
From MLOps to ModelOps
The continued emergence of a separate Chinese AI stack
Below is the video from the event, and below that, the transcript.
In the admittedly small world of people who obsess over data technologies, one of the hottest topics of the last year has been the “data mesh”.
Created by Zhamak Dehghani of ThoughtWorks, the concept struck a chord and made the rounds in countless conversations on Twitter and elswhere.
As I highlighted in the 2021 MAD Landscape, the data mesh concept is both a technological and organizational idea. A standard approach to building data infrastructure and teams so far has been centralization: one big platform, managed by one data team, that serves the needs of business users. This has advantages, but also can create a number of issues (bottlenecks, etc). The general concept of the data mesh is decentralization – create independent data teams that are responsible for their own domain and provide data “as a product” to others within the organization. Conceptually, this is not entirely different from the concept of micro-services that has become familiar in software engineering, but applied to the data domain.
It was a real treat to get to chat with Zhamak at our most recent Data Driven NYC.
Below is the video and below that, the transcript.
At our most recent Data Driven NYC, we had the great pleasure of hosting Bindu Reddy, CEO and co-founder Abacus AI, and formerly GM & creator of AI verticals at AWS, and an ex-Googler. Bindu also has a very witty and entertaining Twitter account (@bindureddy), where she talks about all things machine learning and AI.
This was a very educational and approachable conversation, where we covered:
some key definitions: neural networks, weights and biases, supervised vs unsupervised learning, feature store
Applying neural networks to structured, tabular data
Abacus’ vision around “autonomous AI”
How companies wait too long to start experimenting in ML/AI
Below is the video and below that, the transcript.
Pinterest is near and dear to our hearts at FirstMark because we had the good fortune of being the first institutional investor back in 2009 when the company was just getting started (fun fact: the founders were in New York for a brief moment in time before moving to the Bay Area). Pinterest has had a remarkable ride ever since, and it’s a $49B market cap public company at the time of writing.
So it was a particular pleasure to welcome Dave Burgess, Head of Data Engineering, to come and talk to the Data Driven NYC audience about all things data at Pinterest.
We covered a bunch of interesting topics, including:
Pinterest’s newly open sourced project, QueryBook
The stack Pinterest uses to manage is 400 petabytes of data
The use cases for data analytics and machine learning at Pinterest
Below is the video and below that, the transcript.
Real-time data streaming is an increasingly crucial part of the data ecosystem. While financial services (trading) initially represented the bulk of the demand for streaming, the emergence of more mature technology in the space has unlocked more use cases, which in turn created more demand for better technology.
At a recent Data Driven NYC, we had a very interesting conversation with Arjun Narayan, CEO of Materialize, “the only true SQL streaming database for building internal tools, interactive dashboards, and customer-facing experiences”. Materialize is headquartered in New York and has raised $40M in venture capital money (with a new round rumored to be announced soon, at the time of writing).
This was a very educational discussion, where we covered the following topics:
What is streaming? What is Kakfa?
Why is there a need for a streaming database for analytics?
Why is SQL underrated?
What is Materialize?
Partnering with DBT to make streaming ubiquitous
Materializes’s roadmap
Below is the video and below that, the transcript.
At our most recent Data Driven, we had the great pleasure of hosting Chip Huyen, a writer and computer scientist who also teaches machine learning design at Stanford, for a fascinating and fun conversation.
We covered a range of topics, including:
What is machine learning design?
The MLOps landscape, and how it’s both overdeveloped and under-developed
What is online machine learning?
The divergence between East and West for machine learning and data infrastructure
A couple of book recommendations
Below is the video and below that, the transcript.
While it’s been around for 15+ years, Reddit has been on a tear lately: a $367M Series E round announced a few weeks ago, rumors of an IPO, and plenty of Internet action with r/wallstreetbets in particular.
Interestingly, there was a major gap for many years between the central role Reddit has been playing on the Internet and its relatively small team size. While companies like Facebook are largely AI companies (see our conversation with Jerome Pesenti, Head of AI, Facebook), Reddit’s data team was tiny.
Enter Jack Hanlon, VP Data at Reddit and our guest at our most recent Data Driven NYC event. Jack has been tasked with leading the data team into rapid growth, and we had a really interesting conversations, in particular around the following points:
How is the data team at Reddit organized? (preview: data science, data platform, machine learning, search)
What’s the data stack? (preview: switch from AWS to GCP, Kafka, Airflow, Colab, Amundsen, Great Expectations, Druid/Imply…)
What are the key use cases for data science and machine learning at Reddit?
A book recommendation: “Invisible Women: Data Bias in a World Designed for Men”
Anecdotally, Jack is our second speaker in recent memory who was a regular attendee in the early years of Data Driven NYC, before ascending to leadership responsibilities in a major Internet company! (the other being Alok Gupta, who spoke about leading data at DoorDash).
Below is the video and below that, the transcript.
In just a few years of hyper growth, Snyk has become a $2.7B unicorn, most recently raising $200M in September 2020. A developer-first security company, it has also helped usher the “DevSecOps” category.
At our most recent Data Driven NYC, we had the pleasure of hosting its Founder & President, Guy Podjarny, zooming in late at night from Israel.
We covered many interesting topics, including:
What does DevSecOps mean?
How did Snyk initially get developers to care, and how did they expand horizontally from there?
What is infrastructure as code?
Thoughts Snyk Code and Snyk’s vulnerability database
The nuances of combining a bottoms-up, freemium motion focused on developers, with an enterprise motion focused on economic buyers of Snyk’s products.
Below is the video and below that, the transcript.
If you follow the various talks at Data Driven NYC, and the data ecosystem on general, it’s plenty apparent that the overall tooling for data, data science and machine learning is still in its infancy, particularly compared to the software stack.
While this may feel ironic (yes, I really do think) given the billions in venture capital money that have been poured in the space, it’s worth remembering that the data stack (at least in its “big data” phase) is relatively recent (10-15 years), while the software stack has had several decades of evolution.
In many organizations, the data science and machine learning stack looks a collection of various tools, some open source, some proprietary, glued together with one-off scripts. Teams started experimenting with one tool, then another, then created ad hoc pathways to make it all work together over time, and before you knew it, you ended up with complex environments that are painful to manage.
In response to this situation, various machine learning frameworks have emerged to make abstract away the complexity. Several of those frameworks were developed internally at large tech comapanies to solve their own problems, and then open sourced.
Kedro is one such example. It was developed and maintained by QuantumBlack, an analytics consultancy acquired by McKinsey in 2015. It’s McKinsey’s first open-source product.
Kedro is somewhat hard to categorize. If it had its own category, it might be considered a Machine Learning Engineering Framework. What React did for front-end engineering code is what Kedro does for machine learning code. It allows you to build “design systems” of reusable machine learning code.
At our most recent Data Driven NYC, we had the great pleasure of hosting Yetunde Dada, a Principal Product Manager at QuantumBlack, who has been the key driving force behind Kedro.
Below is the video and below that, the transcript.
There’s an inherent tension at the heart of modern data infrastructure. On the one hand, it’s becoming more mission-critical every day, as companies around the world rely on it to run their business. On the other hand, it’s more complex, and potentially brittle, than ever, an “assembly chain” involving multiple tools and repositories.
This tension has led to the emergence of DataOps as a distinct and very active segment. One particularly important area is known as “data lineage“. The concept is basically to monitor data pipelines and understand the journey of data through its various transformations and usages. This makes it possible to fix any issues that happen along the way, and go to the root of data quality, and potentially fairness, issues.
Because data lineage involves many different tools, platforms and companies, it makes sense for those different parts of the ecosystem to collaborate around standard definitions. This is the concept behind OpenLineage, a cross-industry effort involving creators and contributors from key data projects (DBT, Spark, Pandas, etc.), gathered together at the initiative of the founders of Datakin, an SF startup beyond the open source data lineage project Marquez (originally started at WeWork).
At our most recent Data Driven NYC, we had the pleasure of hosting Julien Le Dem, CTO of Datakin. His talk (video below) is very approachable and educational.
As we close an incredibly active year in the world of data infrastructure, it was a particular treat to host at Data Driven NYC two of the most thoughtful founders in the space, for an in-depth conversation about key trends.
Tristan Handy, is the Founder & CEO of Fishtown Analytics, makers of DBT. DBT is one of the most popular, open-source, command-line tools that enable data analysts and engineers to transform data in their warehouse more effectively. Based in Philadelphia, the company raised both a $12.9M Series A and a $29.5M Series B, back to back in 2020. Tristan also does a great weekly newsletter, The Data Science Roundup.
Jeremiah Lowin, Founder & CEO of Prefect. Prefect is the new standard in dataflow automation, trusted to build, run, and monitor millions of data workflows and pipelines. As another leader in the open-source world, Prefect powers data management for some of the most influential companies in the world.
We had a wide ranging conversation, covering lots of topics: the modern data stack, data lake vs data warehouse, empowering data analysts, workflow automation etc.
“It wasn’t a walk in the park. Today, Gong is a super hot company. But at that time, we got a lot of no’s, by not stupid people. There were a lot of objections, like salespeople are going to hate it as a big brother, and Google and Amazon will compete with you“, says Amit Bendov, the CEO of Gong.
From those early days of facing skepticism, Gong has indeed become a hot startup loved by customers and ushering its own category, revenue intelligence. It’s also had tremendous fundraising success with VCs, raising $305M in less than 18 months, including a $200M round on a $2.2B valuation, announced in August 2020.
We were thrilled to welcome back Amit at Data Driven NYC, where he had spoken a few years ago, when he was CEO of SiSense.
Sisense is a fast-growing business intelligence startup that was ranked #31 in this year’s Forbes Cloud 100, and reached unicorn status at the beginning of 2020 through a $100M Series D led by Insight Partners.
We’ve had Sisense speak twice at Data Driven NYC over the years, first CEO Amit Bendov (now CEO of Gong) (video of the talk here) and then new CEO Amit Orad (video of the talk here).
With all the recent progress, we were particularly excited to hear the update and welcome Ashley Kramer, who recently joined Sisense as Chief Product and Marketing Officer, after a very impressive run at Amazon, Tableau and Alteryx.
We covered a bunch of topics, including:
What does “Business Intelligence” actually mean?
The convergence of BI and data science
How does Sisense position in the context of the consolidation of the BI industry (hint: multi-cloud and focus on different personas, including business users, data analysts and more technical folks)
Where Sisense sits in the modern data stack
How Sisense has been building data network effects with its knowledge graph
Dashboards are great, but embedded analytics are better
As always, Data Driven NYC is a team effort – many thanks to Jack Cohen for co-organizing, Diego Guttierez for the video work and to Karissa Domondon for the transcript!
David Cancel puts the “serial” in serial entrepreneur. David has founded a total of five software companies over the years, which he says make him “certifiable”. The list includes Performable, which was acquired by Hubspot, where David subsequently spent three years as Chief Product Officer.
In 2015, David left Hubspot to start Drift, a Boston-based conversational AI platform for marketing and sales. The company has grown very rapidly and now has a whopping 50,000 customers. Drift has raised a total of $107M from a number of venture firms including Sequoia, General Catalyst and CRV. The company has also been recognized as a Forbes Cloud 100 company.
David also has built a very strong presence and brand in the entrepreneurial community. He writes a popular newsletter, ‘The One Thing’ and hosts a long-running podcast, ‘Seeking Wisdom’. He’s very involved in a number of startups as advisor and angel investor. He’s also an Entrepreneur-in-Residence at Harvard Business School.
David and I had a really interesting, wide-encompassing conversation at our most recent Data Driven NYC event, where we covered a range of topics including:
Building a global SaaS brand with 50,000 customers in an astonishingly short amount of time
How Drift was founded to take advantage of a fundamental paradigm shift
Creating a new type of CRM, driven by conversational data, with automation at the core
One of the biggest recent trends in the data world recently has been the rapid emergence of the “modern data stack”.
This stack is largely centered around the cloud data warehouse, with its massive scalability and elasticity capabilities. Snowflake’s blockbuster IPO this week, and the underlying performance of the company, demonstrate the level of excitement from both customers and investors about the data warehouse.
But the modern data stack is more than just the data warehouse, there’s a whole pipeline involving other technologies, where data gets collected, stored and analyzed. Downstream from the data warehouse, you find business intelligence solutions, as well as some machine learning platforms, to analyze the data. Upstream from it, you find solutions that focus on extracting data from various sources and loading it into the data warehouse (ETL/ELT).
This is where Fivetran comes in. A fast-growing company with a unicorn status, it automates data integration from source to destination, through a large library of connectors.
It was very fun to host Fivetran’s CEO, George Fraser, at our most recent Data Driven NYC event. We had a great conversation, both very approachable for a non-technical audience but also interesting for more technical folks.
