Pulse-check on the market and Data Orchestration's Next Big Thing

Kate Reznykova
January 17, 2023

The YC W2023 batch is on!

The founders are arriving in the Bay area, which means we will have the pleasure of seeing another league of extremely talented entrepreneurs with their disrupting product in March. Let's keep an eye on it 👀

The elephant in the room: is the market changing? 🐘

Given the current market conditions, which are expected to remain challenging for the next couple of years, what are the implications for venture capital limited partners? Will they demand more favorable terms from their venture managers, similar to how venture managers are currently seeking better terms from startup founders?

Limited partners, who are the investors in venture capital funds, are likely to avoid creating any significant changes in their agreements with the top-performing venture funds they have invested in, as they have seen steady returns from them. Similarly, they are not likely to pressure underperforming or newer venture managers as less money is available in the market.

Trending in tech 🔥

Orchestration for Data, ML and Infrastructure

In the last 10 years, new tools have been developed to make it easier for different tools to work together when managing data pipelines. Some examples of these tools are Apache Airflow, Luigi, and Oozie. Some people still use the simple tools, but there are newer and more advanced options like Prefect, Dagster, and Flyte which are better in terms of user experience, can handle larger workloads, are more flexible and keep up with the current challenges in the field.

Data orchestration is the process of gathering, cleaning, organizing, and analyzing data to make informed decisions. It can be done manually or with the help of tools that automate much of the process. Data orchestration can involve several steps, including data integration, quality improvement, enrichment, and cleansing. These steps help prepare data for further consumption down the pipeline. Popular tools for data orchestration include Airbyte, Great Expectations, Pandera, Pydantic, Flyte, Prefect, and Dagster. Machine learning orchestration (MLO) is similar to data orchestration but is specifically designed to handle large-scale and dynamic nature of ML development. ML orchestrators also include model registry, extensive scalability, dynamism, model observability, and more. The ML development cycle includes creating features, setting up the training process, evaluating the model, saving the best model, and monitoring the model. Tools such as Tecton, Feast, and Hopsworks are used for feature engineering.

What do you need to know about this space?

There are tools that claim to orchestrate both data and ML, and choosing among data and ML orchestrators can be tricky.
Scalability is a key consideration when choosing an orchestrator, and teams building ML pipelines often benefit from automating resource allocation and management.
The skillset of the team should be considered when choosing an orchestrator, as some tools may be more complex to use than others.
The learning curve and how often pipelines change should be considered when choosing an orchestrator.
There may be specific integrations that the team needs and whether using an ML orchestrator would be overkill should be considered.

Cash flow 🤑

Oxford Ionics secures £30 million Series A investment led by Oxford Science Enterprises and Braavos Investment Advisers
Inflow, the science-based app that helps members better manage ADHD through Cognitive Behavioral Therapy (CBT) based support, has raised a $11M Series A round led by Octopus Ventures
London-based legal tech startup Apperio has today secured $7m (£5.8m) in funding as it seeks to expand its business in both the UK and the US
Hack The Box has landed £45.2m in Series B funding for its gamified cybersecurity training and upskilling platform
Oxford-based Oxbotica is on a mission to use tech to make the earth move. The startup has just raised a fresh €130 million funding boost as investor confidence in market demand grows

What do people on Twitter think 💬

ChatGPT’s current killer app isn’t search, therapy, doing math, controlling browsers, emulating a virtual machine, or any of that other cherrypicked examples that come with huge disclaimers.
It’s a lot more quotidian:
Reformatting information from any format X to any format Y.
— swyx 🤖 (@swyx)
Jan 3, 2023