Airflow walked, so literally anything could run
Kate Reznykova
September 02, 2022
local news 📰
💣 As part of the layoffs, Snap is shutting down Zenly, a location-based social app with over 40 million monthly active users
Snap bought Zenly for $213M in 2017 and left it to operate largely autonomously, during which time it grew its user base 30X. All Zenly engineering team is based in France. The team is truly exceptional from a technical and human POV with some of the longest tenure I've seen in French startups. All to be laid off.Snap has cancelled more projects in this layoff (acquired apps like Zenly & Voisy, the Pixy drone camera, web3 team, mini-games, original shows, etc.) than any other company.
Zenly's popularity may have actually hindered its success in this instance.
trending 📈
We don’t need a better Airflow, but we need a higher-level one: a system that enables data platform teams to think at a platform level.
Even though it's slow and clunky, the centralized DAG is still good enough for most teams in 2022. It expresses most existing data value, even if it's unbundled into other SaaS tools.
Nowadays, the platform teams face correlational problems as well:
From 10:00am to 10:30am, AWS us-east-1 had an outage. Snowflake was inaccessible during this time, affecting your data replication, observability, transformation, reverse ETL, and dashboard tooling. How do you identify and restart the necessary processes?
It is difficult to understand the full impact of new data assets, bad data assets, or system outages across a large number of processes. This work is similar to that of site reliability engineers. The more users, tools, and use cases you add to the platform, the more acute the pain becomes, scaling with the value.The need for new frameworks, protocols and processes to ensure data is consumable is evident. Personal data must be treated appropriately, toxins from upstream must be prevented from leaching into metrics, and subtle drifts in balance must not ruin machine learning models.
If it sounds like you could simply replace Airflow with basically any other job execution engine.
Another section of the stack that has to be simplified and suited to the demands of data teams involves an understanding of Terraform, AWS, Kubernetes, CI/CD, networking, and security standards. The tool that data engineers use to be productive in this new environment does not run scripts; rather, it organises systems.
cash flow 🤑
London-based Thalamos secures €1 million to digitise the Mental Health Act
UK-based Neurofenix raises €7 million for its digital platform that takes a patient-centric approach to rehabilitation
Spinning laundry sustainably Oxwash picks up €11.7 million and earns B Corp certification
Co-living operator Gravity Co has just raised over €5.7 million to support the international expansion of its co-living and PRS brand
twitter spotlight 🔦
🔥 Killer feature!
*Major* new feature update in Attio 💪
Users are now be able to send highly personalized emails in bulk, leveraging dozens of enriched data points as variables, as well as any custom attribute or data points from integrations.
Best of all, the experience is *delightful*!
— Attio (@attio)
Sep 1, 2022
hot startup jobs🔥
data.ai - backend-engineer
picsart - principal-product-designer
veed.io - head-of-marketing
notion - product-manager
cogsy - full-stack-engineer