Can your data pipeline last longer than the next PM?
Kate Reznykova
October 21, 2022
local news 🇬🇧
📛 Coalesce in London by dbt Labs
In case you missed it, dbt Labs hosted Coalesce 2022 from October 17 - 21. It is the annual conference dedicated to advancing the practice of analytics engineering.
🔥Hot takes:
Data Mesh is getting some well deserved interest
Data contracts could become a key piece of the data quality puzzle
Semantic Layer: The BI Trend You Don’t Want to Miss
"lineage experiences are not designed to find things; they are designed to show things"
trending 👀
🤔 Observability for Data Engineering
Many data pipelines are difficult to monitor and troubleshoot. Observability is a growing concept in the Ops community that has gained much traction recently. Major monitoring/logging companies like Datadog, Splunk, New Relic, and Sumo Logic have helped to promote it, and thought leaders like Datadog's Ben Sussman and Splunk's Jeff Sutherland are leading the way. Observability allows engineers to monitor a system's internal state and context to ensure it runs as expected.
Shall we just apply these tools to data engineering? NOPE.
Why Don’t Existing Tools Cut It?
DAGs typically exhibit very different behaviour from other infrastructure services. With standard alerting, data teams receive many meaningless alerts, with dozens of unread notification emails, and they cannot find the important information.
Long-running processes can produce errors that can take a long time to surface. In order to detect success or failure, you need to wait and watch for a while. This increases the cost of failure because it takes more time to restart jobs if an issue comes up downstream. A team needs a way to gather early warning signs and a way to anticipate failures.
Finding frameworks that are easy to integrate with the modern stack of tools can be difficult. Airflow, Snowflake, Kube, Spark, dbt, etc.) are commonly used and provide the required level of extensibility
Cost attribution can be more difficult when it comes to pipeline monitoring because teams need to look at processes on many different levels
cash flow 🤑
London-headquartered OutThink raises €10 million to rethink cybersecurity
Fintech startup Payable has just secured about €6 million to reimagine payment processes for the modern internet economy
Stability AI, the company behind open-source artificial intelligence models for generating images, audio and video, has secured $101m (£89m) in a funding round that values it at around $1bn
British space startup Orbex has landed £40.4m in a Series C funding round led by the Scottish National Investment Bank to continue working towards vertical launch capability
twitter spotlight 🔦
Scoop: Pollen - the company that raised $200M+ - is looking to be sold... for $250K.
This won't even cover the amount the company owes to Monday .com, which is at £550K ($619K).
Let's not even talk about the other debts.
— Gergely Orosz (@GergelyOrosz)
Oct 20, 2022
hot startup jobs🔥
behavox - software-engineer
productboard - head-of-design
rippling - head-of-growth
picsart - principal-designer
two. - data-engineer