The Rise of the Serverless Data Architectures

For a while, it looked like Serverless was just a convenient way to run stateless functions in the cloud. But in the last year we’ve seen the rapid rise in serverless data stores. This includes data stores that are designed for serverless workloads, ones that abstract the underlying infrastructure and scale based on usage.

In this talk we’ll explore the implications of serverless workloads on the design of data stores. Historically, data stores were designed for predictability - providing consistent performance for stable workloads over long-lived low-latency connections. Serverless workloads still require predictable performance, but workload variability is expected, connections are short lived and have wide geographical distribution. As a result, data architectures evolved toward more flexible scalability in order to be cost-efficient to operate with highly variable workloads.

As examples of the changes in workloads and architectures, I’ll share lessons learned while leading the cloud native Kafka team at Confluent, and while building the data architecture at Nile, a still stealthy startup. I’ll also share insights from recent research papers that I believe will shape the data architecture space in the coming years. Of course, Serverless data architecture may or may not be a good fit for your use-case. As we explore these architectures in depth, you’ll acquire the tools you need to decide when to use them.


Speaker

Gwen Shapira

Founder @Nile, PMC Member @Kafka

Gwen is a co-founder and CPO of Nile (thenile.dev). She has 20+ years of experience working with engineering teams and customers to build reliable and scalable data architectures - most recently as the head of Cloud Native Kafka engineering org at Confluent. Gwen is a committer to Apache Kafka, author of “Kafka - the Definitive Guide” and "Hadoop Application Architectures". You can find her speaking at tech conferences or talking data at the SaaS Developer Community.

Read more
Find Gwen Shapira at:

Date

Tuesday Jun 13 / 01:40PM EDT ( 50 minutes )

Location

Salon D

Topics

Serverless Database Architecture Edge Distributed Systems

Share

From the same track

Session Streaming

Laying the Foundations for a Kappa Architecture - The Yellow Brick Road

Tuesday Jun 13 / 10:35AM EDT

In the ever changing landscape of big data, focus is slowly moving away from batch and towards realtime analytics. Data Science workflows are evolving to adapt to this changing landscape.

Speaker image - Sherin Thomas

Sherin Thomas

Staff Software Engineer @Chime

Session Stream Processing

Streaming from Apache Iceberg - Building Low-Latency and Cost-Effective Data Pipelines

Tuesday Jun 13 / 11:50AM EDT

Apache Flink is a very popular stream processing engine featuring sophisticated state management, even-time semantics, exactly-once state consistency. For low latency processing, Flink jobs typically consume data from streaming sources like Apache Kafka.

Speaker image - Steven Wu

Steven Wu

Software Engineer @Apple and Apache Iceberg PMC

Session Data Architecture

Building a Large Scale Real-Time Ad Events Processing System

Tuesday Jun 13 / 02:55PM EDT

Two years ago, we embarked on building DoorDash's ad platform from the ground up. Today, our platform handles over 2 trillion events every day and our advertising business has experienced significant growth in recent years, becoming a key area of focus for the company.

Speaker image - Chao Chu

Chao Chu

Software Engineer @DoorDash

Session Architecture

Enabling Remote Query Execution Through DuckDB Extensions

Tuesday Jun 13 / 04:10PM EDT

DuckDB is a high-performance, embeddable analytical database system that has gained massive popularity in the last few years.

Speaker image - Stephanie Wang

Stephanie Wang

Founding Engineer @MotherDuck

Session

Unconference: Modern Data Architecture & Engineering

Tuesday Jun 13 / 05:25PM EDT

What is an unconference? An unconference is a participant-driven meeting. Attendees come together, bringing their challenges and relying on the experience and know-how of their peers for solutions.

Speaker image - Ben Linders

Ben Linders

Independent Consultant in Agile, Lean, Quality and Continuous Improvement