← All pins

Pins tagged “databases”

DuckDB Internals: Why is DuckDB Fast?

A walk through why DuckDB, an in process analytical database, runs SQL so fast. It follows a query from parsing to execution and shows how each step avoids extra work: running as a library instead of a server cuts out network and serialization overhead, the optimizer pushes filters down and picks join order with dynamic programming, and columnar storage with zone maps lets whole chunks of data be skipped when they cannot match a filter. It also explains how a query is split into pipelines so the work can run in parallel across threads, each with its own local state. The result is a full picture of the system rather than a list of buzzwords.

There is also a second part from the series which covers how those plans actually run: vectorized execution in batches of 2048 rows, selection vectors that filter without copying, and a push based model that spreads work across CPU cores.

databasesduckdbperformance

Offloading I/O to Dedicated Cores: An Asymmetric io_uring Backend for Seastar and ScyllaDB

A ScyllaDB engineering post about a new io_uring backend for Seastar that breaks the usual shared nothing rule, where every core does its own I/O and compute. The asymmetric backend instead sets aside a few cores as dedicated networking workers while the rest run only application logic, and routes I/O syscalls to those workers through io_uring queues. To make it work the team had to remove a speculative fast path that let a shard skip io_uring and issue a plain syscall on its own core, since that shortcut defeats the point of offloading. The numbers are honest about the tradeoff: raw I/O throughput trails the older linux aio backend, but compute shards get back the CPU time they used to spend on sockets and disk calls.

networkingio_uringdatabases

Aurora DSQL: Scalable, Multi-Region OLTP

A paper from Amazon engineers on how Aurora DSQL works inside. DSQL is a serverless SQL database that runs active active across regions, so any region can take reads and writes at once. The design splits the two apart: reads use multiversion concurrency control with precise timestamps and never coordinate with other nodes, while writes use optimistic concurrency control and coordinate only at commit time, through components called adjudicators and a replication layer called the Journal. Query processors run in small Firecracker microVMs and keep no local state, which lets compute, storage, and coordination scale on their own, from idle up to millions of transactions per second. The paper shows how all of this keeps full ACID transactions even when an availability zone or a whole region fails.

databasesdistributed-systemspapers

Introducing Meerkat: an experiment in global consensus

A Cloudflare post about Meerkat, a consensus system they built to keep control plane state consistent across their 330 plus data centers. It runs on QuePaxa, an algorithm that needs no leader. In Raft a dead leader or a slow network stalls writes until a new one is elected, and the timeouts are hard to tune across the wide area internet. QuePaxa instead lets any replica propose a write at any time, and concurrent proposals help each other reach agreement rather than block each other. The post is honest about the cost, since each write still takes one to three round trips, so Meerkat suits data that changes rarely and must stay correct rather than a busy database. It is also the first time QuePaxa has run at production scale.

distributed-systemsconsensusdatabases