Practice 03 / Data
Data and streaming.
We build the pipelines that carry your events from the moment they happen to the moment someone acts on them. Real-time ingestion, stream processing, change data capture, and analytics stores that answer questions in milliseconds, not overnight.
What we build
Four parts of one streaming path.
Each block below is a piece we build and operate. Most engagements use two or three of them together, wired into one pipeline with a single view of lag, throughput, and freshness.
Streaming ingestion
Durable, ordered ingestion into Kafka with schema validation at the edge. We size partitions to your key cardinality, set retention for replay, and handle the awkward parts: idempotent producers, poison messages, and clients that reconnect in a thundering herd after an outage.
- Backpressure by designProducers slow before brokers saturate, so nothing is dropped silently.
- Schema registryVersioned contracts with compatibility checks in CI, not at 3am.
Stream processing and enrichment
Stateful processing with Flink or Kafka Streams: windowed aggregations, joins against reference data, deduplication, and enrichment from external stores. We handle late and out-of-order events explicitly with watermarks, and we checkpoint state so a restart resumes exactly where it left off.
- Effectively-onceCheckpointed state plus transactional sinks, so retries do not double-count.
- Event-time correctWatermarks and allowed lateness, not wall-clock guesswork.
Change data capture and movement
Change data capture with Debezium turns your existing databases into event streams without touching application code. We read the write-ahead log, publish row-level changes with ordering guarantees, and keep downstream stores in step with the source, including the initial snapshot and the switch to live tailing.
- No dual writesThe database log is the source of truth, so services stay consistent.
- Snapshot and streamBackfill from a snapshot, then cut over to live change events with no gap.
Analytics stores and query
A columnar store (usually ClickHouse) fed straight from the stream, so dashboards and ad hoc queries hit fresh data. For history and cross-team access we land the same events in a lakehouse on Iceberg. One pipeline, two shapes: fast operational reads and cheap analytical scans.
- Sub-second readsMaterialized views and sensible partitioning keep dashboards live.
- Lakehouse for depthIceberg tables for retention, replay, and open access.
How a pipeline comes together
Three stages, one measurable path.
Model the events and the budget
We start from the event, not the tool. What is the key, the schema, the volume, and the tolerable staleness? Out of that we set the delivery guarantee and the end-to-end latency target that the rest of the design has to hold.
Build a thin slice to production
One source, one processor, one sink, running under real traffic in the first weeks. We load test it to the peak you expect, watch consumer lag under stress, and tune partitioning and state before we widen the pipeline.
Instrument, then hand over or run it
Lag, throughput, freshness, and error rate on one dashboard, with alerts tied to the SLO. You get runbooks for the failure modes we found, and we either operate the pipeline for you or hand it to your team cleanly.
The choices that matter
Delivery guarantees, stated plainly.
Every streaming system trades latency, cost, and correctness. We make the trade with you on purpose, and we write down which one each pipeline uses.
| Guarantee | What it means | Typical use | Cost |
|---|---|---|---|
| At most once | Fast, may drop under failure. No duplicates, no retries. | High-volume metrics where a gap is tolerable | Lowest |
| At least once | Nothing lost, some events may repeat on retry. | Downstream sinks that dedupe on a key | Low |
| Effectively once | Checkpointed state plus transactional sinks, no double-counting. | Ledgers, billing, operational aggregates | Higher |
| Ordered per key | Strict order within a partition key, parallel across keys. | Change data capture, per-entity state machines | Moderate |
Questions
What technical buyers ask us
No, but it is usually the right default. Kafka gives you a durable, replayable log, a mature connector ecosystem, and operational tooling that most teams already understand. Where a managed alternative fits your cloud and your scale better, we will say so. The design principles (ordered per key, replayable, backpressure-aware) matter more than the specific broker.
We are honest that true exactly-once across arbitrary systems is a marketing term. What we deliver is effectively-once: checkpointed processor state combined with idempotent or transactional sinks, so a retry after failure does not double-count. The overhead is real but bounded, usually a few milliseconds per checkpoint interval, and we only pay it where correctness demands it, such as ledgers and billing.
Yes, that is what change data capture is for. We read the write-ahead log with Debezium, so your application code and query paths stay untouched. Row-level changes are published in order, downstream stores stay in step with the source, and there are no dual writes to keep consistent. We handle the initial snapshot and the switch to live tailing as one clean operation.
Consumer lag is the first metric we alert on. Because the log is retained, a slow or restarted consumer catches up by replaying from its last committed offset rather than losing data. We size retention for the worst recovery you are willing to fund, tune partitioning so consumers scale out, and set backpressure so upstream producers slow down before anything is dropped. You see lag trending on the dashboard long before it becomes an incident.
For operational analytics fed from a stream, ClickHouse reads fresh data in well under a second and scans large volumes cheaply, which is what live dashboards need. A batch warehouse is a poor fit for sub-second freshness. When you also need long history, open formats, and cross-team access, we land the same events in an Iceberg lakehouse alongside ClickHouse, so you get fast operational reads and cheap analytical scans from one pipeline.
If you want us to, yes. We offer managed operations with defined SLOs on freshness and lag, 24/7 on-call, and the runbooks we wrote during the build. If you would rather own it in-house, we hand over documented architecture, load-test results, and a clean operating guide so your team can carry the pager with confidence.
Start
Have data arriving faster than you can use it?
Tell us the volume, the latency you need, and where the current pipeline hurts. We will tell you honestly what is achievable and what it takes to get there.