Turning robot video into training-ready data with daft-physical-ai
A Python library built on Daft for turning robot video into training-ready data, starting with hand tracking and reward scoring as UDFs, with more to come.
Cutting dead frames from DROID: 500 hours of robot data in 32 seconds
The robot state recorded next to the video tells you which frames matter before you decode any pixels. Trimming all of DROID takes 32 seconds on a laptop.
What if more data alone won't solve robotics? Ken Goldberg on why good old-fashioned engineering still matters
Ken Goldberg argues that robotics needs more than data - modularity, algorithms, and metrics still matter. A breakdown of his MIT talk on the 100,000 year data gap.
How we made our LeRobot video reader up to 15× faster
LeRobot has emerged as the dominant open format for robot learning data, but decoding frames is expensive. Here's how we made Daft's native LeRobot reader up to 15× faster.
Finding a Needle in the Haystack: Querying Physical AI Data with Daft
Pose + semantic search over Apple's EgoDex hand-manipulation dataset with Daft: SigLIP embeddings meet hand-pose geometry. Ctrl+F for physical AI data.
Disk is the data plane: Flight Shuffle in Daft
How Daft rebuilt distributed shuffle around Arrow Flight, local disk, and streaming reads to handle multi-terabyte workloads.
The limiting factor in physical AI isn't compute or architecture - it's data
Robotics is hitting a data wall. The architecture debate gets much of the attention, but the data constraint is more fundamental.
First-class observability in Daft
A new dashboard, per-operator memory attribution, and OTel endpoints for your existing collector. Everything you need to see what Daft is doing with your query.
Scaling As-of Joins
How we built, broke, and re-built our ASOF joins — 5.5x faster, half the memory of pandas, and scaled to a distributed cluster.
daft.VideoFile: Seek Lazily, Get Frames
daft.VideoFile decodes only the frames you need. Keyframes, time-sampled, or windowed seek, built for robotics datasets, dashcams, and moderation queues.
VLAs are dead, long live World Action Models - a summary of Jim Fan's Robotics End Game talk
Jim Fan argues robotics will follow the exact LLM playbook - and VLAs are already being replaced by World Action Models.
What is physical AI - is it more than just hype?
Physical AI has become a real trend, but is there something real here or is it just hype?
Daft Extensions Featuring daft-h3: Native Rust Performance, Community Owned
Daft now supports native extensions via Apache Arrow's C Data Interface. daft-h3 is the first community extension — 9 Rust-native H3 geospatial functions, 3–16x faster than Python UDFs.
Audio transcription at scale with daft.AudioFile
How to transcribe thousands of audio files with Whisper using daft.AudioFile — handling resampling, silence splitting, and worker-resident model loading without the boilerplate.
Image Embeddings: Tutorial & Examples
Learn about the concept of image embeddings, their various use cases, and best practices for handling them in data processing workflows.
daft.File: Lazy Metadata Filters
Filter millions of files by path, size, and content type before opening any of them. Cheap operations first, expensive operations on the survivors.
Multimodal Embeddings: Tutorial & Examples
Learn multimodal embedding techniques for cross-modal search, recommendation systems, and content moderation applications.
Open-sourcing 43 Billion Tokens of SEC EDGAR
Datamule, Teraflop AI, and Eventual collaborated to release the SEC-EDGAR dataset containing 590 GB of data, spanning 8 million samples and 43 billion tokens from all major filings in the SEC EDGAR database.
Daft UDF Patterns: Four Patterns, One Notebook
Row-wise, generator, async, and stateful UDFs — one notebook, one dataset, runnable side by side.
GPU Inference with @daft.cls
Run GPU models on millions of rows without OOM. Real patterns from ByteDance, Essential AI, and more.
Stateful UDFs with daft.cls: Python Classes that Scale
Turn any Python class into a distributed operator. Hold models, connections, and clients across rows with one decorator.
Stateless UDFs with daft.func - four patterns, one decorator
Row-wise, async, generator, and batch UDFs in Daft — one decorator, zero boilerplate, local or distributed.
Daft UDFs: What is a UDF and why do you need one?
Daft User Defined Functions (UDFs) let you run custom Python inside a distributed DataFrame pipeline. Leverage Row-wise, Async, Generators, and Batch.
How We're Making Observability Better in Daft
Daft Observability Roadmap: metrics, OTEL integration, real-time dashboards, and DataFrame APIs for debugging and monitoring distributed pipelines.
Introducing daft.File: Work with Any File, Anywhere
daft.File brings lazy, distributed handling for audio, video, PDFs, and code to Daft DataFrames. One interface, local or remote.
Daft OSS: New Governance Model
Today, we're introducing updates to the Daft OSS governance model defining new roles for contributors and maintainers with expanded permissions.
Tuning Daft's Distributed UDFs: Lessons from ByteDance
Learn from the ByteDance Volcengine LAS Team on how to optimize Daft UDFs on Ray. Discover the formula to evenly distribute data across actors.
Announcing Early Access to Daft Cloud
Early access to Daft Cloud for running model-driven AI pipelines reliably at production scale. Built on Daft OSS for continuous, resilient execution.
Why I joined Eventual
Chris Kelloggs shares why he joined Eventual to build open-source, distributed systems for large-scale AI and multimodal data workloads
Introducing Dynamic Batching: Auto-Tuning for Daft Pipelines
Manually tuning batch sizes is hard. So I implemented dynamic batching to never deal with it ever again.
Daft 2025 Year in Review - Minor Releases, Major Evolution
In 2025, we shipped 56 releases and introduced features that changed how teams run multimodal AI pipelines at scale.
Knowledge curation (not search) is the AI big data problem
Google was Information Retrieval. Wikipedia is Knowledge Curation.
How We Use AI Coding Agents
Our engineering team's best practices for working with AI coding agents.
How Sourcetable Built the World's First AI Spreadsheet with Daft
Sourcetable CTO Andy Grosser discusses their data infrastructure choices and why reliability and scale drove their architecture decisions.
Why I Joined Eventual
Sam Stokes shares why he joined Eventual, the company behind Daft, and what excites him about helping build our large scale data processing platform.
Multimodal Structured Outputs: Evaluating VLM Image Understanding at Scale
Leveraging ablation for contrastive image understanding evaluation in Daft
Processing 99% of U.S. Caselaw for Under $1 in the Common Pile
How Teraflop AI processed 7 million court documents and 40 million pages spanning 365 years of U.S. caselaw for under a dollar using Daft.
Prompting with DataFrames: Massively Parallel LLM Generation is Here
Discover how Daft's prompt function revolutionizes LLM workflows with massively parallel context engineering on DataFrames.
Agentic systems are just query engines for unstructured data
Explores how agentic AI systems act as declarative query engines, revealing how reasoning and orchestration transform unstructured data.
Fall 2025 Review: OSS Updates | UDFs, Functions, & daft.File
Daft Fall 2025: AI Functions, improved UDFs, faster vLLM inference, and new daft.File VideoFile subtype - plus Bigtable sink and Common Crawl loader.
Cutting LLM Batch Inference Time in Half: Dynamic Prefix Bucketing at Scale
Learn how Dynamic Prefix Bucketing reduces LLM batch inference time, improves throughput, and unlocks faster multimodal processing at scale.
Simplifying Voice AI Analytics with Daft: Transcription, Summaries, and Embeddings at Scale
Build a Voice AI analytics pipeline with Daft and Faster-Whisper to convert raw audio into searchable transcripts, summaries, and embeddings at scale.
Using PyTorch DataLoaders to Streamline Multimodal Data
Learn how PyTorch's DataLoader streamlines deep learning pipelines by efficiently loading and shuffling data in batches.
Benchmarks for Multimodal AI: Spark, Ray Data, and Daft
Multimodal AI workloads break traditional data engines. Daft ran 2-7x faster than Ray Data and 4-18x faster than Spark while finishing jobs reliably across audio, video, document, and image workloads.
Introducing Flotilla: Simplifying Multimodal Data Processing at Scale
Flotilla, Daft's new distributed engine, processes terabytes of multimodal data in a single query up to 18x faster than Spark and Ray Data, while running efficiently, reliably, and without manual tuning.
Exploring Daft's Local Execution: The Swordfish Engine
Explore how Daft's Rust-powered engine executes DataFrame and SQL queries. Learn how Swordfish enables fast, streaming image processing at scale.
After the First Run
Using Daft's observability tools to uncover performance pitfalls
Making GPUs Zoom (Part 1)
How Daft is approaching large-scale model inference with advanced GPU optimizations for faster multimodal AI workloads
End-to-End Distributed PDF Processing Pipeline
Build production-ready PDF processing pipelines with distributed computing, OCR, spatial analysis, and GPU embeddings
How to Build Scalable, End-to-end Batch Inference Pipelines with Daft
Daft makes it easy to express these pipelines end-to-end, while seamlessly scaling them up to handle massive workloads.
24 Trillion Tokens, 0 Crashes: How Essential AI Built Essential-Web v1.0 with Daft
Essential AI leveraged Daft's data engine to process a massive web-scale dataset for large language model (LLM) training.
Embedding Millions of Text Documents With Qwen3
Learn how to achieve near-100% GPU utilization processing millions of text documents with Qwen3 embeddings.
Processing 300K Images Without OOM
A Streaming Solution
Multimodal Data Processing Goes Global
Daft Community is expanding to China, bridging the gap between English documentation and Chinese innovation cycles, in partnership with Bytedance Team
Eventual Raises $30M to Build the Future of Data Processing
We've raised $30M to build generational technology for simple, reliable, and performant data processing across all modalities and regardless of scale.
We cloned over 15,000 repos to find the best developers
An adventure in AI and data engineering to analyze developers across Github
High-Performance File System Support With DeepSeek 3FS
Learn how Daft integrates with DeepSeek SmallPond 3FS to deliver faster file access and efficient data handling for modern workloads.
From v0.2 to v0.3: Harder, Better, Faster, Stronger
Join us on the journey from Daft v0.2 to v0.3! Daft v0.3 was released last month, marking the first minor version increment in almost 10 months.
Introducing Daft-SQL for High-Performance Data Exploration
A SQL API enabling users to interact with their data in a new but familiar way. Learn how Daft-SQL brings fast, scalable querying to multimodal workloads, helping teams explore large datasets efficiently with a distributed engine.
Reading Delta Lake with Daft
Discover how Daft reads Delta Lake tables efficiently, giving teams fast access to large datasets and seamless integration into data workflows.
Adversarial file reading: from 10,000 small CSVs to massive Parquet files
Learn how adversarial file reading speeds up data ingestion at scale, enabling fast conversion from thousands of CSVs into efficient Parquet files.
Announcing Daft 0.2: 10x faster IO from S3
Daft 0.2 introduces a redesigned IO layer with 10x faster S3 reads and noticeable speedups across data loading, compute, and end-to-end workflows.
Working with the Apache Parquet file format
This guide shows how Apache Parquet boosts read performance, lowers storage use, and supports efficient workflows for large analytical datasets.
Introducing Daft: A High-Performance Distributed Dataframe Library for Multimodal Data
Discover how Daft accelerates multimodal data processing with a high-performance distributed dataframe engine built for modern AI and analytics tasks.
Newsletter
Get new posts in your inbox
Engineering, product, and company updates from Eventual.