daft.datasets.lerobot reads LeRobot v3 datasets — episode metadata, frame-level sensor data, and camera video — as a lazy DataFrame with joins and video-frame decoding handled for you
daft.Hdf5File gives you typed accessors (hdf5_keys, hdf5_metadata, hdf5_attrs) for inspecting HDF5 files without hand-rolling h5py traversal code
prompt(..., provider="transformers") runs open-weight instruction-tuned models locally through Hugging Face pipeline("text-generation") — no API key, no separate service
daft.datasets.common_crawl can now read from Hugging Face buckets, a credential-free alternative to the AWS S3 mirror for anyone outside us-east-1
daft.datasets.droid gives you a DataFrame API over the DROID robotics dataset — 76k demonstration episodes, camera feeds, and natural language annotations
DataFrame.to_torch_dataloader connects Daft directly to PyTorch training loops without an intermediate dataset abstraction
ignore_corrupt_files skips bad files instead of aborting your entire job, with structured observability so nothing gets silently dropped
Parquet reader rewritten on the arrow-rs array_reader API with coalesced range GETs and two-phase predicate pushdown — up to 17.4x faster remote reads, 3.82x remote aggregate, 1.31x local aggregate
Streaming distributed limits replace materialize-then-truncate with a LimitCounterActor that slices morsels in place, so top-N over large shuffles no longer OOMs materializing the full shuffle output
uuid(version="v7") generates UUIDv7 values with a timestamp prefix, giving chronologically ordered IDs for event tracking and time-series data
json_array_length, json_object_keys, and json_tuple add Spark SQL-compatible JSON inspection for nested payloads
Streaming ASOF joins (v0.7.11) process the right side in parallel batches so it no longer has to fit in worker memory — parallelism is batch count, not entity cardinality
Forward ASOF joins (v0.7.13) add strategy="forward" to match each left row against the earliest later right row, completing bidirectional matching
Arrow PyCapsule Interface (v0.7.11) gives zero-copy DataFrame exchange with PyArrow, Polars, and cuDF via the C Data Interface standard
Iceberg gets idempotent commits through daft.IdempotentCommit snapshot markers (v0.7.11) and now honors table-level write.target-file-size-bytes and write.parquet.row-group-size-bytes (v0.7.12)
Distributed asof joins scale join_asof horizontally using range-repartitioning plus a carryover system for correctness across partition boundaries — linear scaling to 100+ partitions with no accuracy loss
simhash() and hamming_distance() bring near-duplicate document detection to Daft, 1000x faster than embedding-based approaches for deduplication
Native UUID type gains the missing serde feature, so UUID columns round-trip through Arrow IPC as 128-bit binaries instead of falling back to string serialization
Eight new temporal and math functions with Spark-compatible semantics — make_date, make_timestamp, make_timestamp_ltz, last_day, next_day, factorial, hypot, e, pi
Eight new temporal functions — date_add, date_sub, date_diff, date_from_unix_date, timestamp_seconds, timestamp_millis, timestamp_micros, from_unixtime — follow Spark naming for ETL migrations
video_frames() decodes video already loaded in a File column without re-reading from storage, emitting per-frame structs with frame_index, frame_time, is_key_frame, and image bytes
UUID is now a built-in DataType.Uuid backed by FixedSizeBinary(16), so PostgreSQL and Trino UUID columns read natively instead of needing a CAST(id AS VARCHAR) workaround
Dashboard adds per-operator bytes_in/bytes_out counters with volume-scaled arrow widths and expansion/reduction tags on operator edges
Query dashboard adds operator heatmap coloring and directional arrows — slow operators glow red, completed nodes turn green, so the hot path is visible while a query runs
Grouped aggregation rewritten as a two-phase vectorized path over dense group_ids arrays — 10% off total ClickBench time and a 4x improvement on connected components workloads (250s → 70s)
image_hash() ships eight perceptual hashing algorithms implemented in Rust, bit-exact with the Python imagehash library and 5-25x faster than an equivalent Python UDF
df.skip_existing() filters rows already present in the output path for idempotent ETL and RAG re-runs on the distributed Ray runner
The v0.7.5 parquet streaming regression that made aggregations 2-4x slower is fixed — 90 SUMs over 100M rows drops from 1.226s back to 0.652s by sending row-group batches downstream without concatenating them
df.shuffle(seed=...) randomly rearranges rows as a first-class plan node for ML data prep, and a standalone random_int() expression ships with it
coalesce now short-circuits per the SQL spec, so expensive expressions in later positions stop evaluating once an earlier value is non-null
concat_ws concatenates columns with a separator and skips nulls instead of propagating them, unlike concat
daft.read_paimon and df.write_paimon complete native support for all four major lake formats — Iceberg, Delta Lake, Hudi, and Apache Paimon — behind one DataFrame API
Scalar columns are now O(1) — a Column::Scalar variant holds a single literal plus a logical length instead of broadcasting into a full-length array, with lazy materialization keeping the public API unchanged
Swordfish adds fingerprint-based plan caching so repeated executions of the same logical plan reuse one pipeline instead of rebuilding it each time
daft.read_kafka reads Kafka topics as a bounded batch source with earliest/latest, timestamp, or per-partition offset bounds
daft.load_extension() plus a stable C ABI lets any language that compiles to a C-compatible shared library extend the query engine, with data crossing the boundary zero-copy via the Arrow C Data Interface
Late materialization in the arrow-rs Parquet reader makes filtered reads 2.1-2.6x faster at 1% selectivity and nested types up to 5.5x faster on maps, 4.2x on structs, 3.9x on lists
The dashboard becomes a real-time query debugger — collapsible plan trees, live rows in/out per operator, ticking wall-clock and CPU durations, and a Results tab for query output
DataFrame.write_sql() writes back to PostgreSQL, SQLite, or any SQLAlchemy target with parallel worker-side writes and an explicit dtype parameter for target schema control
The arrow2 to arrow-rs migration is effectively complete — 122 PRs and 38,850 lines changed, with interval arithmetic moving to arrow-rs as a breaking change
Apache OpenDAL support gives Daft read and write access to dozens of storage backends — S3, GCS, Azure Blob, HDFS — through one API instead of a dedicated connector per service
Flotilla gains Arrow Flight shuffle, moving data between nodes over gRPC and Arrow IPC to cut serialization overhead during distributed exchange
Metrics get consolidated names with node.type attributes, split duration columns in the metrics DataFrame, and a dashboard CLI split into start/stop subcommands
df.metrics now works for Flotilla runs — a distributed query returns a RecordBatch of operator timings, row counts, and bytes processed on the result DataFrame, same API as a local run
OpenTelemetry metrics export via the standard OTEL_EXPORTER_OTLP_* environment variables routes query metrics to Prometheus, an OTEL collector, or ClickStack without code changes
Nightly builds at nightly.daft.ai — pip install daft --pre --extra-index-url https://nightly.daft.ai gets you whatever landed on main yesterday
Lance namespace read/write, nearest-neighbor vector search, and distance/similarity functions land as native DataFrame expressions for RAG pipelines
