You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Fixed an issue where time-filtered queries would fail with "No files found" errors when the requested time range included partitions that don't exist in S3 storage. This particularly affected queries for recent data (< 24 hours) before daily compaction has run.
184
184
185
185
**Cause:** The partition pruner generated paths for all hours in a time range without verifying existence. For local storage, it used `filepath.Glob()` to filter paths, but for S3/Azure storage, paths were passed directly to DuckDB which threw errors for missing partitions.
186
186
187
+
Additionally, for day-level paths (`year/month/day/*.parquet`), the pruner only checked if the **directory** existed (which passes when hourly subdirectories exist), but didn't verify that actual `.parquet` files exist at the day level.
188
+
187
189
**Fix:** Extended `filterExistingPaths()` to handle S3/Azure storage:
188
190
- Uses `ListDirectories()` to verify which partition paths actually exist
191
+
- For day-level paths (5 segments), verifies that `.parquet` files exist directly at that level (not just in subdirectories)
189
192
- Filters out non-existent partitions before passing to DuckDB
190
193
- Also fixed a pre-existing bug where `filepath.Join()` was mangling S3 URLs (`s3://bucket` → `s3:/bucket`)
191
194
192
-
**Result:** Queries on sparse datasets (with gaps in time partitions) now succeed and return data from existing partitions instead of failing.
195
+
**Result:** Queries on sparse datasets (with gaps in time partitions) now succeed and return data from existing partitions instead of failing. Grafana dashboards querying recent data (< 24 hours) on S3 now work correctly.
196
+
197
+
*Day-level file verification contributed by [@khalid244](https://github.com/khalid244)*
193
198
194
199
### Server Timeout Config Values Ignored (Issue #126)
195
200
@@ -996,7 +1001,7 @@ None
996
1001
Thanks to the following contributors for this release:
997
1002
998
1003
-[@schotime](https://github.com/schotime) (Adam Schroder) - Data-time partitioning, compaction API triggers, UTC fixes, Azure SSL certificate fix
0 commit comments