What's Changed

Exciting New Features ✨

feat: add server to admin udfs api by @everpcpc in #17141
feat(sqlsmith): Support generate fuzz query sql by @b41sh in #17153
feat(ci): add ttc (tcp testing container) by @sundy-li in #17160
feat(query): support direct the spill data to a different bucket by @forsaken628 in #17159
feat(cluster): support system-managed cluster by @zhang2014 in #17051
feat(query): support send spill file stats to client by @sundy-li in #17186
feat(query): load ndjson support ts with diff units. by @youngsofun in #17203
feat(sqlsmith): fuzz query support sqllogic test cases by @b41sh in #17216
feat: hilbert clustering by @zhyass in #17045
feat: support vacuum aggregating index by @SkyFan2002 in #17231
feat(cluster): support warehouse level show processlist and kill query by @zhang2014 in #17249
feat: orc support fill missing tuple field by @youngsofun in #17247
feat(sqlsmith): fuzz test support read Sqlite and Duckdb test by @b41sh in #17301
feat(ddl): support alter table cast tuple type column by @b41sh in #17310
feat: support vacuum inverted index by @SkyFan2002 in #17291
feat: Add disable_list_batch for webhdfs by @Xuanwo in #17345
feat: Add username support for webhdfs by @Xuanwo in #17350
feat(ddl): Support add stored computed column by @b41sh in #17347
feat: introduce block meta cache by @dantengsky in #17360
feat(query): support desc procedure by @TCeason in #17364
feat(query): support Support from <table> to simplify select * from <table> by @TCeason in #17372
feat(query): Virtual column support alias name by @b41sh in #17365
feat(query): support timestamp diff return interval type by @TCeason in #17381
feat(query): support extract EPOCH from timestamp by @TCeason in #17385
feat(query): interval support agg func sum by @TCeason in #17379
feat(query): support DATE_TRUNC('WEEK', EXPR)->DATE by @TCeason in #17407
feat(auth): auto retry on jwks key not found by @flaneur2020 in #17410
feat(query): Allow extract parts from interval type by @TCeason in #17417
feat(query): array function support variant type as argument by @b41sh in #17428
feat(query): add bucket name to storage http request metrics by @everpcpc in #17431
feat: deprecate prefix_filter and change default level of logging. by @youngsofun in #17444

Thoughtful Bug Fix 🔧

fix(ci): flaky test by @zhyass in #17149
fix(query): return semantic error for SRFs in WHERE and HAVING clauses by @takaebato in #17167
fix(cluster): disable forward warehouse request in self-managed cluster by @zhang2014 in #17176
fix(query): check state is allocated or not when oom by @sundy-li in #17179
fix(cluster): fix unassign warehouse nodes failure by @zhang2014 in #17195
fix(cluster): skip the recovery for warehouses that are not running state. by @zhang2014 in #17193
fix(query): fix aggregating index conflict with min/max rewrite by @b41sh in #17182
fix: name resolution bug in quoted ident case sensitive mode by @zhyass in #17198
fix(query): record state_rows in aggregate payload by @sundy-li in #17194
fix: system.tables table visibility by @TCeason in #17202
fix(query): interval total microseconds add overflow check by @TCeason in #17215
fix(cluster): fix warehouse audit log not working with add cluster by @zhang2014 in #17219
fix(query): fix register function working with nullable scalar by @sundy-li in #17217
fix(query): stick the created_by infos in parquet writer by @sundy-li in #17220
fix: Upload to stage in streaming way instead by @Xuanwo in #17230
fix(cluster): fix heartbeat self managed after network fail by @zhang2014 in #17241
fix(query): fix array_agg failed if the argument is NULL by @b41sh in #17244
fix(query): increase state_rows in copy agg state rows by @sundy-li in #17252
fix(query): hash table scatter will always send agg meta by @sundy-li in #17245
fix(query): fix lazy columns missed in constant table scan by @Damon07 in #17258
fix: vacuum index not work if index is dropped by create or replace by @SkyFan2002 in #17263
fix(query): window function support params by @b41sh in #17282
fix: copy into table collect files twice some times. by @youngsofun in #17300
fix: set query_kind earlier to ensure it takes effect. by @youngsofun in #17302
fix: use full path as cache key for parquet meta. by @youngsofun in #17313
fix: column not found in analyze by @zhyass in #17321
fix(query): unsupport datetime format item should not return panic error by @TCeason in #17323
fix: support returning rows larger than 10MB. by @youngsofun in #17344
fix(query): fix histogram binder and checker by @sundy-li in #17342
fix(query): add check window frame by @sundy-li in #17316
fix(query): RawValueSource::parse_fallback fully parse input by @forsaken628 in #17340
fix(query): aviod building TransformSortMergeLimit with a big number limit by @forsaken628 in #17339
fix(query): disallow push down filters that use columns in window order by columns or function argument columns by @forsaken628 in #17353
fix(query): procedure support string type by @TCeason in #17325
fix(query): interval order by is incorrect by @forsaken628 in #17357
fix(query): cast string to timestamp overflow by @TCeason in #17376
fix(query): fix time parser with enable_strict_datetime_parser = 0 by @sundy-li in #17399
fix(query): fix lambda function bind column failed by @b41sh in #17402
fix(query): fix months_between overflow error by @TCeason in #17403
fix(query): interval type support cast to string by @TCeason in #17395
fix(query): Align TRIM functions semantically with Snowflake by @forsaken628 in #17375
fix(query): fix resolve decimal by @sundy-li in #17409
fix(ci): fix Link Checker by @sundy-li in #17416
fix(query): to_yyyymmddhhmmss only support timestamp type by @TCeason in #17422
fix(query): group checker subquery by @sundy-li in #17420
fix(query): support vacuum temp files for create and refresh by @zhang2014 in #17421
fix(query): if arg is Scalar::null or a null column ignore null will invalid by @TCeason in #17423
fix(query): cte recursive check by @sundy-li in #17427
fix(query): fix variant value can not cast to not null type by @b41sh in #17434
fix(doc): fix load tpcds tpch document by @sundy-li in #17438
fix(query): fix join subquery scalar rewrite by @sundy-li in #17441
fix: fix inconsistency comment about bitmap by @forsaken628 in #17452

Code Refactor 🎉

refactor: make fuse_amend implementation more convenient for on-premise deployments by @dantengsky in #17147
refactor: Bump OpenDAL to 0.51 by @Xuanwo in #17161
refactor: Optimize When Querying the tables_history Table with the Condition table_id = n by @TCeason in #17166
refactor: Use opendal to implement recover object instead by @Xuanwo in #17165
refactor: include key information in errors when listing meta-service key-values by @drmingdrmer in #17192
refactor: create/drop_warehouse return WarehouseInfo by @TCeason in #17254
refactor: only set global settings need to check super privilege by @TCeason in #17255
refactor(query): only call once finish callback by @sundy-li in #17293
refactor(query): record state_offsets to reduce the memory leak of agg by @sundy-li in #17285
refactor(query): introduce udf runtime pool by @forsaken628 in #17304
refactor(query): Multiple states Aggregate function by @forsaken628 in #17148
refactor: rename components of the replace into impl by @dantengsky in #17355
refactor: refine replace into by caching individual BlockMeta by @dantengsky in #17368
refactor(query): refactor variant cast to types by @sundy-li in #17435

Build/Testing/CI Infra Changes 🔌

ci: fix release type for nightly cron by @everpcpc in #17429
ci: fix missing return for stable release by @everpcpc in #17454
ci: fix publish binary to R2 by @everpcpc in #17456

Documentation 📔

docs:Update release-readme.txt by @wubx in #17221

Others 📒

chore: Split functions crate into multi crates by @TCeason in #17133
chore(query): bump ast 0.1.3 by @sundy-li in #17145
chore: fix cargo check by @TCeason in #17152
chore: adjust fuse_time_travel_size() by @SkyFan2002 in #17164
chore: more logging for vacuum drop table execution by @SkyFan2002 in #17168
chore: Remove unused settings for ssl by @Xuanwo in #17173
chore: check schema for small parquet file too. by @youngsofun in #17175
chore(ci): support backport release by @everpcpc in #17185
chore(ci): auto add pr-backport label by @everpcpc in #17190
chore: simplify setting API by @drmingdrmer in #17138
chore: fix flaky datetime test by @TCeason in #17188
chore: try split number cast function impl by @TCeason in #17155
chore: add udf compat ci by @TCeason in #17184
chore(ci): upload sccache log for build by @everpcpc in #17201
chore(planner): improve histogram cardinality estimation by @Dousir9 in #17200
chore: Fix warning for backtrace while compiling by @Xuanwo in #17204
chore(ci): fix sccache log path by @everpcpc in #17206
chore(ci): update sccache log artifact name by @everpcpc in #17211
chore(ci): adjust build & test workflows by @everpcpc in #17213
chore: bring back function set_cache_capacity by @dantengsky in #17196
chore: try fix flaky ci by @TCeason in #17222
chore(ci): fix release find last tag by @everpcpc in #17226
chore(ci): fix release bump log by @everpcpc in #17227
chore(ci): stable release ignore patch versions by @everpcpc in #17228
chore(ci): better generate release by @everpcpc in #17232
chore(ci): fix release sha for backport by @everpcpc in #17238
chore: fix flaky dictionary test by @b41sh in #17199
chore(ci): presign artifact for download by @everpcpc in #17253
chore(sqllogictests): Check container and stop first to avoid conflict running by @b41sh in #17261
chore: Bump opendal to pinned tag version by @Xuanwo in #17259
chore(ci): fix release bump version by @everpcpc in #17277
chore(ci): fix release patch version by @everpcpc in #17278
chore(ci): fix gen next patch release by @everpcpc in #17281
chore(ci): unit test with 16c64g by @everpcpc in #17283
chore: try fix flaky ci by @TCeason in #17265
chore(ci): add bendsql binary to release by @everpcpc in #17289
chore: databend-test print more stderr by @TCeason in #17288
chore: Bump opendal to include a fix for gcs by @Xuanwo in #17290
chore: make error message of conflicting transactions more user-friendly by @SkyFan2002 in #17307
chore: add enable_prune_cache setting by @dqhl76 in #17308
chore(query): improve fuse row fetch, use accumulating by @sundy-li in #17309
chore(ci): fix pack binaries by @everpcpc in #17332
chore(ci): fix pack binaries in release by @everpcpc in #17335
chore: fix flaky test by @SkyFan2002 in #17333
chore: Clean up not used crates by @Xuanwo in #17260
chore: enable sort spill in recluster by @zhyass in #17236
chore(query): improve udf server headers by @sundy-li in #17349
chore(ci): intergration ttc-go by @forsaken628 in #17280
chore(ci): fix intergration ttc-go by @forsaken628 in #17363
chore: remove unused _row_number by @zhyass in #17336
chore: remove unused code by @SkyFan2002 in #17371
chore(query): bump jiff crate by @sundy-li in #17366
chore(query): fix hash join hang by @Dousir9 in #17389
chore(query): row fetch will be fetched by distinct block batch by @sundy-li in #17391
chore(udf): tune the log message about udf decode error by @flaneur2020 in #17414
chore: refine copy into tests by @dantengsky in #17413
chore(metrics): add metric about opendal retries count by @flaneur2020 in #16724
chore: Use tagged version of opendal by @Xuanwo in #17419
chore: use Box::leak() to ensure logging guard lives for the program's lifetime by @drmingdrmer in #17425
chore(query): cast variant to other type allow cast JSON null as SQL NULL by @b41sh in #17412
chore(query): add new session setting: query_tag by @TCeason in #17430
chore: update enterprise edition license url by @TCeason in #17445
chore(query): testing tpch/tpcds using 1G scale factor by @sundy-li in #17449