Databend is a powerful cloud data warehouse. Built for elasticity and efficiency. Free and open. Also available in the cloud: https://app.databend.com .
What's New
Check out what we've done this week to make Databend even better for you.
Features & Improvements ✨
Multiple Catalogs
- extends show databases SQL (#9152)
Stage
- support select from URI (#9247)
Streaming Load
- support syntax in streaming load insert sql (#9063)
file_format
Planner
- push down to
limit
(#9210)union
Query
Storage
- use distinct count calculated by the xor hash function (#9159)
- read meta before read data (#9154)
read_parquet
- push down filter to parquet reader (#9199)
- prune row groups before reading (#9228)
Open Sharing
- add prototype open sharing and add sharing stateful tests (#9177)
Code Refactoring 🎉
*
- simplify the global data registry logic (#9187)
Storage
- refactor deletion (#8824)
Build/Testing/CI Infra Changes 🔌
Bug Fixes 🔧
Format
- support ASCII control code hex as format field delimiter (#9160)
Planner
- prewhere_column empty and predicate is not const will return empty (#9116)
- don't push down topk to Merge when it's child is Aggregate (#9183)
- fix nullable column validity not equal (#9220)
Query
- address unit test hang on test_insert (#9242)
Storage
What's On In Databend
Stay connected with the latest news about Databend.
Breaking Change: Unified File Format Options
To simplify, we're rolling out a set of unified file format options as follows for the COPY INTO command, the Streaming Load API, and all the other cases where users need to describe their file formats:
[ FILE_FORMAT = ( TYPE = { CSV | TSV | NDJSON | PARQUET | XML} [ formatTypeOptions ] ) ]
- Please note that the current format options starting with will be deprecated.
format_*
- will still be accepted by the ClickHouse handler.
... FORMAT CSV ...
- Support for customized formats created by will be added in a future release:
CREATE FILE FORMAT ...
.... FILE_FORMAT = (format_name = 'MyCustomCSV') ....
Learn More
- Issue | unify format options and remove format_ settings
- PR | support file_format syntax in streaming load insert sql
Open Sharing
Open Sharing is a simple and secure data-sharing protocol designed for databend-query nodes running in a multi-cloud environment.
- Simple & Free: Open Sharing is open-source and basically a RESTful API implementation.
- Secure: Open Sharing verifies incoming requesters' identities and access permissions, and provides an audit log.
- Multi-Cloud: Open Sharing supports a variety of public cloud platforms, including AWS, Azure, GCP, etc.
Learn More
What's Up Next
We're always open to cutting-edge technologies and innovative ideas. You're more than welcome to join the community and bring them to Databend.
Refactoring Stage-Related Tests
We're about to run stage-related tests again using the Streaming Load API to move files to a stage instead of an AWS command like this:
aws --endpoint-url ${STORAGE_S3_ENDPOINT_URL} s3 cp s3://testbucket/admin/data/ontime_200.csv s3://testbucket/admin/stage/internal/s1/ontime_200.csv >/dev/null 2>&1
This is because Databend users do not need to take care of, or do not even know the stage paths that the AWS command requires.
Issue 8528: refactor stage related tests
Please let us know if you're interested in contributing to this issue, or pick up a good first issue at https://link.databend.com/i-m-feeling-lucky to get started.
Changelog
You can check the changelog of Databend Nightly for details about our latest developments.
- v0.8.152-nightly
- v0.8.151-nightly
- v0.8.150-nightly
- v0.8.149-nightly
- v0.8.148-nightly
- v0.8.147-nightly
- v0.8.146-nightly
- v0.8.145-nightly
Contributors
Thanks a lot to the contributors for their excellent work this week.
ariesdevil | b41sh | BohuTANG | Chasen-Zhang | ClSlaid | dantengsky |
drmingdrmer | hantmac | lichuang | mergify[bot] | PsiACE | RinChanNOWWW |
soyeric128 | sundy-li | wubx | Xuanwo | xudong963 | youngsofun |
ZhiHanZ | zhyass | zzzdong |
Subscribe to our newsletter
Stay informed on feature releases, product roadmap, support, and cloud offerings!