Automate Data Loading from Amazon S3

We're happy to announce the addition of pipeline support in Databend Cloud! With pipelines, you can now automatically discover file updates from Amazon S3 and seamlessly load them into your tables in Databend Cloud for further analysis.

This feature is now available and completely free of charge, with a limit of 4 pipelines per organization. Read on to learn more about how pipelines can streamline your data loading process and enhance your data analytics workflows.

Why Pipeline?

Pipelines in Databend Cloud are designed to simplify and automate the process of loading data from Amazon S3 into your tables. Here are some scenarios where using a pipeline is recommended:

1. Large-scale data loading: If you have a large number of CSV or Parquet files in an S3 bucket that you need to load into Databend Cloud in one go, pipelines can efficiently handle the task, saving you time and effort.

2. Automated data ingestion: If AWS automatically loads data into your S3 buckets, such as billing data or user behavior logs, pipelines can automatically detect and load the updates into Databend Cloud, enabling you to visualize and analyze the data in real-time.

3. Continuous data streaming: If you have a continuous stream of user behavior logs or other data being stored in S3, pipelines can automatically load the data into Databend Cloud, allowing you to perform real-time analysis and gain valuable insights.

How to Create a Pipeline

Creating a pipeline in Databend Cloud is easy and straightforward. For a step by step instruction on how to create a pipeline, see https://docs.databend.com/load-transform-data/pipeline

Pipelines in Databend Cloud provide a powerful and convenient way to automate data loading from Amazon S3 into your tables, making it easier than ever to analyze and gain insights from your data. During the public beta phase, pipelines are offered for free on Databend Cloud, with a limit of 4 pipelines per organization.

We encourage you to try out this exciting new feature and experience the benefits of streamlined data loading in your data analytics workflows. If you have any questions or need additional support, please don't hesitate to contact us by submitting a ticket. We're here to help!