Introducing Debezium: Real-time Change Data Capture Made Easy with Databend

In today's fast-paced world, businesses require real-time data synchronization and event-driven architectures to stay competitive. In this blog, we will explore Debezium and its lightweight counterpart, debezium-server-databend, developed by Databend. We will discuss how to install and use debezium-server-databend to effortlessly monitor and capture database changes and integrate them into Databend.

Introducing Debezium & debezium-server-databend

Debezium is a robust set of distributed services designed to capture changes within databases, enabling applications to respond promptly to those changes. This is achieved through recording all row-level changes within each database table and streaming these change events to applications in chronological order.

Debezium's potential reaches new heights with debezium-server-databend, a lightweight CDC tool developed by Databend. It harnesses the capabilities of Debezium Engine to capture real-time changes in relational databases and stream them as events into Databend. What sets this tool apart is its simplicity, as it requires no large data infrastructures like Flink, Kafka, or Spark.

Installing debezium-server-databend

debezium-server-databend can be installed independently without the need for installing Debezium beforehand. Once you have decided to install debezium-server-databend, you have two options available. The first one is to install it from source by downloading the source code and building it yourself. Alternatively, you can opt for a more straightforward installation process using Docker.

For step-by-step instructions on how to install the tool, see the Databend documentation at https://docs.databend.com/doc/load-data/load-db/debezium#installing-debezium-server-databend

Loading Data with debezium-server-databend

The key of importing data using debezium-server-databend lies in the configuration file named "application.properties," which serves as the core of the entire data import process. This file needs to be tailored according to specific requirements. Below is an example of an "application.properties" file for importing data from MYSQL to Databend:

debezium.sink.type=databend
debezium.sink.databend.upsert=true
debezium.sink.databend.upsert-keep-deletes=false
debezium.sink.databend.database.databaseName=debezium
debezium.sink.databend.database.url=jdbc:databend://<your-databend-host>:<port>
debezium.sink.databend.database.username=<your-username>
debezium.sink.databend.database.password=<your-password>
debezium.sink.databend.database.primaryKey=id
debezium.sink.databend.database.tableName=products
debezium.sink.databend.database.param.ssl=true

# enable event schemas
debezium.format.value.schemas.enable=true
debezium.format.key.schemas.enable=true
debezium.format.value=json
debezium.format.key=json

# mysql source
debezium.source.connector.class=io.debezium.connector.mysql.MySqlConnector
debezium.source.offset.storage.file.filename=data/offsets.dat
debezium.source.offset.flush.interval.ms=60000

debezium.source.database.hostname=127.0.0.1
debezium.source.database.port=3306
debezium.source.database.user=root
debezium.source.database.password=123456
debezium.source.database.dbname=mydb
debezium.source.database.server.name=from_mysql
debezium.source.include.schema.changes=false
debezium.source.table.include.list=mydb.products
# debezium.source.database.ssl.mode=required
# Run without Kafka, use local file to store checkpoints
debezium.source.database.history=io.debezium.relational.history.FileDatabaseHistory
debezium.source.database.history.file.filename=data/status.dat
# do event flattening. unwrap message!
debezium.transforms=unwrap
debezium.transforms.unwrap.type=io.debezium.transforms.ExtractNewRecordState
debezium.transforms.unwrap.delete.handling.mode=rewrite
debezium.transforms.unwrap.drop.tombstones=true

# ############ SET LOG LEVELS ############
quarkus.log.level=INFO
# Ignore messages below warning level from Jetty, because it's a bit verbose
quarkus.log.category."org.eclipse.jetty".level=WARN

Databend provides a sample "application.properties" file, which can be found at the following link: https://github.com/databendcloud/debezium-server-databend/blob/main/debezium-server-databend-dist/src/main/resources/distro/conf/application.properties.example. This file serves as a useful starting point for configuring your data import process using debezium-server-databend. For explanations of the parameters in the configuration file, please refer to https://github.com/databendcloud/debezium-server-databend/blob/main/docs/docs.md.

The Databend documentation also offers an example of importing data from MYSQL, available at https://docs.databend.com/doc/load-data/load-db/debezium#usage-example.