Step 1 : Create RDS Database
a. SQL Server Standard Edition
b. In the security group allow TCP traffic from all sources on port 1433
Step 2 : Create Database, table , enable CDC on table and insert the data into table
a. execute the scripts given under database/cdc_script.sql
Step 3 : Create S3 Bucket , and Setup S3 VPC endpoint
a. When creating endpoint choose type "Gateway" and service name "com.amazonaws.us-east-1.s3" (change region according to your bucket)
Step 3: Create MSK Cluster
a. Cluster type : Provisioned
b. Security group : allow inbound and outbound traffic from everywhere (you can keep it more restricted to allow only from required sources)
c. Authentication type : IAM
Step 4: Create an IAM Role for MSK with the policy given in Repo
a. IAM policy details given here : IAM/MSKConnectorRole.json
Step 5: Create connector plugins for SQL Server Source and S3 Sink
a. Debezium Connector for SQL Server : https://www.confluent.io/hub/debezium/debezium-connector-sqlserver
b. Kafka-S3-Sink Connector: https://www.confluent.io/hub/confluentinc/kafka-connect-s3
Step 6: Create the source connector
follow the steps explained in the video to create the source connector
Step 7: Create and EC2 instance (This step is optional , we are doing this to just to view the data in Kafka topic)
a. Create EC2 instance
b. Create EC2 service role with IAM permissions given here : IAM/ecRole.json
Step 8: Verify the data in topic (This step is optional , we are doing this to just to view the data in Kafka topic)
a. Install Kafka by using the commands given here : kafka/kafka_commands.txt
b. Use the console consumer command (#7 in above file) to view the data in topic
Step 9: Create S3 Sink Connector
follow the steps explained in the video to create the Sink connector
Step 10: Verify the data in target S3 Bucket
