Strimzi An Opensource Data Pipeline.

  1. Create Kubernetes cluster over GCP.
  2. Create Namespace with a name as per your requirements.
  3. Apply Strimzi installation file over Kubernetes Cluster: ``` kubectl apply -f ‘https://strimzi.io/install/latest?namespace=kafka' -n kafka``` (Here namespace = ‘kafka’)
  4. Provision the Apache Kafka cluster from GitHub repository (https://github.com/strimzi/strimzi-kafka-operator/tree/master/examples/kafka) : ``` kubectl apply -f kafka-persistent-single.yaml — namespace=kafka```. We will get a response ‘Response: configured’
  5. We need to apply wait command: ```kubectl wait kafka/my-cluster — for=condition=Ready — timeout=300s — namespace=kafka```.
  6. Provision kafka connector using the operator with following yaml: ```kubectl apply -f kafka-connect.yaml — namespace=kafka```. Github repository (https://github.com/strimzi/strimzi-kafka-operator/blob/release-0.19.x/examples/connect/kafka-connect.yaml)
  7. Download Biquery connector & Configure it: download here.

Few Common used commands are:-

--

--

--

Hi, I am a Certified Google Cloud Data engineer. I use Medium platform to share my experience with other members of Medium network.

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Send Emails Using SendGrid with Azure in .NET Core

AWS Cloud Practitioner Study Notes 2021

Scrum Ceremonies Experts Q&A — Part 1: Sprint Planning

Culture — Launching on Fridays

I Am Not a Programmer

Web scraping for financial statements using Python — 1

Simulation Project

Breaking the Ice with Graphs

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Vibhor Gupta

Vibhor Gupta

Hi, I am a Certified Google Cloud Data engineer. I use Medium platform to share my experience with other members of Medium network.

More from Medium

Databricks Workspace SSO: Integration with Keycloak and SAML 2.0

Databricks admin console single sign-on form

How to Run a Cassandra Operation in Docker

Running Kafka locally inside Kubernetes

Efficient fetching of data from Oracle database in Golang