Strimzi An Opensource Data Pipeline.

  1. Create Kubernetes cluster over GCP.
  2. Create Namespace with a name as per your requirements.
  3. Apply Strimzi installation file over Kubernetes Cluster: ``` kubectl apply -f ‘https://strimzi.io/install/latest?namespace=kafka' -n kafka``` (Here namespace = ‘kafka’)
  4. Provision the Apache Kafka cluster from GitHub repository (https://github.com/strimzi/strimzi-kafka-operator/tree/master/examples/kafka) : ``` kubectl apply -f kafka-persistent-single.yaml — namespace=kafka```. We will get a response ‘Response: configured’
  5. We need to apply wait command: ```kubectl wait kafka/my-cluster — for=condition=Ready — timeout=300s — namespace=kafka```.
  6. Provision kafka connector using the operator with following yaml: ```kubectl apply -f kafka-connect.yaml — namespace=kafka```. Github repository (https://github.com/strimzi/strimzi-kafka-operator/blob/release-0.19.x/examples/connect/kafka-connect.yaml)
  7. Download Biquery connector & Configure it: download here.

Few Common used commands are:-

· Kubectl get kafka -n kafka

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Vibhor Gupta

Vibhor Gupta

11 Followers

Hi, I am a Certified Google Cloud Data engineer. I use Medium platform to share my experience with other members of Medium network.