Published in DataDrivenInvestor·PinnedCDC Pipeline From Mysql to SnowflakeMySQL is an open-source relational database system that is part of most of the enterprise tech stack today. In a general architecture, MySQL is used as the transactional database with a separate data warehouse to support the analytical and reporting or to other downstream application requirements. Change-Data-Capture (CDC) allows capturing…Snowflake6 min read
Published in Mac O’Clock·PinnedInstalling PrestoDB on Mac in 3 minsWHAT IS PRESTO? Presto is an open source distributed SQL query engine for running interactive analytic queries against data sources of all sizes ranging from gigabytes to petabytes. Installation Dependency to install Presto is only Java 8+ ( homebrew will take care of dependency ) brew install presto As part of the install, the…Presto3 min read
Published in Analytics Vidhya·Jan 20Avro Utility Tool: Swiss Army knifeDo you require to deal with Avro data format in your day-to-day job? If your answer is yes, then you must learn how to use the Avro-tool utility. Before we jump to this utility let’s recap: What does Avro provide? Rich data structures. A compact, fast, binary data format. …Data Science3 min read
Published in Towards AWS·Jan 4EC2 Hibernate — Good to knowI am working with AWS EC2 for quite a long time now. But recently Stop — Hibernate option caught my eye and I was amused that we work in such autopilot mode sometimes that we ignore small details which were in front of our eyes all the time. …AWS2 min read
Jun 16, 2021Become certified calico operator: L1In this article, I will share my experience with Calico Operator certification course. What do you learn in this course? This free and self-paced course will arm you with the knowledge you need to understand how Kubernetes networking works, how to configure and manage a Calico network, and how to…Calico2 min read
Published in Towards AWS·May 24, 2021How to Configuring Linux ulimit with Docker and AWS ECSWhat is ulimit? The purpose of ulimits is to limit a program's resource utilization to prevent a run-away bug or security breach from bringing the whole system down. It is easy for modern applications to exceed default open files and other limits very quickly. How to check the Hard limit and soft limit? Check hard limit ulimit -a -H Check soft limit…AWS3 min read
May 4, 2021StreamSets Transformer pipeline to Dataproc in 5 minsThis tutorial is for Spark developers who want to learn an easy and quick way to run a Transformer pipeline on the Dataproc cluster via StreamSets Transformer In my previous article explained what is StreamSets Transformer and how to deploy this in minishift. …Gcp3 min read
Published in DataDrivenInvestor·May 3, 2021Deploying Hadoop in Kubernetes via helmIn this article, we are going to deploy the Hadoop cluster to our local K8s cluster. First thing first, Let’s create a local k8s cluster. For this purpose, we will install minikube. Install Minikube Prerequisites Install Docker for Mac. Docker is used to create, manage, and run our containers. …Kubernetes5 min read
Published in Analytics Vidhya·Feb 1, 2021Data Preparation: Getting spark ready with StreamSets Data CollectorSpark can analyze data stored on files in many different formats: plain text, JSON, XML, Parquet, and more. But just because you can get a Spark job to run on a given data input format doesn’t mean you’ll get the same performance with all of them. …Spark4 min read
Oct 20, 2020Streaming Kafka Messages to Google Cloud Pub/Sub with StreamSetsWhether your Kafka is provisioned in the Cloud or on-premise, you might want to push to a subset of Pub/Sub topics. Why? For the flexibility of having Pub/Sub as your GCP event notifier or to use topics to trigger Cloud Functions. …Streaming3 min read
