Data Engineering on Data Science Blog

Introduction to Kafka

Sat, 07 Jul 2018 00:00:00 +0000

Apache Kafka is a message publishing framework that works in a distributed environment. Kafka can be scaled horizontally with high fault-tolerance.

Spark Structured Streaming - File-to-File Real-time Streaming (3/3)

Thu, 28 Jun 2018 00:00:00 +0000

In this post we will see how to build a simple application to process file to file real time processing.

Spark Structured Streaming - Socket Word Count (2/3)

Wed, 20 Jun 2018 00:00:00 +0000

Structured Streaming is a new of looking at realtime streaming. In this post we will see how to build our very first Structured Streaming app to perform Word Count over network.

Spark Structured Streaming - Introduction (1/3)

Thu, 14 Jun 2018 00:00:00 +0000

Structured Streaming is a new of looking at realtime streaming. With abstraction on DataFrame and DataSets, structured streaming provides alternative for the well known Spark Streaming. Structured Streaming is built on top of Spark SQL Engine. Some of the main features of Structured Streaming are -

MongoDB Data Processing (Python)

Mon, 21 May 2018 00:00:00 +0000

This post will give an insight of data processing from MonogDB in Python.

Mongo Shell Commands - Mongo Document Queries

Wed, 16 May 2018 00:00:00 +0000

This post will introduce mongo shell and basic query operations that can be performed on mongo shell with examples.

Spark - MongoDB Data Processing (Scala)

Wed, 16 May 2018 00:00:00 +0000

We will look into basic details of how to process data from MongoDB using Apache Spark.

Install MongoDB

Tue, 15 May 2018 00:00:00 +0000

This post is a step-by-step guide to install MongoDB on Mac.

Detailed Guide to Setting up Scalable Apache Spark Infrastructure on Docker - Standalone Cluster With History Server

Fri, 11 May 2018 00:00:00 +0000

This post is a complete guide to build a scalable Apache Spark on using Dockers. We will see how to enable History Servers for log persistence. To be able to scale up and down is one of the key requirements of today’s distributed infrastructure. By the end of this guide, you should have pretty fair understanding of setting up Apache Spark on Docker and we will see how to run a sample program.

How to Setup PyCharm to Run PySpark Jobs

Fri, 27 Apr 2018 00:00:00 +0000

This post will give a walk through of how to setup your local system to test PySpark jobs. Followed by demo to run the same code using spark-submit command.

Spark - Cassandra Data Processing (Scala)

Thu, 26 Apr 2018 00:00:00 +0000

We will look into basic details of how to process data from Cassandra using Apache Spark. Data Processing from a NoSQL DB is very efficient when we use a distributed processing system like Spark in Scala

How to Setup Eclipse to Run Spark Programs Using Gradle as Build Tool

Thu, 19 Apr 2018 00:00:00 +0000

This post will give a walk through of how to setup your local system to test Spark programs. We will use Gradle as build tool. Additionally we will see how to run the same code using spark-submit command.

Install Apache Cassandra

Thu, 19 Apr 2018 00:00:00 +0000

This post will guide you through installation of Apache Cassandra.

Introduction to Cassandra Query Language (CQL)

Thu, 19 Apr 2018 00:00:00 +0000

This post gives a quick introduction to Cassandra using CQL.

Install Apache Spark 2.3

Mon, 09 Apr 2018 00:00:00 +0000

This post will guide you through installation of Apache Spark 2.3.

Install Gradle

Sat, 07 Apr 2018 00:00:00 +0000

Install Gradle on Mac system easily and quickly. This post will give a complete walk through of installing Gradle using brew and manual installation

Install Maven

Sat, 07 Apr 2018 00:00:00 +0000

Install Maven on Mac system easily and quickly. This post will give a complete walk through of installing Maven using brew and manual installation

Quick Introduction to Build Tools

Fri, 06 Apr 2018 00:00:00 +0000

In this post we will look at a basic introduction to build tools and a few distinguishing factors between widely used build tools.