Introduction to Big Data with Spark and Hadoop

Explain the impact of big data, including use cases, tools, and processing methods.
Describe Apache Hadoop architecture, ecosystem, practices, and user-related applications, including Hive, HDFS, HBase, Spark, and MapReduce.
Apply Spark programming basics, including parallel programming basics for DataFrames, data sets, and Spark SQL.
Use Spark’s RDDs and data sets, optimize Spark SQL using Catalyst and Tungsten, and use Spark’s development and runtime environment options.

Course Content

Module 1: What is Big Data?

Course Introduction

00:00
What is Big Data?

00:00
Impact of Big Data

00:00
Parallel Processing, Scaling, and Data Parallelism

00:00
Big Data Tools and Ecosystem

00:00
Open Source and Big Data

00:00
Beyond the Hype

00:00
Big Data Use Cases

00:00

1 reading

2 assignments

1 plugin

Module 2: Introduction to the Handoop Ecosystem

1 reading

2 assignments

3 app items

2 plugins

Module 3: Apache Spark

1 reading

2 assignments

1 app item

2 plugins

Module 4: DataFrames and Spark SQL

1 reading

2 assignments

2 app items

4 plugins

Module 5: Development and Runtime Environment Options

2 readings

3 assignments

2 app items

4 plugins

Module 6: Monitoring and Tuning

1 reading

2 assignments

1 app item

3 plugins

Module 7: Final Project and Assessment

Student Ratings & Reviews

No Review Yet

What Will You Learn?

Course Content

Module 1: What is Big Data?

Course Introduction

What is Big Data?

Impact of Big Data

Parallel Processing, Scaling, and Data Parallelism

Big Data Tools and Ecosystem

Open Source and Big Data

Beyond the Hype

Big Data Use Cases

1 reading

Summary and Highlights: Introduction to Big Data

2 assignments

Practice Quiz: Introduction to Big Data

Graded Quiz: What Is Big Data?

1 plugin

Module 1 Glossary: What Is Big Data?

Module 2: Introduction to the Handoop Ecosystem

Introduction to Hadoop

Intro to MapReduce

Hadoop Ecosystem

HDFS

HIVE

HBASE

1 reading

Summary and Highlights: Introduction to Hadoop

2 assignments

Practice Quiz: Introduction to Hadoop

Graded Quiz: Introduction to Hadoop Ecosystem

3 app items

Hands-on Lab: Getting Started with Hive

Hands-on Lab: Hadoop MapReduce

Hands-on lab : Hadoop Cluster (Optional)

2 plugins

Cheat Sheet: Introduction to the Hadoop Ecosystem

Module 2 Glossary: Introduction to the Hadoop Ecosystem

Module 3: Apache Spark

Why use Apache Spark?

Functional Programming Basics

Parallel Programming using Resilient Distributed Datasets

Scale out / Data Parallelism in Apache Spark

Dataframes and SparkSQL

1 reading

Summary and Highlights: Introduction to Apache Spark

2 assignments

Practice Quiz: Introduction to Apache Spark

Graded Quiz: Apache Spark

1 app item

Hands-on Lab: Getting Started with Spark using Python

2 plugins

Cheat Sheet: Apache Spark

Module 3 Glossary: Apache Spark

Module 4: DataFrames and Spark SQL

RDDs in Parallel Programming and Spark

Data-frames and Datasets

Catalyst and Tungsten

ETL with DataFrames

Real-world usage of SparkSQL

1 reading

Summary and Highlights: Introduction to DataFrames and Spark SQL

2 assignments

Practice Quiz: Introduction to DataFrames & Spark SQL

Graded Quiz: DataFrames and Spark SQL

2 app items

Hands-on Lab: Introduction to DataFrames

Hands-On Lab: Introduction to SparkSQL

4 plugins

Reading: User-Defined Schema (UDS) for DSL and SQL

Reading: Common Transformations and Optimization Techniques in Spark

Cheat Sheet: DataFrames and Spark SQL

Module 4 Glossary: DataFrames and Spark SQL

Module 5: Development and Runtime Environment Options

Apache Spark Architecture

Overview of Apache Spark Cluster Modes

How to Run an Apache Spark Application

Using Apache Spark on IBM Cloud

Setting Apache Spark Configuration

Running Spark on Kubernetes

2 readings