Managing Big Data in Clusters and Cloud Storage

Module 1: Orientation to Data in Clusters and Cloud Storage

Welcome to the Course

00:00
Browsing Tables with Hue

00:00
Browsing Tables with SQL Utility Statements

00:00
Browsing HDFS with the Hue File Browser

00:00
Browsing HDFS from the Command Line

00:00
Understanding S3 and Other Cloud Storage Platforms

00:00
Browsing S3 Buckets from the Command Line

00:00

3 readings

1 quiz

1 discussion prompt

Module 2: Defining Databases, Tables and Columns

12 readings

Creating Databases and Tables with Hue

00:00
Creating Databases and Tables with SQL

00:00
Permissions to Create Databases and Tables

00:00
The ROW FORMAT Clause

00:00
The STORED AS Clause

00:00
The LOCATION Clause

00:00
CREATE TABLE Shortcuts

00:00
Using Hive SerDes

00:00
Working with Unstructured and Semi-Structured Data

00:00
Examining Table Structure

00:00
Dropping Databases and Tables

00:00
Modifying Existing Tables

00:00

2 quizzes

1 discussion prompt

Module 3: Data Types and File Types

12 readings

Integer Data Types

00:00
Decimal Data Types

00:00
Character String Data Types

00:00
Other Data Types

00:00
Examining Data Types

00:00
Out-of-Range Values

00:00
Text Files

00:00
Avro Files

00:00
Parquet Files

00:00
ORC Files

00:00
Other File Types

00:00
Creating Tables with Avro and Parquet Files

00:00

2 quizzes

1 discussion prompt

Module 4: Managing Datasets in Clusters and Cloud Storage

13 readings

More about HDFS Shell Commands

00:00
Chaining and Scripting with HDFS Commands

00:00
HDFS Permissions

00:00
Other Ways to Load Files into S3

00:00
S3 Permissions

00:00
Missing Values

00:00
Character Sets

00:00
Using Sqoop to Import Data

00:00
More Sqoop Import Options

00:00
Using Sqoop to Export Data

00:00
SQL LOAD DATA Statements

00:00
SQL INSERT Statements

00:00
SQL INSERT … SELECT and CTAS Statements

00:00

2 quizzes

1 peer review

1 discussion prompt

Module 5: Optimizing Hive and Impala (Honors)

20 readings

Creating and Querying Views

00:00
Modifying and Removing Views

00:00
Materialized and Non-Materialized Views

00:00
The ORDER BY Clause in Views

00:00
Choosing Which Query Engine to Use

00:00
Understanding Map Tasks and Reduce Tasks

00:00
Hive Query Performance Patterns

00:00
Understanding Execution Plans

00:00
Table and Column Statistics

00:00
Other Strategies for Query Optimization

00:00
Creating Partitioned Tables

00:00
Loading Data with Dynamic Partition

00:00
Loading Data with Static Partitioning

00:00
Risks of Using Partitioning

00:00
Complex Data Types

00:00
Creating Tables with Complex Data

00:00
Querying Complex Data with Hive

00:00
Querying Complex Data with Impala

00:00
Complex Data in Practice

00:00
Overview of Apache Kudu

00:00

2 quizzes

1 discussion prompt

What Will You Learn?

Course Content

Module 1: Orientation to Data in Clusters and Cloud Storage

Welcome to the Course

Browsing Tables with Hue

Browsing Tables with SQL Utility Statements

Browsing HDFS with the Hue File Browser

Browsing HDFS from the Command Line

Understanding S3 and Other Cloud Storage Platforms

Browsing S3 Buckets from the Command Line

3 readings

Review and Preparation

Instructions for Downloading and Installing the Exercise Environment

Troubleshooting the VM

1 quiz

Week 1 Graded Quiz

1 discussion prompt

Introduce Yourself

Module 2: Defining Databases, Tables and Columns

Week 2 Introduction

Introduction to the CREATE TABLE Statement

Using Different Schemas on the Same Data

Specifying TBLPROPERTIES

Examining, Modifying, and Removing Tables

Hive and Impala Interoperability

Impala Metadata Refresh

12 readings

Creating Databases and Tables with Hue

Creating Databases and Tables with SQL

Permissions to Create Databases and Tables

The ROW FORMAT Clause

The STORED AS Clause

The LOCATION Clause

CREATE TABLE Shortcuts

Using Hive SerDes

Working with Unstructured and Semi-Structured Data

Examining Table Structure

Dropping Databases and Tables

Modifying Existing Tables

2 quizzes

Week 2 Graded Quiz

Week 2 Practice Quiz

1 discussion prompt

Most Difficult to Understand

Module 3: Data Types and File Types

Week 3 Introduction

Overview of Data Types

Choosing the Right Data Types

Overview of File Types

Choosing the Right File Types

12 readings

Integer Data Types

Decimal Data Types

Character String Data Types

Other Data Types

Examining Data Types

Out-of-Range Values

Text Files

Avro Files

Parquet Files

ORC Files

Other File Types

Creating Tables with Avro and Parquet Files

2 quizzes

Week 3 Graded Quiz

Week 3 Practice Quiz

1 discussion prompt

What’s Your Type

Module 4: Managing Datasets in Clusters and Cloud Storage

Week 4 Introduction

Refresh Impala’s Metadata Cache after Loading Data

Loading Files into HDFS with Hue’s Table Browser

Loading Files into HDFS with Hue’s File Browser

Loading Files into HDFS from the Command Line

Loading Files into S3 from the Command Line

Using Hive and Impala to Load Data into Tables

Conclusion

13 readings

More about HDFS Shell Commands

Chaining and Scripting with HDFS Commands