Big Data Integration and Processing
About Course
There are 7 modules in this course
At the end of the course, you will be able to:
*Retrieve data from example database and big data management systems *Describe the connections between data management operations and the big data processing patterns needed to utilize them in large-scale analytical applications *Identify when a big data problem needs data integration *Execute simple big data integration and processing on Hadoop and Spark platforms This course is for those new to data science. Completion of Intro to Big Data is recommended. No prior programming experience is needed, although the ability to install applications and utilize a virtual machine is necessary to complete the hands-on assignments. Refer to the specialization technical requirements for complete hardware and software specifications. Hardware Requirements: (A) Quad Core Processor (VT-x or AMD-V support recommended), 64-bit; (B) 8 GB RAM; (C) 20 GB disk free. How to find your hardware information: (Windows): Open System by clicking the Start button, right-clicking Computer, and then clicking Properties; (Mac): Open Overview by clicking on the Apple menu and clicking “About This Mac.” Most computers with 8 GB RAM purchased in the last 3 years will meet the minimum requirements.You will need a high speed internet connection because you will be downloading files up to 4 Gb in size. Software Requirements: This course relies on several open-source software tools, including Apache Hadoop. All required software can be downloaded and installed free of charge (except for data charges from your internet provider). Software requirements include: Windows 7+, Mac OS X 10.10+, Ubuntu 14.04+ or CentOS 6+ VirtualBox 5+.
What Will You Learn?
- Earn new concepts from industry experts
- Gain a foundational understanding of a subject or tool
- Develop job-relevant skills with hands-on projects
- Earn a shareable career certificate
Course Content
Module 1: Welcome to Big Data Integration and Processing
-
What is in this Course?
00:00 -
Summary of Big Data Modeling and Management
00:00 -
Why is Big Data Processing Different?
00:00
6 readings
-
Slides: Summary & Why Is Big Data Processing Different
00:00 -
Downloading and Installing the Cloudera VM Instructions (Windows)
00:00 -
Downloading and Installing the Cloudera VM Instructions (Mac)
00:00 -
Software Installation Frequently Asked Questions (FAQ)
00:00 -
Instructions for Downloading Hands On Datasets
00:00 -
Instructions for Starting Jupyter
00:00
1 discussion prompt
-
Getting to know you: Tell us about yourself and why you are taking this course.
00:00
Module 2: Retrieving Big Data (Part 1)
-
What is Data Retrieval? Part 1
00:00 -
What is Data Retrieval? Part 2
00:00 -
Querying Two Relations
00:00 -
Subqueries
00:00 -
Querying Relational Data with Postgres
00:00
2 readings
-
Slides: What is Data Retrieval?
00:00 -
Querying Relational Data with Postgres
00:00
Module 3: Retrieving Big Data (Part 2)
-
Querying JSON Data with MongoDB
00:00 -
Aggregation Functions
00:00 -
Querying Aerospike
00:00 -
Querying Documents in MongoDB
00:00 -
Exploring Pandas DataFrames
00:00
3 readings
-
Slides: Querying Data Part 2
00:00 -
Querying Documents in MongoDB
00:00 -
Exploring Pandas DataFrames
00:00
2 quizzes
-
Retrieving Big Data Quiz
00:00 -
Postgres, MongoDB, and Pandas
00:00
1 discussion prompt
-
Let’s Discuss: MongoDB
00:00
Module 4: Big Data Integration
-
Overview of Information Integration
00:00 -
A Data Integration Scenario
00:00 -
Integration for Multichannel Customer Analytics
00:00 -
Big Data Management and Processing Using Splunk and Datameer
00:00 -
Why Splunk?
00:00 -
Connected Cars with Ford’s OpenXC and Splunk
00:00 -
Big Data Management and Processing using Datameer
00:00 -
Installing Splunk Enterprise on Windows
00:00 -
Installing Splunk Enterprise on Linux
00:00 -
Exploring Splunk Queries
00:00 -
Optional: Creating Pivot Reports in Splunk
00:00
4 readings
-
Slides: Information Integration
00:00 -
Downloading Splunk Enterprise
00:00 -
Exploring Splunk Queries
00:00 -
Optional: Instructions for Splunk Pivot Tutorial
00:00
2 quizzes
-
Information Integration – Quiz
00:00 -
Hands-On With Splunk
00:00
1 discussion prompt
-
Let’s Discuss: Big Data Integration
00:00
Module 5: Processing Big Data
-
Big Data Processing Pipelines
00:00 -
Some High-Level Processing Operations in Big Data Pipelines
00:00 -
Aggregation Operations in Big Data Pipelines
00:00 -
Typical Analytical Operations in Big Data Pipelines
00:00 -
Overview of Big Data Processing Systems
00:00 -
The Integration and Processing Layer
00:00 -
Introduction to Apache Spark
00:00 -
Getting Started with Spark
00:00 -
WordCount in Spark
00:00
4 readings
-
Big Data Processing Pipelines Slides
00:00 -
Big Data Workflow Management
00:00 -
Slides for Big Data Processing Tools and Systems
00:00 -
WordCount in Spark
00:00
2 quizzes
-
Pipeline and Tools
00:00 -
WordCount in Spark
00:00
3 discussion prompts
-
Let’s Discuss: Big Data Pipelines in Your World
00:00 -
Let’s Discuss: Big Data Processing Systems
00:00 -
Let’s Discuss: Word Count
00:00
Module 6: Big Data Analytics using Spark
-
Spark Core: Programming In Spark using RDDs in Pipelines
00:00 -
Spark Core: Transformations
00:00 -
Spark Core: Actions
00:00 -
Spark SQL
00:00 -
Spark Streaming
00:00 -
Spark MLLib
00:00 -
Spark GraphX
00:00 -
Exploring SparkSQL and Spark DataFrames
00:00 -
Analyzing Sensor Data with Spark Streaming
00:00
5 readings
-
Slides for Module 5 Lesson 1
00:00 -
Slides for Module 5 Lesson 2
00:00 -
Exploring SparkSQL and Spark DataFrames
00:00 -
Instructions for Configuring VirtualBox for Spark Streaming
00:00 -
Analyzing Sensor Data with Spark Streaming
00:00
2 quizzes
-
More on Spark
00:00 -
SparkSQL and Spark Streaming
00:00
1 discussion prompt
-
Let’s Discuss: The Spark Ecosystem
00:00
Module 7: Learning by Doing: Putting MongoDB and Spark to Work
-
In this module you will get some practical hands-on experience applying what you learned about Spark and MongoDB to analyze Twitter data.
00:00