English | MP4 | AVC 1280×720 | AAC 48KHz 2ch | 3.5 Hours | 414 MB
Learn analyzing large data sets with Apache Spark by 10+ hands-on examples. Take your big data skills to the next level.
This course covers all the fundamentals about Apache Spark with Java and teaches you everything you need to know about developing Spark applications with Java. At the end of this course, you will gain in-depth knowledge about Apache Spark and general big data analysis and manipulations skills to help your company to adapt Apache Spark for building big data processing pipeline and data analytics applications.
This course covers 10+ hands-on big data examples. You will learn valuable knowledge about how to frame data analysis problems as Spark problems. Together we will learn examples such as aggregating NASA Apache web logs from different sources; we will explore the price trend by looking at the real estate data in California; we will write Spark applications to find out the median salary of developers in different countries through the Stack Overflow survey data; we will develop a system to analyze how maker spaces are distributed across different regions in the United Kingdom. And much much more.
What will you learn from this lecture:
In particularly, you will learn:
- An overview of the architecture of Apache Spark.
- Develop Apache Spark 2.0 applications with Java using RDD transformations and actions and Spark SQL.
- Work with Apache Spark’s primary abstraction, resilient distributed datasets(RDDs) to process and analyze large data sets.
- Deep dive into advanced techniques to optimize and tune Apache Spark jobs by partitioning, caching and persisting RDDs.
- Scale up Spark applications on a Hadoop YARN cluster through Amazon’s Elastic MapReduce service.
- Analyze structured and semi-structured data using Datasets and DataFrames, and develop a thorough understanding of Spark SQL.
- Share information across different nodes on an Apache Spark cluster by broadcast variables and accumulators.
- Best practices of working with Apache Spark in the field.
- Big data ecosystem overview.
Table of Contents
Get Started with Apache Spark
1 Course Overview
2 Introduction to Spark
3 Sides
4 Java 9 Warning
5 Install Java and Git
6 Source Code
7 Set up Spark project with IntelliJ IDEA
8 Set up Spark project with Eclipse
9 Text lecture Set up Spark project with Eclipse
10 Run our first Spark job
11 Trouble shooting running Hadoop on Windows
RDD
12 RDD Basics
13 Create RDDs
14 Text Lecture Create RDDs
15 Map and Filter Transformation
16 Solution to Airports by Latitude Problem
17 FlatMap Transformation
18 Text Lectures flatMap Transformation
19 Set Operation
20 Sampling With Replacement and Sampling Without Replacement
21 Solution for the Same Hosts Problem
22 Actions
23 Solution to Sum of Numbers Problem
24 Important Aspects about RDD
25 Summary of RDD Operations
26 Caching and Persistence
Spark Architecture and Components
27 Spark Architecture
28 Spark Components
Pair RDD
29 Introduction to Pair RDD
30 Create Pair RDDs
31 Filter and MapValue Transformations on Pair RDD
32 Reduce By Key Aggregation
33 Sample solution for the Average House problem
34 Group By Key Transformation
35 Sort By Key Transformation
36 Sample Solution for the Sorted Word Count Problem
37 Data Partitioning
38 Join Operations
39 Extra Learning Material How are Big Companies using Apache Spark
Advanced Spark Topic
40 Accumulators
41 Text Lecture Accumulators
42 Solution to StackOverflow Survey Follow-up Problem
43 Broadcast Variables
Spark SQL
44 Introduction to Spark SQL
45 Spark SQL in Action
46 Spark SQL practice House Price Problem
47 Spark SQL Joins
48 Strongly Typed Dataset
49 Use Dataset or RDD
50 Dataset and RDD Conversion
51 Performance Tuning of Spark SQL
52 Extra Learning Material Avoid These Mistakes While Writing Apache Spark Program
Running Spark in a Cluster
53 Introduction to Running Spark in a Cluster
54 Package Spark Application and Use spark-submit
55 Run Spark Application on Amazon EMR Elastic MapReduce cluster
Additional Learning Materials
56 Coupons to Our Other Courses
Resolve the captcha to access the links!