Big Data

• Dealing with web-scale data
• How big is big data?
• Where is the data?

Map/Reduce Algorithms

• Map-Only
• Sorting
• Inverted Indexes
• Counting and Summing
• Filtering
• Trying out a few examples

Map/Reduce

• The traditional parallel programming paradigm
• Issues with traditional parallel programming
• Introduction to Map/Reduce
• Thinking in Map/Reduce

Introduction to Spark

• What is Spark?
• Differences from Pig and Hive
• Installing Spark
• Where to use Spark
• Linking with Spark
• Using the Shell

Resilient Distributed Datasets (RDDs)

• Parallelized Collections
• External Datasets
• RDD Operations
  • Basics
  • Passing functions to Spark
  • Closures
  • Working with Key-Value Pairs
  • Transformations
  • Actions
  • Shuffle operations

RDD Persistence

• Which Storage Level to Choose?
• Removing Data

Shared Variables

• Broadcast Variables
• Accumulators

Deploying to a Cluster

• Launching Application with spark-submit
• Advanced Dependency Management

Launching Spark jobs from Java/Scala

• Spark Launcher
• Spark App handle

Using Spark with Different Languages

• Scala
• Java
• Python
• R

PySpark and Jupyter

• Introduction to PySpark
• Using Jupyter notebook
• Using pySpark shell
• PySpark in Jupyter