Overview

What is EMR?

• Elastic MapReduce is a managed clustered platform to run big data frameworks

  

  • Over 21 different frameworks available!
  • They are “out of box”, pre-installed
  • But really Spark is the one we’re interested in

• It’s on-demand

  • Companies can save money by spinning up clusters up/down without having to setup their own Hadoop clusters
  • Cost savings using EMR managed auto-scaling

• Effective for OLAP (Online analytical processing) and Batch Processing jobs

  • At a minimum, you’ll need terabytes worth of data processing
  • Suitable for “embarrassingly parallel” tasks

• By default, EMR uses YARN (Yet Another Resource Negotiator) for Cluster resource management

  • Consists of Resource Manager and Node Managers to coordinate job scheduling

  

  YARN was introduced in Apache Hadoop 2.0 to help with cluster management

Integration with AWS Services

There exists many AWS Services that integrate with EMR. The biggest ones to know are

• Amazon S3
  • Leverage S3 as our persistent data store
  • Decouple storage and compute. Avoids “cold data”
  • EMR File System (EMRFS) lets you to write to S3

• Amazon EC2
  • Provision virtual machines for our worker nodes
  • Many different family types
    • i.e. m5 for general batch processing, c5 for machine learning, x1 for memory intensive, d2 for storage
  • Fleet instances allow you to mix and match on-demand, spot, reserved instances

• Amazon VPC
  • Secure your networking environment
  • Limit your IP addresses, subnets, and traffic
  • Don’t put all your services in a single, default VPC

Other services worth knowing: AWS Step functions for orchestration, AWS Glue for serverless

Architecture for Amazon EMR

An EMR cluster is made of 3 node types