Yet Another Resource Negotiator

YARN Architecture

Launching an application

yarn jar application.jar DriverClass input output

List running jobs

yarn application -list
yarn top

See application logs

yarn logs -applicationId applicationId

Kill an application

yarn application -kill applicationId

Fair Scheduler

  • Resources will be shared with the rest of users using the YARN fair share scheduler
  • Dominant Resource Fairness: both CPU and memory considered
  • Jobs should be composed of lots of short running tasks so they share resources nicely with other jobs
  • Long running tasks that monopolize resources during large times can be preempted to allow other applications to run

Fair Scheduler Queues

  • root.users.[username]: default queue, one per user
  • interactive: Jupyter Notebooks and interactive jobs
  • urgent: limited resources that can be used for urgent jobs

Web Job Browser

You can access the HUE Job Browser from the WebUI: