CS553 Homework #8 Sort on Hadoop/Spark

HOMEWORK > CS553 Homework #8 Sort on Hadoop/Spark

Illinois Institute Of Technology CS 553 Instructions: 1. Assigned date: Wednesday, 04/24/19 2. Due date: 11:59PM on Friday, 05/03/19 3. Maximum Points: 100% 4. This programming assignment must be done individually 5. Please post your questions to the Piazza forum 6. Only a softcopy submission is required; it will automatic ...[Show More]

Illinois Institute Of Technology
CS 553
Instructions:

1. Assigned date: Wednesday, 04/24/19

2. Due date: 11:59PM on Friday, 05/03/19

3. Maximum Points: 100%

4. This programming assignment must be done individually

5. Please post your questions to the Piazza forum

6. Only a softcopy submission is required; it will automatically be collected through GIT after the deadline;

email confirmation will be sent to your HAWK email address

7. Late submission will be penalized at 10% per day; an email to [email protected] with the subject

“CS553: late homework submission” must be sent; late submissions will not be accepted after May 8th, 2019

1. Introduction

The goal of this programming assignment is to enable you to gain experience programming with:

• The Hadoop framework (http://hadoop.apache.org/)

• The Spark framework (http://spark.apache.org/)

In Homework #6, you implemented an external sort and compared it to the Linux sort. You will now expand into

implementing sort with Hadoop and with Spark.

2. Your Assignment

This programming assignment covers sort through Hadoop and Spark on multiple nodes. You must use a

Chameleon node using Bare Metal Provisioning (https://www.chameleoncloud.org). You must deploy Ubuntu

Linux 16.04 using either “compute-skylake” or “compute-haswell” nodes, at either UC or TACC sites. Once you

create a lease (up to 7 days are allowed), and start your 1 physical node, and Linux boots, you will find yourself

with a physical node with 24 CPU cores, 48 hardware threads, 128GB to 192GB of memory (depending on if its

Haswell or Skylake), and 250GB SSD hard drive. You will install your favorite virtualization or containerization

tools (e.g. virtualbox, KVM, qemu, Docker, lxc/lxd), and use it to deploy two different VMs/containers with the

following sizes: small.instance (4-cores, 8gb ram, 50gb disk), and large.instance (16-cores, 32GB ram, 200gb

disk)