5.2 The Manager
5.3 The Workers
9.2 The AWS SDK
9.6 Resizing Images
- Remember, your credit card is listed under your amazon account; so if you pass the 100$ credit your card will be charged. Yes, most of the students who had that issue were refunded, but why go there in the first place?! Save your credentials only on your computer, and don't allow any external access to it.
- The FAQ has been updated.
- Perform these tutorials.
- Read the assignment description.
- Read the reading the material, and make sure you understand it.
- Write the code that resizes an image, run it on your computer and make sure it works.
- Write the local application code and make sure it works.
- Write the manager code, run it on your computer and make sure it works.
- Run the manager, the local application, together with a worker on your computer and make sure they work.
- Run the local application on your computer, and let it start and run the manager on EC2, and let the manager start and run the workers on EC2.
AbstractIn this assignment you will code a real-world application to distributively process a list of images, resize them, and display the result on a web page. The goal of the assignment is to experiment with AWS.
More DetailsThe application is composed of a local application, and non-local instances running on Amazon cloud. The application will get as an input a text file containing a list of Images (image per line). Then, instances will be launched in AWS (workers & a manager) to resize each one of the images to 50x50 pixels. The results are displayed on a webpage.
The use-case is as follows:
- User starts the application and supplies as input a file with images, an integer n stating how many images/messages per worker (your choice), and an optional argument terminate, if received the local application sends a terminate message to the Manager.
- User gets back an html file containing the resized images clicking on an a thumbnail (resized image) should lead to the url of the original image.
See submission section bellow for instructions of what to submit.
Input File FormatEach line in the input file contains a url to an image.
Example input file.
Output File FormatThe output is an HTML file containing a thumbnail for each input image, linking to the url of the original image.
System ArchitectureThe system is composed of 3 elements:
The elements will communicate with each other using queues (SQS) and storage (S3). It is up to you to decide how many queues to use and how to split the images/jobs among the workers, but, and you will be graded accordingly, your system should strive to work in parallel. It should be as efficient as possible in terms of memory, time, and money.
Local ApplicationThe application resides on a local (non-cloud) machine. Once started, it reads the input file from the user, and:
- Checks if a Manager node is active on the EC2 cloud. If it is not, the application will start the manager node.
- Uploads the file to S3.
- Sends a message to an SQS queue, stating the location of the file on S3
- Checks an SQS queue for a message indicating the process is done and the response (the summary file) is available on S3.
- Downloads the summary file from S3, and create an html file representing the results.
- Sends a termination message to the Manager if it was supplied as one of its input arguments.
IMPORTANT: There can be more than one than one local application running at the same time, and requesting service from the manager.
The ManagerThe manager process resides on an EC2 node. It checks a special SQS queue for messages from local applications. Once it receives a message it:
- If the message is that of a new task it:
- Downloads the input file from S3.
- Distributes the operations to be performed on the images to the workers using SQS queue/s.
- Checks the SQS message count and starts Worker processes (nodes) accordingly.
- The manager should create a worker for every n messages, if there are no running workers.
- If there are k active workers, and the new job requires m workers, then the manager should create
m-knew workers, if possible.
- Note that while the manager creates a node for every n messages, it does not delegate messages to specific nodes. All of the worker nodes take their messages from the same SQS queue; so it might be the case that with 2n messages, hence two worker nodes, one node processed n+(n/2) messages, while the other processed only n/2.
- After the manger receives response messages from the workers on all the files on an input file, then it:
- Creates a summary output file accordingly,
- Uploads the output file to S3,
- Sends a message to the application with the location of the file.
- If the message is a termination message, then the manager:
- Does not accept any more input files from local applications. However, it does serve the local application that sent the termination message.
- Waits for all the workers to finish their job, and then terminates them.
- Creates response messages for the jobs, if needed.
IMPORTANT: the manager must process requests from local applications simultaneously; meaning, it must not handle each request at a time, but rather work on all requests in parallel.
The WorkersA worker process resides on an EC2 node. Its life cycle is as follows:
- Get a message from an SQS queue.
- Resize the received Image, and upload the resized image to s3.
- Put a message in an SQS queue with the url of the original image, and the url of the resized image.
- remove the processed message from the SQS queue.
- The worker will also save the following statistics in a file on s3:
- Its id.
- Its start time.
- Its average run-time on a single url.
- Total number of urls it handles.
- A list of all the successful urls it handled and their number.
- A list of all the failed urls (urls that caused an exception) together with the exception and their number.
- Its finish time.
- If a worker stops working unexpectedly before finishing its work on a message, then some other worker should be able to handle that message.
The Queues and MessagesAs described above, queues are used for:
- communication between the local application and the manager.
- communication between the manager and the workers.
It is up to you to decide what the jobs and the messages are, and how many queues to use, but your system should run as efficiently as possible in terms of time, memory, and money!
Running the ApplicationThe application should be run as follows:
java -jar yourjar.jar inputFileName outputFileName nor, if you want to terminate the manager:
java -jar yourjar.jar inputFileName outputFileName n terminate
yourjar.jar is the name of the jar file containing your code (do not include the libraries in it when you create it).
inputFileName is the name of the input file.
outputFileName is the name of the output file.
n is: workers - files ratio (how many urls per worker).
- Local Application uploads the file with the list of image urls to S3.
- Local Application sends a message (queue) stating the location of the input file on S3.
- Local Application does one of the following:
- Starts the manager.
- Checks if a manager is active and if not, starts it.
- Manager downloads a list of images.
- Manager distributes jobs to the workers.
- Manager bootstraps nodes to process messages.
- Worker gets a message from an SQS queue.
- Worker performs the requested job/s on the image.
- Worker puts a message in an SQS queue indicating the original image together with the url of the resized image.
- Manager reads all Workers' messages from SQS and creates one summary file.
- Manager uploads the summary file to S3.
- Manager posts an SQS message about the summary file.
- Local Application reads SQS message.
- Local Application downloads the summary file from S3.
- Local Application creates html output file.
- Local application send a terminate message to the manager if it received terminate as one of its arguments.
Which AMI Image to Use?You can choose whatever image you want. You probably want to have one which supports user-data. If you don't want to choose on your own, you can just use this one:
ami-146e2a7cuse the small one.
The AWS SDKThe assignment will be written in Java, you'll need the SDK for Java for it. We advise you to read the Getting Started guide, and get comfortable with the SDK.
SQS Visibility Time-outRead the following regarding SQS timeout, understand it, and use it in your implementation: Visibility Timeout
BootstrappingWhen you create and boot up an EC2 node, it turns on with a specified image, and that's it. You need to load it with software and run the software in order for it to do something useful. We refer to this as "bootstrapping" the machine.
The bootstrapping process should download
.jar files from an S3 bucket and run them. In order to do that, you need a way to pass instructions to a new node. You can do that using this guide, and another guide. User-data allows you to pass a shell-script to the machine, to be run when it starts up. Notice that the script you're passing should be encoded to base64. Here's a code example of how to do that.
If you use an AMI without Java or the AWS SDK for Java, you will need to download and install them via the bootstrap script.
Downloading from the console:
In Linux, the command
wget is usually installed. You can use it to download web files from the shell.
wget http://www.cs.bgu.ac.il/~dsps151/Main -O dsp.html
will download the content at http://www.cs.bgu.ac.il/~dsp151/Main and save it to a file named dsp.html. wget man
Installing from the console:
In Ubuntu (or Debian) Linux, you can use the
apt-get command (assuming you have root access to the machine). Example:
apt-get install wget will install the
wget command if it is not installed. You can use it to install Java, or other packages.
shell scripting with bash: your user-data scripts can be written in any language you want (e.g. Python, Perl, tsch, bash).
bash is a very common choice. Your scripts are going to be very simple. Nonetheless, you might find these bash tutorials useful.
Checking if a Manager Node is ActiveYou can check if the manager is running by listing all your running instances, and checking if one of them is a manager. Use the "tags" feature of Amzon EC2 API to mark a specific instance as one running a Manager:
Resizing ImagesFortunately, you don't have to write an algorithm for resizing images by yourself, but rather use an already implemented and tested one in an existing class called Graphics2D.
Here is an example code.
To work with images, use this library.
- The assignment will be graded in a frontal setting.
- All information mentioned in the assignment description, or learnt in class is mandatory for the assignment.
- You will be reduced points for not reading the relevant reading material, not implementing the recommendations mentioned there, and not understanding them.
- Students belonging to the same group will not necessarily receive the same grade.
- All the requirements in the assignment will be checked, any missing functionality will cause a point reduction. You have the "freedom" to choose how to implement things that are not precisely defined in the assignment.
NotesCloud services are cheap but not free. Even if they were free, waste is bad. Therefore, please keep in mind that:
- It should be possible for you to easily remove all the things you put on S3. You can do that by putting them in a specified bucket under a folder, which you could delete later.
- While it is the Manager's job to turn off all the Worker nodes, do verify yourself that all the nodes really did shut down, and turn of the manger manually if it is still running.
- You won't be able to download the files unless you make them public.
- You may assume there will not be any race conditions; conditions were 2 local applications are trying to start a manger at the same time etc.
A Very Important Note about SecurityAs part of your application, you will have a machine on the cloud contacting an amazon service (SQS and S3). For the communication to be successful, the machine will have to supply the service with a security credentials(password) to authenticate. Security credentials is sensitive data – if someone gets it, they can use it to use amazon services for free (from your budget). You need to take good care to store the credentials in a secure manner. One way of doing that is by compressing the jar files with a password. Remember, your credit card is listed under your amazon account; so if you pass the 100$ credit your card will be charged. Yes, most of the students who had that issue were refunded, but why go there in the first place?! Save your credentials only on your computer, and don't allow any external access to it.
Use the submission system to submit a zip file that contains:
- all sources and binaries, without the libraries that you're supposed to download;
- the output of running your system on this input file;
- the statistics file of each one of the workers;
- a text file called README with instructions on how to run your project, and an explanation of how your program works. It will help you remember your implementation. Your README must also contain what type of instance you used (ami + type:micro/small/large ), how much time it took your program to finish working on the above input file, and what was the n you used.