I was working on a simple project on how to manage my Amazon EC2 and S3 buckets from my local machine and decided to document how to achieve this using the Amazon EMR CLI for Windows. There are several tools and options how there that you can also use to manage your Amazon EC2 and S3 buckets, for instance Windows PowerShell.
For the purpose of this post, I will focus mainly on Amazon EMR CLI.
1. Install Ruby
– Download Ruby 1.8.7 patch 358 (Note: Amazon EMR CLI is not compatible with later versions of Ruby)
– Double-click the downloaded exe file → Accept the License and click Next
– Ensure both ‘Add Ruby executables to your PATH’ and ‘Associate .rb files and .rbw files with this Ruby installation’ are checked. Click Install
– Click Finish when installation is completed
– Verify both ‘Ruby version’ & ‘RubyGems package’ using the command lines as shown below respectively
2. Install Amazon EMR CLI
– Create a new folder anywhere on your machine (Preferably root drive)
– Download the Amazon EMR zip file and extract content to the newly created folder
3. Configure EMR Credentials
– Simply create a JSON file called ‘credentials.json’ in the newly created folder where the EMR CLI files where extracted into
– Open the file ‘credentials.json’ using a text editor and paste the following into it:
[sourcecode language=”plain”]
{
“access_id”: “Your AWS Access Key ID”,
“private_key”: “Your AWS Secret Access Key”,
“keypair”: “Your key pair name”,
“key-pair-file”: “The path and name of your PEM file”,
“log_uri”: “A path to a bucket you own on Amazon S3, such as, s3n:///”,
“region”: “The region of your cluster, either us-east-1, us-west-2, us-west-1, eu-west-1, ap-northeast-1, ap-southeast-1, ap-southeast-2, or sa-east-1”
}
[/sourcecode]
Note: – all values must be inside the double quotation mark ” “
– access_id & private_key: This should be available to you during your account setup
– keypair & key-pair-file: You need to have a Key Pair setup to enable secure access to your EC2 instances. Note the region where you create the key pairs as this is what you will add in the ‘region’ line. See Create Amazon Web Service EC2 Key Pair for details
4. Setup SSH configuration using Putty
– You need to configure your SSH credentials for use with Putty. See Setup EC2 instance Private Key using Putty for details
– Open command prompt → change directory to the folder that was earlier created where the EMR files where installed → run the command ‘ruby elastic-mapreduce –version‘ to verify Amazon EMR CLI is working
5. Command Line options
– Type the following command line to see available options ‘ruby elastic-mapreduce options‘
Now you are ready to start work on your Elastic MapReduce Job Flows. This will be covered in next post
