Apache Storm cluster on Amazon EC2

How to automate Apache Storm cluster creation on Amazon EC2.

Install lein

Install Leiningen 2 (a tool to handle Clojure projects):

$ wget https://raw.github.com/technomancy/leiningen/stable/bin/lein
$ mv lein /usr/local/bin/
$ chmod +x /usr/local/bin/lein
$ lein

Download storm-deploy

Clone storm-deploy and download dependencies

git clone https://github.com/nathanmarz/storm-deploy.git
cd storm-deploy
lein deps

Configure storm-deploy for our cluster

Create ~/.pallet/config.clj file with the following content:

(defpallet
:services
{
:default {
:blobstore-provider “aws-s3”
:provider “aws-ec2”
:environment {:user {:username “storm”
:private-key-path “PRIVATE_KEY_PATH”
:public-key-path “PUBLIC_KEY_PATH”}
:aws-user-id “AWS_ACCOUNT_ID”}
:identity “AWS_ACCESS_KEY”
:credential “AWS_ACCESS_KEY_SECRET”
:jclouds.regions “AWS_REGION”
}
})

Change cluster configuration if you want: ~/storm-deploy/conf/clusters.yaml

Start/stop the cluster

Go to ~/storm-deploy folder and start cluster:

$ lein deploy-storm --start --name CLUSTER_NAME [--branch BRANCH_ID] [--commit COMMIT_ID]

Stop:

$ lein deploy-storm --stop --name CLUSTER_NAME

Troubleshooting

Problems I faced when starting:

Problem: error “Security group does not exist”
Solution: double check :jclouds-regions value in ~/.pallet/config.clj

Problem: error “could not  initialize class com.jcraft.jsch.agentproxy.usocket.JNAUSocketFactory$CLibrary
Solution: 
$ cd /usr/share/ant/lib

$ sudo rm ant-jsch.jar
$ sudo ln -s ../../java/jsch.jar ant-jsch.jar

Problem: error: failed to create temporary file for jnidispatch library: java.io.IOException: Permission denied
Solution:
$ sudo chmod 777 /tmp/jna

Problem: connection timeout in logs
Solution: in AWS dashboard edit security group. Add custom TCP rules in inbound rules for ports: 8080, 6627, 8000, 2181, 6699-6800

 

Leave a Reply

Your email address will not be published.