hadoop-aws-*.jar library is not in classpath by default, but it exists in $HADOOP_HOME/tools/lib folder. To fix it, edit $HADOOP_HOME/etc/hadoop/hadoop-env.sh and add the following line:
|
1 |
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$HADOOP_HOME/share/hadoop/tools/lib/* |
Zookeeper installation steps are nice and easy for dev environment.
How to start HBase client in AWS EMR and query external HBase DB
|
1 2 3 4 5 6 7 8 9 |
aws emr create-cluster \ --release-label emr-4.6.0 \ --name "my-hbase-cluster" \ --instance-type r3.2xlarge \ --instance-count 2 \ --enable-debugging \ --ec2-attributes KeyName=MyKeyPair \ --use-default-roles \ --applications Name=Hadoop Name=HBase |
2. Establish ssh connection to the cluster
|
1 |
ssh -i ~/MyKeyPair.pem hadoop@<cluster-ip-address>.us-west-2.compute.amazonaws.com |
3.To work with external database, set zookeeper quorum in /etc/hbase/conf/hbase-site.xml
|
1 2 3 4 5 6 7 |
<configuration> <property> <name>hbase.zookeeper.quorum</name> <value>my-hbase-zookeeper-address</value> </property> .... </configuration> |
3. Start HBase shell
|
1 |
hbase shell |
4. In shell do queries like that:
|
1 2 3 4 5 6 7 8 9 10 11 |
create 'person', {NAME=>'name'}, {NAME=>'addr'} put 'person', '1', 'name:firstName', 'John' put 'person', '1', 'name:lastName', 'Smith' put 'person', '1', 'addr:planet', 'Earth' put 'person', '1', 'addr:continent', 'Australia' list describe 'person' scan 'person' get 'person', '1', {COLUMNS => ['name']} get 'person', '1', {COLUMNS => ['addr:planet']} |
I got an issue with s3-dist-cp command on Spark AWS EMR 4.5 cluster.
The issue: s3-dist-cp command step fails with error: java.lang.RuntimeException: java.io.IOException: Cannot run program “s3-dist-cp” (in directory “.”): error=2, No such file or directory Continue reading
Create Spark cluster and run custom jar on it. Continue reading
Tried to open Lambda page in AWS web console: https://us-west-2.console.aws.amazon.com/lambda/home?region=us-west-2#
.. and got a message: “You must have permission to perform lambda:ListFunctions to access the AWS Lambda console.” Continue reading
Here is Amazon sign in form with 3 fields: Account, User name, Password. I always forget it and decided to store it here to not search it again in internet every time.
https://12345678.signin.aws.amazon.com/console
Usually Amazon offers another log in form with 2 fields, which does not suit me.
How to install and configure Ganglia monitoring tool on a cluster.
Prerequisites:
Deployment plan:
My bash scripts to clear Storm and Zookeeper cluster remotely by ssh.
Connect to every zookeeper server by ssh and stop zookeeper, then delete data folder. Then connect to every Storm node by ssh, kill Storm processes and delete data folder. Connect to zookeeper servers again, start them. Connect to Storm nodes again, start them. Continue reading
Symptoms: when trying to run a Storm supervisor node it is not listed in cluster. We can see the following error in log:
Error when processing event java.lang.RuntimeException: java.io.EOFException at backtype.storm.utils.Utils.deserialize(Utils.java:69) ~[storm-core-0.9.0.1.jar:na]
Solution worked for me: Stop zookeeper and supervisor, clean data dirs. This will probably not suite the production because of data lost.
