For single machine deployment target directory can be on local file system (use file: instead of hdfs:)
|
1 |
hbase org.apache.hadoop.hbase.mapreduce.Export 'myns1:mytable1' 'hdfs://myserver1:9000/mytable1directory' |
|
1 |
hbase org.apache.hadoop.hbase.mapreduce.Import 'myns1:mytable1' 'hdfs://myserver1:9000/mytable1directory' |
Working on newly configured cluster. Can browse HDFS but any write attempt produces an error:
|
1 2 |
WARN [Thread-6] hdfs.DFSClient (DFSOutputStream.java:run(557)) - DataStreamer Exception org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /tmp/hive/index.html._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1). There are 1 datanode(s) running and no node(s) are excluded in this operation. |
Also I can create and view tables in Hive, but any insert attempt fails with the same error. Continue reading
Short memo of links and commands to monitor Hadoop and his friends on default ports.
Hadoop name node: http://localhost:50070
Yarn resource manager: http://localhost:8088
Hive web interface: http://localhost:10002
Beeline: beeline -u jdbc:hive2://localhost:10000
By default, hive runs with embedded derby metastore, which allows only one connection. This article is about how to hive with derby network server. Assume hive is installed to /opt/hive folder
|
1 2 3 4 5 6 7 8 9 |
<property> <name>javax.jdo.option.ConnectionURL</name> <value>jdbc:derby://localhost:1527/metastore_db;create=true</value> </property> <property> <name>javax.jdo.option.ConnectionDriverName</name> <value>org.apache.derby.jdbc.ClientDriver</value> <description>Driver class name for a JDBC metastore. EmbeddedDriver or ClientDriver</description> </property> |
|
1 |
nohup /opt/derby/bin/startNetworkServer & |
|
1 |
/opt/hive/bin/schematool -initSchema -dbType derby |
|
1 |
nohup /opt/hive/bin/hiveserver2 & |
|
1 |
/opt/hive/bin/beeline -u jdbc:hive2://localhost:10000 |
Recently installed Hive 2.1.1 on Ubuntu 16.04. Maybe not optimal steps, but it worked for me.
1. Download Hive from https://hive.apache.org/downloads.html, unpack archive to folder /opt/hive
2. Copy /opt/hive/conf/hive-default.xml.template to hive-site.xml
Edit hive-site.xml, replace all occurences of ${system:java.io.tmpdir}/${system:user.name} with /tmp/hive Continue reading
hadoop-aws-*.jar library is not in classpath by default, but it exists in $HADOOP_HOME/tools/lib folder. To fix it, edit $HADOOP_HOME/etc/hadoop/hadoop-env.sh and add the following line:
|
1 |
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$HADOOP_HOME/share/hadoop/tools/lib/* |
Notes how to install and use Aerospike.
Installation steps from here: http://www.aerospike.com Continue reading
Issue. Starting Hadoop HDFS with command start-dfs.sh. Entering correct password, but getting permission denied: Continue reading
Ssh to master node, print applicaton list and find your app:
|
1 |
yarn application -list |
Kill your application using app id:
|
1 |
yarn application -kill <My application id> |
Spark 1.6.1, HBase 1.2.1, run on EMR 4.7.1
Spark and HBase installation:
http://dmitrypukhov.pro/install-apache-spark-on-ubuntu/,
http://dmitrypukhov.pro/install-hbase-on-linux-dev/
Edit spark-defaults.conf and ensure spark.driver.extraClassPath and spark.executor.extraClassPath contain path to hbase libraries. For me it is /usr/lib/hbase/lib/*
My extra class pathes: Continue reading
