Tag Archives: Hadoop

File could only be replicated to 0 nodes instead of minReplication (=1)

Working on newly configured cluster. Can browse HDFS but any write attempt produces an error:

WARN  [Thread-6] hdfs.DFSClient (DFSOutputStream.java:run(557)) - DataStreamer Exception
org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /tmp/hive/index.html._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1).  There are 1 datanode(s) running and no node(s) are excluded in this operation.

WARN [Thread-6] hdfs.DFSClient (DFSOutputStream.java:run(557)) - DataStreamer Exception

org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /tmp/hive/index.html._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1). There are 1 datanode(s) running and no node(s) are excluded in this operation.

Also I can create and view tables in Hive, but any insert attempt fails with the same error. Continue reading →

Useful Hadoop ports to remember

Leave a reply

Short memo of links and commands to monitor Hadoop and his friends on default ports.

Hadoop name node: http://localhost:50070

Yarn resource manager: http://localhost:8088

Hive web interface: http://localhost:10002

Beeline: beeline -u jdbc:hive2://localhost:10000

Accessing AWS s3 from on premises Hadoop

Leave a reply

Add aws libraries to class path

hadoop-aws-*.jar library is not in classpath by default, but it exists in $HADOOP_HOME/tools/lib folder. To fix it, edit $HADOOP_HOME/etc/hadoop/hadoop-env.sh and add the following line:

export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$HADOOP_HOME/share/hadoop/tools/lib/*

1	export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$HADOOP_HOME/share/hadoop/tools/lib/*

Continue reading →

start-dfs.sh: Permission denied, please try again

Leave a reply

Issue. Starting Hadoop HDFS with command start-dfs.sh. Entering correct password, but getting permission denied: Continue reading →

How to stop Yarn application

Leave a reply

Ssh to master node, print applicaton list and find your app:

yarn application -list

1	yarn application -list

Kill your application using app id:

yarn application -kill <My application id>

1	yarn application -kill <My application id>

s3-dist-cp is missing in EMR 4

Leave a reply

I got an issue with s3-dist-cp command on Spark AWS EMR 4.5 cluster.

The issue: s3-dist-cp command step fails with error: java.lang.RuntimeException: java.io.IOException: Cannot run program “s3-dist-cp” (in directory “.”): error=2, No such file or directory Continue reading →

Install Hadoop on Ubuntu

1 Reply

The best article of Hadoop installation I found is http://codesfusion.blogspot.ru/2013/10/setup-hadoop-2x-220-on-ubuntu.html

I adopted it a little for my configuration: Ubuntu 14.04, Hadoop 2.2.0 single node, JDK 6. Not latest versions of Java and Hadoop but I had to reproduce existing production system.

My steps are:

Install prerequisites

JDK 6 and ssh:

$ sudo apt-get install oracle-java6-installer $ sudo apt-get install python-software-properties $ sudo add-apt-repository ppa:webupd8team/java $ sudo apt-get update $ sudo apt-get install oracle-java6-installer $ sudo apt-get install ssh

Continue reading →

Dmitry Pukhov

Software Developer, Architect

Tag Archives: Hadoop

File could only be replicated to 0 nodes instead of minReplication (=1)

Useful Hadoop ports to remember

Accessing AWS s3 from on premises Hadoop

Add aws libraries to class path

start-dfs.sh: Permission denied, please try again

How to stop Yarn application

s3-dist-cp is missing in EMR 4

Install Hadoop on Ubuntu

Install prerequisites