I got an issue with s3-dist-cp command on Spark AWS EMR 4.5 cluster.
The issue: s3-dist-cp command step fails with error: java.lang.RuntimeException: java.io.IOException: Cannot run program “s3-dist-cp” (in directory “.”): error=2, No such file or directory Continue reading
Create Spark cluster and run custom jar on it. Continue reading
Using scp command for secure copying file to remote machine:
|
1 |
scp <file> <user>@<machine>:<path on remote machine> |
Example:
|
1 |
scp -i ~/.ssh/myKey.pem ~/file1.txt ubuntu@192.168.0.2:"~/file2.txt" |
Filter processes by string pattern (let it be “java” processes):
|
1 |
ps -eaf | grep java |
Find my process in resulted list and kill by id:
|
1 |
kill -9 123 |
Kill all processes by “java” pattern:
|
1 |
kill -9 `ps -eaf | grep java | grep -v grep | awk '{print$2}'` |
I used s3cmd cp and s3cmd sync commands to copy files from source s3 folder to s3 destination. Copying is too slow, I see many attempts, failed by timeout, like this
|
1 2 |
WARNING: Retrying failed request: ... (timeout) WARNING: Waiting 3 sec... |
In ~/.s3cfg file I found socket_timeout setting, which is 100 by default. Setting it to 1000 helped me:
|
1 |
socket_timeout = 1000 |
Starting Tomcat7 from Eclipse produced an error:
Could not load the Tomcat server configuration at /usr/share/tomcat7/conf. The configuration may be corrupt or incomplete.
Solution:
Tomcat7 split into different folders by default. Eclipse expects Tomcat to be in one folder. To resolve, we create symbolic links in Tomcat home dir.
|
1 2 3 4 5 6 |
cd /usr/share/tomcat7 sudo ln -s /var/lib/tomcat7/conf conf sudo ln -s /var/log/tomcat7 log sudo ln -s /etc/tomcat7/policy.d/03catalina.policy conf/catalina.policy sudo chmod -R a+rwx /usr/share/tomcat7/conf /etc/init.d/tomcat7 stop |
Now Eclipse should be able to start Tomcat.
Starting a simple Spark project in IntelliJ Idea and getting an exception:
Exception in thread “main” java.lang.NoClassDefFoundError: org/apache/spark/SparkConf
at …
Caused by: java.lang.ClassNotFoundException: org.apache.spark.SparkConf
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
… 2 more
Solution:
Change Spark dependencies scope from provided to compile in pom.xml
|
1 2 3 4 5 6 |
<dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-core_2.10</artifactId> <version>1.5.2</version> <scope>compile</scope> </dependency> |
Simple steps to run Spark 1.5.2 on Ubuntu 14.04 with Yarn (Hadoop 2.7.1)
Install Hadoop (see http://dmitrypukhov.pro/install-hadoop-on-ubuntu/)
Download pre-build Spark from http://spark.apache.org/downloads.html and unpack, say to /opt/spark/ Continue reading
I have my data in parquet format and want to load and query it using Spark SQL.
Start Spark shell
|
1 |
spark-shell |
Load parquet folder to table
|
1 2 |
val myDataFrame=sqlContext.load("s3://my-bucket/my-parquet-folder/") myDataFrame.registerTempTable("myTable") |
Now we can use this table for SQL queries:
|
1 |
sqlContext.sql("select * from myTable").first() |
How to package the project with mvn command when external repos are not available, but we are happy to get dependencies from local repositories:
|
1 |
mvn clean -o package |
