My bash scripts to clear Storm and Zookeeper cluster remotely by ssh.
Main idea:
Connect to every zookeeper server by ssh and stop zookeeper, then delete data folder. Then connect to every Storm node by ssh, kill Storm processes and delete data folder. Connect to zookeeper servers again, start them. Connect to Storm nodes again, start them.
cleanCluster.sh – main script
cleanCluster.sh is a main script to run. zkClean.sh and stormClean.sh are dependants, they are called from cleanCluster.sh
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 |
#!/bin/bash ############################## # Zookeeper servers # Set zkServers array to zookeeper machines ############################## zkServers[0]=192.168.0.1 zkServers[1]=192.168.0.2 zkServers[2]=192.168.0.3 ####################################### # Storm servers: nimbus and supervisors # Set stormServers array to storm machines ####################################### # Big cluster storm servers # Nimbus stormServers[0]=192.168.1.1 # Supervisors stormServers[1]=192.168.1.2 stormServers[2]=192.168.1.3 stormServers[3]=192.168.1.4 stormServers[4]=192.168.1.5 stormServers[5]=192.168.1.6 # Single server script ZK_SCRIPT=zkClean.sh STORM_SCRIPT=stormClean.sh KEY_FILE=inrix-tapp.pem echo "Configuration:" echo "Zookeeper servers number: ${#zkServers[@]}" for server in "${zkServers[@]}" do echo $server done echo "Storm servers number: ${#stormServers[@]}" for server in "${stormServers[@]}" do echo $server done echo "Cleaning zookeeper servers" for server in "${zkServers[@]}" do echo "Processing zookeeper $server" ssh -oStrictHostKeyChecking=no -i $KEY_FILE ubuntu@$server "sh " < $ZK_SCRIPT done echo "Cleaning storm servers" for server in "${stormServers[@]}" do echo "Processing storm $server" ssh -oStrictHostKeyChecking=no -i $KEY_FILE ubuntu@$server "sh " < $STORM_SCRIPT done echo "Rebooting servers" for server in "${zkServers[@]}" do echo "Rebooting zookeeper server $server" ssh -oStrictHostKeyChecking=no -i $KEY_FILE ubuntu@$server "sudo reboot" done for server in "${stormServers[@]}" do echo "Rebooting storm server $server" ssh -oStrictHostKeyChecking=no -i $KEY_FILE ubuntu@$server "sudo reboot" done |
zkClean.sh – dependent script
|
1 2 3 4 5 6 7 8 9 10 |
#!/bin/bash ZK_DIR=/opt/zookeeper ZK_DATA=/opt/zookeeper/data echo "Stopping zookeeper" sudo $ZK_DIR/bin/zkServer.sh stop echo "Removing $ZK_DATA" mv $ZK_DATA/myid ~/myid sudo rm -R $ZK_DATA/* mv ~/myid $ZK_DATA/myid |
stormClean.sh – dependent script
|
1 2 3 4 5 6 7 8 |
#!/bin/bash STORM_DIR=/opt/storm STORM_DATA=/opt/storm/storm-local echo "Stopping storm processes" sudo pkill -9 java echo "Removing $STORM_DATA" sudo rm -R $STORM_DATA/* |
