How to copy multiple files from localhost to vagrant node

I have a 3 node hadoop cluster installed via vagrant in my laptop. Yesterday i wanted to move a set of files to the vagrant node from my local host, here are the steps i followed to achieve it.

firstly get the vagrant node ssh configuration, since i have it installed on the local machine i dont have the password/SSH keys set up. You will find much more access restrictions on a professional setup.

First check the options in vagrant help

>> vagrant help

you can see an option “ssh-config” along with a brief description.

ssh-config outputs OpenSSH valid configuration to connect to the machine

Now, list the ssh configuration for my nodes

>>vagrant ssh-config node1

Host node1
 User vagrant
 Port 2222
 UserKnownHostsFile /dev/null
 StrictHostKeyChecking no
 PasswordAuthentication no
 IdentityFile /opt/dev/hdp/hadoop-ops-course-master/.vagrant/machines/node1/virtualbox/private_key
 IdentitiesOnly yes
 LogLevel FATAL

Once we have the configuration, use the below scp command to move the file, please note you have to change the parameters based on your environment.

>> scp -P 2222 -i /opt/dev/hdp/hadoop-ops-course-master/.vagrant/machines/node1/virtualbox/private_key /home/dmadavil/Downloads/names/*.txt vagrant@

This move files from my local machine path to vagrant node1 home directory.

You may need to run the hadoop fs -put or CopyFromLocal to move the file to HDFS.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s