I have a 3 node hadoop cluster installed via vagrant in my laptop. Yesterday i wanted to move a set of files to the vagrant node from my local host, here are the steps i followed to achieve it.
firstly get the vagrant node ssh configuration, since i have it installed on the local machine i dont have the password/SSH keys set up. You will find much more access restrictions on a professional setup.
First check the options in vagrant help
>> vagrant help
you can see an option “ssh-config” along with a brief description.
ssh-config outputs OpenSSH valid configuration to connect to the machine
Now, list the ssh configuration for my nodes
>>vagrant ssh-config node1 Host node1 HostName 127.0.0.1 User vagrant Port 2222 UserKnownHostsFile /dev/null StrictHostKeyChecking no PasswordAuthentication no IdentityFile /opt/dev/hdp/hadoop-ops-course-master/.vagrant/machines/node1/virtualbox/private_key IdentitiesOnly yes LogLevel FATAL
Once we have the configuration, use the below scp command to move the file, please note you have to change the parameters based on your environment.
>> scp -P 2222 -i /opt/dev/hdp/hadoop-ops-course-master/.vagrant/machines/node1/virtualbox/private_key /home/dmadavil/Downloads/names/*.txt email@example.com:~
This move files from my local machine path to vagrant node1 home directory.
You may need to run the hadoop fs -put or CopyFromLocal to move the file to HDFS.