Explain Common Hadoop Shell Commands

1 week ago

1. Create a directory in HDFS at given path(s).

Usage:

hadoop fs -mkdir <paths> Example:

hadoop fs -mkdir /user/saurzcode/dir1 /user/saurzcode/dir2

2. List the contents of a directory.

Usage :

hadoop fs -ls <args> Example:

hadoop fs -ls /user/saurzcode

3. Upload and download a file in HDFS. 

hadoop fs -put: Copy single src file, or multiple src files from local file system to the Hadoop data file system

Usage:

hadoop fs -put <localsrc> ... <HDFS_dest_Path> Example:

hadoop fs -put /home/saurzcode/Samplefile.txt /user/saurzcode/dir3/

Download: 

hadoop fs -get: Copies/Downloads files to the local file system

Usage:

hadoop fs -get <hdfs_src> <localdst> Example:

hadoop fs -get /user/saurzcode/dir3/Samplefile.txt /home/

4. See contents of a file

Same as unix cat command:

Usage:

hadoop fs -cat <path[filename]> Example:

hadoop fs -cat /user/saurzcode/dir1/abc.txt

5. Copy a file from source to destination

This command allows multiple sources as well in which case the destination must be a directory.

Usage:

hadoop fs -cp <source> <dest> Example:

hadoop fs -cp /user/saurzcode/dir1/abc.txt /user/saurzcode/dir2

6. Copy a file from/To Local file system to HDFS

copyFromLocal

Usage:

hadoop fs -copyFromLocal <localsrc> URI Example:

hadoop fs -copyFromLocal /home/saurzcode/abc.txt /user/saurzcode/abc.txt

Similar to put command, except that the source is restricted to a local file reference. copyToLocal

Usage:

hadoop fs -copyToLocal [-ignorecrc] [-crc] URI <localdst>

Similar to get command, except that the destination is restricted to a local file reference.

7. Move file from source to destination.

Note: - Moving files across file system is not permitted.

Usage :

hadoop fs -mv <src> <dest> Example:

hadoop fs -mv /user/saurzcode/dir1/abc.txt /user/saurzcode/dir2

8.  Remove a file or directory in HDFS.

Remove files specified as argument. Deletes directory only when it is empty

Usage :

hadoop fs -rm <arg> Example:

hadoop fs -rm /user/saurzcode/dir1/abc.txt

Recursive version of delete.

Usage :

hadoop fs -rmr <arg>

Example:

hadoop fs -rmr /user/saurzcode/

9.  Display last few lines of a file.

Similar to tail command in Unix.

Usage :

hadoop fs -tail <path[filename]> Example:

hadoop fs -tail /user/saurzcode/dir1/abc.txt

10. Display the aggregate length of a file.

Usage :

hadoop fs -du <path> Example:

hadoop fs -du /user/saurzcode/dir1/abc.txt

Raju Singhaniya
Oct 14, 2021
More related questions

Questions Bank

View all Questions

Top