There are many interfaces to HDFS available, but the command line (CLI) is one of the simplest and, to many developers, the most familiar interface. You can perform most advanced and basic Hadoop HDFS filesystem operations using CLI.
Basic Hadoop HDFS Filesystem Operations
The when Hadoop HDFS filesystem is set, you can do all of the basic HDFS filesystem operations, such as reading files, creating directories, moving files, deleting data, and listing directories. You can also perform the advance Hadoop HDFS filesystem operations such as updates, administrator from command line.
There are lot of books that provide you information on the HDFS file systems in the Hadoop ecosystem. You may want to take look at best Hadoop books:
You can type hadoop fs -help to get detailed help on every available command in Hadoop filesystem.
Related reading:
Basic Hadoop HDFS Filesystem Operations Examples
Create HDFS direcory:
Create hdfs directory and lists using below commands
$ hdfs dfs -mkdir /data $ hdfs dfs -ls / Found 7 items drwxrwxrwx - hdfs supergroup 0 2016-08-10 14:35 /benchmarks drwxr-xr-x - cloudera supergroup 0 2016-10-30 23:49 /data drwxr-xr-x - hbase supergroup 0 2016-10-16 07:40 /hbase drwxr-xr-x - solr solr 0 2016-08-10 14:37 /solr drwxrwxrwt - hdfs supergroup 0 2016-09-02 22:36 /tmp drwxr-xr-x - hdfs supergroup 0 2016-08-10 14:37 /user drwxr-xr-x - hdfs supergroup 0 2016-08-10 14:37 /var $
Copy file from the local filesystem to HDFS:
Transfer and store a data file from local systems to the Hadoop file system using the put and copyFromLocal command.
$ hdfs dfs -copyFromLocal test.dat hdfs://localhost/data $ hdfs dfs -ls /data Found 1 items -rw-r--r-- 1 cloudera supergroup 60 2016-10-31 00:01 /data/test.dat $ $ hdfs dfs -put test123.dat hdfs://localhost/data $ hdfs dfs -ls /data Found 2 items -rw-r--r-- 1 cloudera supergroup 60 2016-10-31 00:01 /data/test.dat -rw-r--r-- 1 cloudera supergroup 60 2016-10-31 00:03 /data/test123.dat $
Copy file from the HDFS filesystem to local using copyToLocal and get commands:
$ hdfs dfs -copyToLocal hdfs://localhost/data/test.dat /home/cloudera/ $ ls -ltr /home/cloudera/test.dat -rw-r--r-- 1 cloudera cloudera 60 Oct 31 00:09 /home/cloudera/test.dat $ $ hdfs dfs -get hdfs://localhost/data/test123.dat /home/cloudera/ $ ls -ltr test123.dat -rw-r--r-- 1 cloudera cloudera 60 Oct 31 00:12 test123.dat $
View HDFS file content using cat command:
$ hdfs dfs -cat /data/test.dat asdasd,asdas,dasdasd asdasf,dgdsg,fhfdhe sfsdfa,afdsd,dfsfd $
List the files in the root directory of the local filesystem:
You can use -ls command to list the files in the root directory of local file system. Below is the example of usage of command:
$ hadoop fs -ls file:/// Found 22 items -rw-r--r-- 1 root root 0 2016-10-16 07:27 file:///.autofsck -rw-r--r-- 1 root root 0 2016-08-10 14:03 file:///.autorelabel dr-xr-xr-x - root root 4096 2016-10-16 07:19 file:///bin dr-xr-xr-x - root root 1024 2016-08-10 21:41 file:///boot drwxr-xr-x - root root 3660 2016-10-16 07:30 file:///dev drwxr-xr-x - root root 12288 2016-10-30 23:20 file:///etc drwxrwxr-x - root root 4096 2016-08-10 19:52 file:///home dr-xr-xr-x - root root 4096 2016-08-10 14:06 file:///lib dr-xr-xr-x - root root 12288 2016-10-16 07:19 file:///lib64 drwx------ - root root 16384 2016-08-10 14:00 file:///lost+found drwxr-xr-x - root root 4096 2011-09-23 04:50 file:///media drwxr-xr-x - root root 4096 2016-08-10 21:41 file:///mnt $