HDFS important and useful commands

-

To check Hadoop Version

$ hadoop version

Creating user home directory

hadoop fs -mkdir -p /user/dpq  (-p will create directory if directory not present if directory already present then it wont throw exception)

hadoop fs -mkdir  /user/retails (without -p  if directory already present then it will throw exception)

List all directories

hdfs dfs -ls /

hdfs dfs -ls /user

Copy file from local to HDFS

hadoop fs -copyFromLocal /etc/data/retial_data.csv /user/retailshadoop fs -put /etc/data/retial_data.csv /user/retails (if 'retail_data.csv' already present inside '/user/retails' hdfs directory then it will throw excpetion 'File already present')hadoop fs -put -f /etc/data/retial_data.csv /user/retails (-f will forcefully override retail_data.csv if already present)

Checking data in HDFS

hdfs dfs -cat /user/retails/retial_data.csv

Create empty file on HDFS

hdfs dfs -touchz /user/retails/empty.csv

Copy file from HDFS

hdfs dfs -get /user/retails/retial_data.csv

hdfs dfs -get /user/retails

Copy files in HDFS from one location to another location

hdfs dfs -cp  <src(on hdfs)>  <dest(on hdfs)>hdfs dfs -cp /user/retails/empty.csv /user/dpq/

To rename file or location

hdfs dfs -mv <src(on hdfs)> <src(on hdfs)>
hdfs dfs -mv /user/retails/empty.csv /user/dpq/emp.csv (changing file name)
hdfs dfs -mv /user/retails/ /user/stocks/ (changing hdfs location now emp.csv will be present under stocks hdfs directory)

To remove file or directory

hdfs dfs -rmr /user/stocks/emp.csv (it will remove emp.csv file)

hdfs dfs -rmr /user/stocks (it will remove stocks)

du: It will give the size of each file in directory

hdfs dfs -du /geeks

dus:: This command will give the total size of directory/file

hdfs dfs -dus /geeks

stat: It will give the last modified time of directory or path. In short it will give stats of the directory or file.

hdfs dfs -stat /geeks

setrep: This command is used to change the replication factor of a file/directory in HDFS. By default it is 3 for anything which is stored in HDFS (as set in hdfs core-site.xml).

Example 1: To change the replication factor to 6 for geeks.txt stored in HDFS.

hdfs dfs -setrep -R -w 6 geeks.txt

Example 2: To change the replication factor to 4 for a directory geeksInput stored in HDFS.

hdfs dfs -setrep -R  4 /geeks

Note: The -w means wait till the replication is completed. And -R means recursively, we use it for directories as they may also contain many files and folders inside them.

Note: There are more commands in HDFS but we discussed the commands which are commonly used when working with Hadoop. You can check out the list of dfs commands using the following command:

hdfs dfs

Few more commands:

hdfs dfs -ls /data/dpq//dummy/
hdfs dfs -mkdir -p /data/dpq/dummy/temp
hdfs dfs -put /data/sparkSetup/files/*.* /data/dpq/dummy/temp/
hdfs dfs -ls /data/dpq/dummy/temp/ –default replication will be 3
hdfs dfs -rm /data/dpq/dummy/temp/GDGA111_1_20180918111340752.csv — it will trash

To change replication factor below command will be used while storing data to HDFS
hdfs dfs -Ddfs.replication=1 -put /data/dpq/dummy/temp/GDGA111_1_20180918111340752.csv /data/dpq/temp

hdfs dfs -get /data/dpq/temp/GDGA111_1_20180918111340752.csv
hdfs dfs -cat /data/dpq/temp/GDGA111_1_20180918111340752.ctrl

hdfs dfs -du -h /data/dpq/temp/ — size with replication can be checked here

hdfs dfs -chmod 777 /data/dpq/temp/GDGA111_1_20180918111340752.csv

hdfs dfs -mkdir -p /data/dpq/model_temp
hdfs dfs -ls /data/dpq/
hdfs dfs -chmod 777 /data/dpq/model_temp
hdfs dfs -ls /data/dpq/

hdfs dfs -touchz /data/dpq/model_temp/1.txt –create empty file

Popular posts from this blog

Window function in PySpark with Joins example using 2 Dataframes (inner join)

-
Input details: #● File has json records #● Each record has fields: #○ user_id #○ card_num #○ merchant #○ category #○ amount #○ ts ### Below analysis to be done Sample data: +------+--------+---------+--------+----------+-------+|amount|card_num| category|merchant| ts|user_id|+------+--------+---------+--------+----------+-------+| 243| C_108| food| M_102|1579532902| U_104|| 699| C_106|cosmetics| M_103|1581759040| U_103|| 228| C_104| children| M_110|1584161986| U_103| Solution: from pyspark.sql.functions import col # this can be done without using window function creditCardData = spark.read.json(“card_transactions.json”) useridMaxSpendDF = df.groupby(‘user_id’).max(‘amount’) useridMaxSpendDF=useridMaxSpendDF.withColumnRenamed(“max(amount)”,”max_amount”) useridMaxSpendDF=useridMaxSpendDF.withColumnRenamed(“user_id”,”m_user_id”) cond = [creditCardData.user_id == useridMaxSpen...
Read more

Complex SQL: fetch the users who logged in consecutively 3 or more times (lead perfect example)

-
From the login_details table, fetch the users who logged in consecutively 3 or more times. Table Name : LOGIN_DETAILS Approach : We need to fetch users who have appeared 3 or more times consecutively in login details table. There is a window function which can be used to fetch data from the following record. Use that window function to compare the user name in current row with user name in the next row and in the row following the next row. If it matches then fetch those records. –Table Structure: drop table login_details; create table login_details( login_id int primary key, user_name varchar(50) not null, login_date date); delete from login_details; insert into login_details values (101, ‘Michael’, current_date), (102, ‘James’, current_date), (103, ‘Stewart’, current_date+1), (104, ‘Stewart’, current_date+1), (105, ‘Stewart’, current_date+1), (106, ‘Michael’, current_date+2), (107, ‘Michael’, cur...
Read more

Credit Card Data Analysis using PySpark (how to use auto broadcast join after disabling it)

-
Input details: #● File has json records #● Each record has fields: #○ user_id #○ card_num #○ merchant #○ category #○ amount #○ ts ### Below analysis to be done Sample data: +------+--------+---------+--------+----------+-------+|amount|card_num| category|merchant| ts|user_id|+------+--------+---------+--------+----------+-------+| 243| C_108| food| M_102|1579532902| U_104|| 699| C_106|cosmetics| M_103|1581759040| U_103|| 228| C_104| children| M_110|1584161986| U_103| Application: Here we will disable auto broadcast and then join 2 data frames and will see execution plan which should use broadcast join Solution: from pyspark.sql import SparkSession from pyspark.sql.functions import broadcast spark = SparkSession.builder.master(‘local[2]’)\ .appName(‘RDD_Methods_Examples’)\ .getOrCreate() print(spark.conf.get(“spark.sql.autoBroadcastJoinThreshold”)) spark.conf.set(“spark.sql.autoBroadcastJoinThreshold”, -1) pr...
Read more