/HDFS-quota-groups

scripts used to configure HDFS space quota to users that belong to a group (already in ambari, local groups, LDAP groups, etc)

Primary LanguagePythonMIT LicenseMIT

steps to create directories in HDFS for every user of an specific group and assign a Space Quota

  1. login in the principal Ambari Server with superadmin user (HDFS superadmin) (superuser group of hdfs is configured in dfs.permissions.superusergroup in hdfs-site.xml, to add an existent user to the default superuser group do: usermod -a -G hdfs <username>)

  2. set the environment variables export AMBARI_USER=llenar_usuario_de_ambari AMBARI_PASSWD=llenar_password AMBARI_HOST=llenar_con_host CLUSTER_NAME=llenar_con_nombre_cluster C_GROUP=llenar_con_nombre_de_grupo

  3. assign a space quota to a group/user(local or LDAP ambari user): python create_dirs_and_quota.py

  4. you can chek the assigned quota with: hdfs dfs -count -h -q /user/*

MANUAL SETTING: How to use space Quota in HDFS

if you prefer to do it manually without the scripts provided

The hdfs admin can create Quota. There is two types of quota:

  • space quota ( how much storage can be used )
  • file quota ( how many files can be created )

This alert only check the space quota, it doesn't look into file quota.

Set a quota of 2GB on folder /application/digital - value are in bytes = 210241024*1024 = 2147483648

hdfs dfsadmin -setSpaceQuota 2147483648  /application/digital

NB: Quota works on raw storage

Check the quota ( can be run by any user w/ read permission on folder )

hdfs dfs -count -q /application/digital
       none             inf      2147483648      1832910848            1            1          104857600 /application/digital
                                    |               |
                                  Quota           Usage

Delete quota

hdfs dfsadmin -clrSpaceQuota /application/digital

references: