Difference between revisions of "Disk quotas"

From UCB Math Wiki
Jump to: navigation, search
m (Disk Quotas)
(Disk Quotas)
Line 3: Line 3:
 
graduate students and visitors, and 4 GiB for all other accounts. GiB means 1024<sup>3</sup> bytes.
 
graduate students and visitors, and 4 GiB for all other accounts. GiB means 1024<sup>3</sup> bytes.
  
If your account is over this limit, you will be notified upon
+
<del>If your account is over this limit, you will be notified upon
 
login and will be given 5 login opportunities to clean up and reduce your disk
 
login and will be given 5 login opportunities to clean up and reduce your disk
usage to an under-quota status.
+
usage to an under-quota status.</del>
  
 
Disk quotas are necessary on a system the size of Math in order to ensure
 
Disk quotas are necessary on a system the size of Math in order to ensure
 
that all users have sufficient disk space available to them.
 
that all users have sufficient disk space available to them.
  
To avoid further slowing down the file servers, the system currently in
+
<del>To avoid further slowing down the file servers, the system currently in
 
place determines each user's disk quota approximately once per day, in
 
place determines each user's disk quota approximately once per day, in
the early morning hours. If a user is over quota, then he or she is
+
the early morning hours. If a user is over quota, then he or she is
 
restricted to five logins until the quota system next determines that
 
restricted to five logins until the quota system next determines that
the user is within quota. When (or if) that occurs, the status is then
+
the user is within quota. When (or if) that occurs, the status is then
cleared.
+
cleared.</del>
  
Thus, login restrictions are imposed or lifted in 24-hour units. This
+
<del>Thus, login restrictions are imposed or lifted in 24-hour units. This
gives a slightly stronger incentive to stay within quotas. It is hoped
+
gives a slightly stronger incentive to stay within quotas. It is hoped
 
that users who inadvertently exceed their disk quotas can bring themselves
 
that users who inadvertently exceed their disk quotas can bring themselves
 
back under quota quickly, and can be fully productive with only a little
 
back under quota quickly, and can be fully productive with only a little
additional planning to keep from logging in more than five times on that day.
+
additional planning to keep from logging in more than five times on that day.</del>
  
 
Users who need additional disk space are required to submit
 
Users who need additional disk space are required to submit
Line 28: Line 28:
  
 
==Notes==
 
==Notes==
 +
There are a few points which often lead to misunderstandings:
  
There are a few points which often lead to misunderstandings:
+
* The files in your home directory are compressed/decompressed on the fly while writing/reading. The transparent data compression is enabled in the underlying (ZFS) file system used for home directories. The average compression ratio is about 1.6. This means you may need about 20 GB of space if you want to transfer the data from your network home directory to your own storage device (1 GB = 0.93 GiB).
  
* The disk usage is only determined once per day. Therefore, if one is flagged as being over quota, the login restrictions will remain in effect for the whole day, even if the user immediately deletes files to get back under quota. Conversely, users can go well beyond their quotas with impunity during the day, as long as they reduce their disk usage before the next run of the quota accounting. (This practice is not recommended, because one may forget to delete the files before the end of the day) The reason for running the quota program only once per day is that it involves ascertaining the ownership of every file on a large (home) file system. It is therefore too resource-intensive to be run more often.
+
* <del>The disk usage is only determined once per day. Therefore, if one is flagged as being over quota, the login restrictions will remain in effect for the whole day, even if the user immediately deletes files to get back under quota. Conversely, users can go well beyond their quotas with impunity during the day, as long as they reduce their disk usage before the next run of the quota accounting. (This practice is not recommended, because one may forget to delete the files before the end of the day) The reason for running the quota program only once per day is that it involves ascertaining the ownership of every file on a large (home) file system. It is therefore too resource-intensive to be run more often.</del>
* Disk totals are computed by file ownership, not by whether they reside under your home directory. For most users this will not make any difference, but there are circumstances in which this may arise.
+
* Disk totals are computed by file ownership, not by whether they reside under your home directory. For most users this will not make any difference, but there are circumstances in which this may arise.
 
* The '''ls''' command leaves out files and directories beginning with "." unless the '''-A''' option is used.
 
* The '''ls''' command leaves out files and directories beginning with "." unless the '''-A''' option is used.
  
 
==Quotas on Mail==
 
==Quotas on Mail==
 
The quotas described here are different from mail stored on  
 
The quotas described here are different from mail stored on  
[http://bmail.berkeley.edu bMail]. The campus wide bMail is a part of entirely separate system
+
[http://bmail.berkeley.edu bMail]. The campus wide bMail is a part of entirely separate system
 
called [http://bconnected.berkeley.edu/ bConnected]. The bConnected storage was declared "unlimited" on Oct 30, 2014.
 
called [http://bconnected.berkeley.edu/ bConnected]. The bConnected storage was declared "unlimited" on Oct 30, 2014.
  
 
(Prior to the conversion to CalMail in 2009 and then to bMail in 2013, the Math department disk quota system
 
(Prior to the conversion to CalMail in 2009 and then to bMail in 2013, the Math department disk quota system
 
applied also to mail boxes used by incoming mail, but that is no longer
 
applied also to mail boxes used by incoming mail, but that is no longer
the case. The output of '''dqstatus''' still refers to the separate quota
+
the case. <del>The output of '''dqstatus''' still refers to the separate quota
on incoming mail, but that comment is vestigial only.)
+
on incoming mail, but that comment is vestigial only.</del>)
  
 
==Determining Your Disk Usage==
 
==Determining Your Disk Usage==
There are two programs to aid you in determining your disk usage.
+
<del>There are two programs to aid you in determining your disk usage.</del>
  
First, there is the program '''dqstatus''' (available on the departmental SSH gateway '''login.math.berkeley.edu'''). This will print your quota,
+
<del>First, there is the program '''dqstatus''' (available on the departmental SSH gateway '''login.math.berkeley.edu'''). This will print your quota,
 
your disk usage as of the last time the quota system was run, and
 
your disk usage as of the last time the quota system was run, and
 
the number of logins you have left (if you are currently restricted).
 
the number of logins you have left (if you are currently restricted).
 
It may also happen that you have files located in places other than your
 
It may also happen that you have files located in places other than your
home directory. These files or directories are also reported by '''dqstatus'''.
+
home directory. These files or directories are also reported by '''dqstatus'''.</del>
  
The other program is '''du'''. This will tell you your current disk usage.
+
The <del>other</del> program is '''du'''. This will tell you your current disk usage.
 
In particular, the command
 
In particular, the command
 
  du -sk ~
 
  du -sk ~
will tell you how much space (in kilobytes) you are using in your home directory. At the
+
will tell you how much space (in kilobytes) you are using in your home directory. <del>At the
 
beginning of the day this should agree exactly with the output from
 
beginning of the day this should agree exactly with the output from
 
'''dqstatus'''
 
'''dqstatus'''
 
(unless there are files in your home directory that have somebody else
 
(unless there are files in your home directory that have somebody else
as an owner). As the day goes on, however, the numbers will differ since
+
as an owner). <del>As the day goes on, however, the numbers will differ since
 
'''du''' reflects the current situation while '''dqstatus''' is only updated
 
'''du''' reflects the current situation while '''dqstatus''' is only updated
once daily. '''du''' is a standard Unix command; it has many other options
+
once daily.</del> '''du''' is a standard Unix command; it has many other options
 
which are described in the manual page for '''du'''.
 
which are described in the manual page for '''du'''.
  
If you want to know what takes space in your current working directory, use  
+
If you want to know what takes space in your current working directory, use
 
the following command:
 
the following command:
 
  du -sk `ls -A` | sort -rn | head
 
  du -sk `ls -A` | sort -rn | head
 
The output will list up to ten largest files or directories including names
 
The output will list up to ten largest files or directories including names
 
beginning with "." (specified by the output of '''ls -A''' command).
 
beginning with "." (specified by the output of '''ls -A''' command).

Revision as of 16:20, 2 January 2016

Disk Quotas

Standard quotas for the Math network home directories at this time are: 12 GiB for faculty, graduate students and visitors, and 4 GiB for all other accounts. GiB means 10243 bytes.

If your account is over this limit, you will be notified upon login and will be given 5 login opportunities to clean up and reduce your disk usage to an under-quota status.

Disk quotas are necessary on a system the size of Math in order to ensure that all users have sufficient disk space available to them.

To avoid further slowing down the file servers, the system currently in place determines each user's disk quota approximately once per day, in the early morning hours. If a user is over quota, then he or she is restricted to five logins until the quota system next determines that the user is within quota. When (or if) that occurs, the status is then cleared.

Thus, login restrictions are imposed or lifted in 24-hour units. This gives a slightly stronger incentive to stay within quotas. It is hoped that users who inadvertently exceed their disk quotas can bring themselves back under quota quickly, and can be fully productive with only a little additional planning to keep from logging in more than five times on that day.

Users who need additional disk space are required to submit a request to request@math.berkeley.edu with a careful explanation of what the excess disk space will be used for.

Notes

There are a few points which often lead to misunderstandings:

  • The files in your home directory are compressed/decompressed on the fly while writing/reading. The transparent data compression is enabled in the underlying (ZFS) file system used for home directories. The average compression ratio is about 1.6. This means you may need about 20 GB of space if you want to transfer the data from your network home directory to your own storage device (1 GB = 0.93 GiB).
  • The disk usage is only determined once per day. Therefore, if one is flagged as being over quota, the login restrictions will remain in effect for the whole day, even if the user immediately deletes files to get back under quota. Conversely, users can go well beyond their quotas with impunity during the day, as long as they reduce their disk usage before the next run of the quota accounting. (This practice is not recommended, because one may forget to delete the files before the end of the day) The reason for running the quota program only once per day is that it involves ascertaining the ownership of every file on a large (home) file system. It is therefore too resource-intensive to be run more often.
  • Disk totals are computed by file ownership, not by whether they reside under your home directory. For most users this will not make any difference, but there are circumstances in which this may arise.
  • The ls command leaves out files and directories beginning with "." unless the -A option is used.

Quotas on Mail

The quotas described here are different from mail stored on bMail. The campus wide bMail is a part of entirely separate system called bConnected. The bConnected storage was declared "unlimited" on Oct 30, 2014.

(Prior to the conversion to CalMail in 2009 and then to bMail in 2013, the Math department disk quota system applied also to mail boxes used by incoming mail, but that is no longer the case. The output of dqstatus still refers to the separate quota on incoming mail, but that comment is vestigial only.)

Determining Your Disk Usage

There are two programs to aid you in determining your disk usage.

First, there is the program dqstatus (available on the departmental SSH gateway login.math.berkeley.edu). This will print your quota, your disk usage as of the last time the quota system was run, and the number of logins you have left (if you are currently restricted). It may also happen that you have files located in places other than your home directory. These files or directories are also reported by dqstatus.

The other program is du. This will tell you your current disk usage. In particular, the command

du -sk ~

will tell you how much space (in kilobytes) you are using in your home directory. At the beginning of the day this should agree exactly with the output from dqstatus (unless there are files in your home directory that have somebody else as an owner). As the day goes on, however, the numbers will differ since du reflects the current situation while dqstatus is only updated once daily. du is a standard Unix command; it has many other options which are described in the manual page for du.

If you want to know what takes space in your current working directory, use the following command:

du -sk `ls -A` | sort -rn | head

The output will list up to ten largest files or directories including names beginning with "." (specified by the output of ls -A command).