Difference between revisions of "Disk quotas"

From UCB Math Wiki
Jump to: navigation, search
m (Notes)
 
(15 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 
=Disk Quotas=
 
=Disk Quotas=
Standard quotas on the Math Network at this time are: 600 MB for faculty,
+
Standard quotas for the Math network home directories at this time are 12 GiB for faculty,
450 MB for graduate students and 300 MB for visitors' accounts.
+
graduate students and visitors, and 4 GiB for all other accounts. 1 GiB = 1024<sup>3</sup> bytes.
  
For the purposes of this web page, MB means 1024000 bytes.
+
<del>If your account is over this limit, you will be notified upon
 
 
If your account is over this limit, you will be notified upon
 
 
login and will be given 5 login opportunities to clean up and reduce your disk
 
login and will be given 5 login opportunities to clean up and reduce your disk
usage to an under-quota status.
+
usage to an under-quota status.</del>
  
 
Disk quotas are necessary on a system the size of Math in order to ensure
 
Disk quotas are necessary on a system the size of Math in order to ensure
 
that all users have sufficient disk space available to them.
 
that all users have sufficient disk space available to them.
  
To avoid further slowing down the file servers, the system currently in
+
<del>To avoid further slowing down the file servers, the system currently in
 
place determines each user's disk quota approximately once per day, in
 
place determines each user's disk quota approximately once per day, in
the early morning hours. If a user is over quota, then he or she is
+
the early morning hours. If a user is over quota, then he or she is
 
restricted to five logins until the quota system next determines that
 
restricted to five logins until the quota system next determines that
the user is within quota. When (or if) that occurs, the status is then
+
the user is within quota. When (or if) that occurs, the status is then
cleared.
+
cleared.</del>
  
Thus, login restrictions are imposed or lifted in 24-hour units. This
+
<del>Thus, login restrictions are imposed or lifted in 24-hour units. This
gives a slightly stronger incentive to stay within quotas. It is hoped
+
gives a slightly stronger incentive to stay within quotas. It is hoped
 
that users who inadvertently exceed their disk quotas can bring themselves
 
that users who inadvertently exceed their disk quotas can bring themselves
 
back under quota quickly, and can be fully productive with only a little
 
back under quota quickly, and can be fully productive with only a little
additional planning to keep from logging in more than five times on that day.
+
additional planning to keep from logging in more than five times on that day.</del>
  
 
Users who need additional disk space are required to submit
 
Users who need additional disk space are required to submit
a request to [mailto:math@lscr.berkeley.edu math@lscr.berkeley.edu] with
+
a request to [mailto:request@math.berkeley.edu request@math.berkeley.edu] with
 
a careful explanation of what the excess disk space will be used for.
 
a careful explanation of what the excess disk space will be used for.
  
 
==Notes==
 
==Notes==
 +
There are a few points which often lead to misunderstandings:
  
There are a few points which often lead to misunderstandings:
+
* The transparent data compression is enabled in the underlying (ZFS) file system used for home directories. The files are compressed/decompressed on the fly while writing/reading. The average compression ratio is about 1.6. This means you may need up to 20 GB of space if you want to transfer the data from your network home directory to your own storage device (1 GB = 10<sup>9</sup> B = 0.93 GiB).
  
* The disk usage is only determined once per day. Therefore, if one is flagged as being over quota, the login restrictions will remain in effect for the whole day, even if the user immediately deletes files to get back under quota. Conversely, users can go well beyond their quotas with impunity during the day, as long as they reduce their disk usage before the next run of the quota system. (This practice is not recommended, however--(a) one may forget to delete the files before the end of the day, and (b) if everybody does this then the problems with full file systems will recur.) The reason for running the quota program only once per day is that it involves ascertaining the ownership of every file on every disk. It is therefore too resource-intensive to be run more often.
+
* <del>The disk usage is only determined once per day. Therefore, if one is flagged as being over quota, the login restrictions will remain in effect for the whole day, even if the user immediately deletes files to get back under quota. Conversely, users can go well beyond their quotas with impunity during the day, as long as they reduce their disk usage before the next run of the quota accounting. (This practice is not recommended, because one may forget to delete the files before the end of the day) The reason for running the quota program only once per day is that it involves ascertaining the ownership of every file on a large (home) file system. It is therefore too resource-intensive to be run more often.</del>
* Disk totals are computed by file ownership, not by whether they reside under your home directory. For most users this will not make any difference, but there are circumstances in which this may arise.
+
* Disk totals are calculated by file ownership, not by whether they reside under your home directory. For most users this will not make any difference, but there are circumstances in which this may arise.
* The '''ls''' command leaves out files and directories beginning with "." unless the '''-a''' option is used.
+
* The '''ls''' command leaves out files and directories beginning with "." unless the '''-A''' option is used.
  
 
==Quotas on Mail==
 
==Quotas on Mail==
The quotas described here are entirely separate from the 1GB quota on
+
The quotas described here are different from mail stored on  
mail stored on the CalMail system. That is an entirely separate system,
+
[http://bmail.berkeley.edu bMail]. The campus wide bMail is a part of entirely separate system
with different quota requirements.
+
called [http://bconnected.berkeley.edu/ bConnected]. The bConnected storage was declared "unlimited" on Oct 30, 2014.
  
(Prior to the conversion to CalMail, the math department disk quota system
+
(Prior to the conversion to CalMail in 2009 and then to bMail in 2013, the Math department disk quota system
applied also to disk storage used by incoming mail, but that is no longer
+
applied also to mail boxes used by incoming mail, but that is no longer
the case. The output of '''dqstatus''' still refers to the separate quota
+
the case. <del>The output of '''dqstatus''' still refers to the separate quota
on incoming mail, but that comment is vestigial only.)
+
on incoming mail, but that comment is vestigial only.</del>)
  
 
==Determining Your Disk Usage==
 
==Determining Your Disk Usage==
There are two programs to aid you in determining your disk usage.
+
<del>There are two programs to aid you in determining your disk usage.</del>
  
First, there is the program '''dqstatus'''. This will print your quota,
+
<del>First, there is the program '''dqstatus''' (available on the departmental SSH gateway '''login.math.berkeley.edu'''). This will print your quota,
 
your disk usage as of the last time the quota system was run, and
 
your disk usage as of the last time the quota system was run, and
 
the number of logins you have left (if you are currently restricted).
 
the number of logins you have left (if you are currently restricted).
 
It may also happen that you have files located in places other than your
 
It may also happen that you have files located in places other than your
home directory. These files or directories are also reported by '''dqstatus'''.
+
home directory. These files or directories are also reported by '''dqstatus'''.</del>
  
The other program is '''du'''.  This will tell you your current disk usage.
+
The <del>other</del> Unix command '''du''' will tell you your current disk usage.
 
In particular, the command
 
In particular, the command
 
  du -sk ~
 
  du -sk ~
will tell you how much space you are using in your home directory. At the
+
will tell you how much space (in kilobytes) you are using in your home directory. <del>At the
 
beginning of the day this should agree exactly with the output from
 
beginning of the day this should agree exactly with the output from
 
'''dqstatus'''
 
'''dqstatus'''
 
(unless there are files in your home directory that have somebody else
 
(unless there are files in your home directory that have somebody else
as an owner). As the day goes on, however, the numbers will differ since
+
as an owner). As the day goes on, however, the numbers will differ since
 
'''du''' reflects the current situation while '''dqstatus''' is only updated
 
'''du''' reflects the current situation while '''dqstatus''' is only updated
once daily. '''du''' is a standard Unix command; it has many other options
+
once daily.</del> The '''du''' command has many other options
which are described in the manual page for '''du'''.
+
which are described in the manual page ('''man du''').
  
If you want to know what takes space in your current working directory, use  
+
If you want to know what takes space in your current working directory, use
 
the following command:
 
the following command:
  du -sk .??* * | sort -rn | head
+
  du -sk `ls -A` | sort -rn | head
 
The output will list up to ten largest files or directories including names
 
The output will list up to ten largest files or directories including names
beginning with "." (specified by .??*).
+
beginning with "." (specified by the output of '''ls -A''' command).

Latest revision as of 00:22, 11 January 2016

Disk Quotas

Standard quotas for the Math network home directories at this time are 12 GiB for faculty, graduate students and visitors, and 4 GiB for all other accounts. 1 GiB = 10243 bytes.

If your account is over this limit, you will be notified upon login and will be given 5 login opportunities to clean up and reduce your disk usage to an under-quota status.

Disk quotas are necessary on a system the size of Math in order to ensure that all users have sufficient disk space available to them.

To avoid further slowing down the file servers, the system currently in place determines each user's disk quota approximately once per day, in the early morning hours. If a user is over quota, then he or she is restricted to five logins until the quota system next determines that the user is within quota. When (or if) that occurs, the status is then cleared.

Thus, login restrictions are imposed or lifted in 24-hour units. This gives a slightly stronger incentive to stay within quotas. It is hoped that users who inadvertently exceed their disk quotas can bring themselves back under quota quickly, and can be fully productive with only a little additional planning to keep from logging in more than five times on that day.

Users who need additional disk space are required to submit a request to request@math.berkeley.edu with a careful explanation of what the excess disk space will be used for.

Notes

There are a few points which often lead to misunderstandings:

  • The transparent data compression is enabled in the underlying (ZFS) file system used for home directories. The files are compressed/decompressed on the fly while writing/reading. The average compression ratio is about 1.6. This means you may need up to 20 GB of space if you want to transfer the data from your network home directory to your own storage device (1 GB = 109 B = 0.93 GiB).
  • The disk usage is only determined once per day. Therefore, if one is flagged as being over quota, the login restrictions will remain in effect for the whole day, even if the user immediately deletes files to get back under quota. Conversely, users can go well beyond their quotas with impunity during the day, as long as they reduce their disk usage before the next run of the quota accounting. (This practice is not recommended, because one may forget to delete the files before the end of the day) The reason for running the quota program only once per day is that it involves ascertaining the ownership of every file on a large (home) file system. It is therefore too resource-intensive to be run more often.
  • Disk totals are calculated by file ownership, not by whether they reside under your home directory. For most users this will not make any difference, but there are circumstances in which this may arise.
  • The ls command leaves out files and directories beginning with "." unless the -A option is used.

Quotas on Mail

The quotas described here are different from mail stored on bMail. The campus wide bMail is a part of entirely separate system called bConnected. The bConnected storage was declared "unlimited" on Oct 30, 2014.

(Prior to the conversion to CalMail in 2009 and then to bMail in 2013, the Math department disk quota system applied also to mail boxes used by incoming mail, but that is no longer the case. The output of dqstatus still refers to the separate quota on incoming mail, but that comment is vestigial only.)

Determining Your Disk Usage

There are two programs to aid you in determining your disk usage.

First, there is the program dqstatus (available on the departmental SSH gateway login.math.berkeley.edu). This will print your quota, your disk usage as of the last time the quota system was run, and the number of logins you have left (if you are currently restricted). It may also happen that you have files located in places other than your home directory. These files or directories are also reported by dqstatus.

The other Unix command du will tell you your current disk usage. In particular, the command

du -sk ~

will tell you how much space (in kilobytes) you are using in your home directory. At the beginning of the day this should agree exactly with the output from dqstatus (unless there are files in your home directory that have somebody else as an owner). As the day goes on, however, the numbers will differ since du reflects the current situation while dqstatus is only updated once daily. The du command has many other options which are described in the manual page (man du).

If you want to know what takes space in your current working directory, use the following command:

du -sk `ls -A` | sort -rn | head

The output will list up to ten largest files or directories including names beginning with "." (specified by the output of ls -A command).