Difference between revisions of "Disk quotas"
m (→Notes) |
|||
| (16 intermediate revisions by 2 users not shown) | |||
| Line 1: | Line 1: | ||
=Disk Quotas= | =Disk Quotas= | ||
| − | Standard quotas | + | Standard quotas for the Math network home directories at this time are 12 GiB for faculty, |
| − | + | graduate students and visitors, and 4 GiB for all other accounts. 1 GiB = 1024<sup>3</sup> bytes. | |
| − | + | <del>If your account is over this limit, you will be notified upon | |
| − | |||
| − | If your account is over this limit, you will be notified upon | ||
login and will be given 5 login opportunities to clean up and reduce your disk | login and will be given 5 login opportunities to clean up and reduce your disk | ||
| − | usage to an under-quota status. | + | usage to an under-quota status.</del> |
Disk quotas are necessary on a system the size of Math in order to ensure | Disk quotas are necessary on a system the size of Math in order to ensure | ||
that all users have sufficient disk space available to them. | that all users have sufficient disk space available to them. | ||
| − | To avoid further slowing down the file servers, the system currently in | + | <del>To avoid further slowing down the file servers, the system currently in |
place determines each user's disk quota approximately once per day, in | place determines each user's disk quota approximately once per day, in | ||
| − | the early morning hours. | + | the early morning hours. If a user is over quota, then he or she is |
restricted to five logins until the quota system next determines that | restricted to five logins until the quota system next determines that | ||
| − | the user is within quota. | + | the user is within quota. When (or if) that occurs, the status is then |
| − | cleared. | + | cleared.</del> |
| − | Thus, login restrictions are imposed or lifted in 24-hour units. | + | <del>Thus, login restrictions are imposed or lifted in 24-hour units. This |
| − | gives a slightly stronger incentive to stay within quotas. | + | gives a slightly stronger incentive to stay within quotas. It is hoped |
that users who inadvertently exceed their disk quotas can bring themselves | that users who inadvertently exceed their disk quotas can bring themselves | ||
back under quota quickly, and can be fully productive with only a little | back under quota quickly, and can be fully productive with only a little | ||
| − | additional planning to keep from logging in more than five times on that day. | + | additional planning to keep from logging in more than five times on that day.</del> |
Users who need additional disk space are required to submit | Users who need additional disk space are required to submit | ||
| − | a request to [mailto:math | + | a request to [mailto:request@math.berkeley.edu request@math.berkeley.edu] with |
a careful explanation of what the excess disk space will be used for. | a careful explanation of what the excess disk space will be used for. | ||
==Notes== | ==Notes== | ||
| + | There are a few points which often lead to misunderstandings: | ||
| − | + | * The transparent data compression is enabled in the underlying (ZFS) file system used for home directories. The files are compressed/decompressed on the fly while writing/reading. The average compression ratio is about 1.6. This means you may need up to 20 GB of space if you want to transfer the data from your network home directory to your own storage device (1 GB = 10<sup>9</sup> B = 0.93 GiB). | |
| − | * The disk usage is only determined once per day. | + | * <del>The disk usage is only determined once per day. Therefore, if one is flagged as being over quota, the login restrictions will remain in effect for the whole day, even if the user immediately deletes files to get back under quota. Conversely, users can go well beyond their quotas with impunity during the day, as long as they reduce their disk usage before the next run of the quota accounting. (This practice is not recommended, because one may forget to delete the files before the end of the day) The reason for running the quota program only once per day is that it involves ascertaining the ownership of every file on a large (home) file system. It is therefore too resource-intensive to be run more often.</del> |
| − | * Disk totals are | + | * Disk totals are calculated by file ownership, not by whether they reside under your home directory. For most users this will not make any difference, but there are circumstances in which this may arise. |
| − | * The '''ls''' command leaves out files and directories beginning with "." unless the '''- | + | * The '''ls''' command leaves out files and directories beginning with "." unless the '''-A''' option is used. |
==Quotas on Mail== | ==Quotas on Mail== | ||
| − | The quotas described here are | + | The quotas described here are different from mail stored on |
| − | mail stored on | + | [http://bmail.berkeley.edu bMail]. The campus wide bMail is a part of entirely separate system |
| − | + | called [http://bconnected.berkeley.edu/ bConnected]. The bConnected storage was declared "unlimited" on Oct 30, 2014. | |
| − | (Prior to the conversion to CalMail, the | + | (Prior to the conversion to CalMail in 2009 and then to bMail in 2013, the Math department disk quota system |
| − | applied also to | + | applied also to mail boxes used by incoming mail, but that is no longer |
| − | the case. | + | the case. <del>The output of '''dqstatus''' still refers to the separate quota |
| − | on incoming mail, but that comment is vestigial only.) | + | on incoming mail, but that comment is vestigial only.</del>) |
==Determining Your Disk Usage== | ==Determining Your Disk Usage== | ||
| − | There are two programs to aid you in determining your disk usage. | + | <del>There are two programs to aid you in determining your disk usage.</del> |
| − | First, there is the program '''dqstatus'''. | + | <del>First, there is the program '''dqstatus''' (available on the departmental SSH gateway '''login.math.berkeley.edu'''). This will print your quota, |
your disk usage as of the last time the quota system was run, and | your disk usage as of the last time the quota system was run, and | ||
the number of logins you have left (if you are currently restricted). | the number of logins you have left (if you are currently restricted). | ||
It may also happen that you have files located in places other than your | It may also happen that you have files located in places other than your | ||
| − | home directory. | + | home directory. These files or directories are also reported by '''dqstatus'''.</del> |
| − | The other | + | The <del>other</del> Unix command '''du''' will tell you your current disk usage. |
In particular, the command | In particular, the command | ||
du -sk ~ | du -sk ~ | ||
| − | will tell you how much space you are using in your home directory. | + | will tell you how much space (in kilobytes) you are using in your home directory. <del>At the |
beginning of the day this should agree exactly with the output from | beginning of the day this should agree exactly with the output from | ||
'''dqstatus''' | '''dqstatus''' | ||
(unless there are files in your home directory that have somebody else | (unless there are files in your home directory that have somebody else | ||
| − | as an owner). | + | as an owner). As the day goes on, however, the numbers will differ since |
'''du''' reflects the current situation while '''dqstatus''' is only updated | '''du''' reflects the current situation while '''dqstatus''' is only updated | ||
| − | once daily. | + | once daily.</del> The '''du''' command has many other options |
| − | which are described in the manual page | + | which are described in the manual page ('''man du'''). |
| − | If you want to know what takes space in your current working directory, use | + | If you want to know what takes space in your current working directory, use |
the following command: | the following command: | ||
| − | du -sk | + | du -sk `ls -A` | sort -rn | head |
The output will list up to ten largest files or directories including names | The output will list up to ten largest files or directories including names | ||
| − | beginning with "." ( | + | beginning with "." (specified by the output of '''ls -A''' command). |
Latest revision as of 23:22, 10 January 2016
Disk Quotas
Standard quotas for the Math network home directories at this time are 12 GiB for faculty, graduate students and visitors, and 4 GiB for all other accounts. 1 GiB = 10243 bytes.
If your account is over this limit, you will be notified upon
login and will be given 5 login opportunities to clean up and reduce your disk
usage to an under-quota status.
Disk quotas are necessary on a system the size of Math in order to ensure that all users have sufficient disk space available to them.
To avoid further slowing down the file servers, the system currently in
place determines each user's disk quota approximately once per day, in
the early morning hours. If a user is over quota, then he or she is
restricted to five logins until the quota system next determines that
the user is within quota. When (or if) that occurs, the status is then
cleared.
Thus, login restrictions are imposed or lifted in 24-hour units. This
gives a slightly stronger incentive to stay within quotas. It is hoped
that users who inadvertently exceed their disk quotas can bring themselves
back under quota quickly, and can be fully productive with only a little
additional planning to keep from logging in more than five times on that day.
Users who need additional disk space are required to submit a request to request@math.berkeley.edu with a careful explanation of what the excess disk space will be used for.
Notes
There are a few points which often lead to misunderstandings:
- The transparent data compression is enabled in the underlying (ZFS) file system used for home directories. The files are compressed/decompressed on the fly while writing/reading. The average compression ratio is about 1.6. This means you may need up to 20 GB of space if you want to transfer the data from your network home directory to your own storage device (1 GB = 109 B = 0.93 GiB).
The disk usage is only determined once per day. Therefore, if one is flagged as being over quota, the login restrictions will remain in effect for the whole day, even if the user immediately deletes files to get back under quota. Conversely, users can go well beyond their quotas with impunity during the day, as long as they reduce their disk usage before the next run of the quota accounting. (This practice is not recommended, because one may forget to delete the files before the end of the day) The reason for running the quota program only once per day is that it involves ascertaining the ownership of every file on a large (home) file system. It is therefore too resource-intensive to be run more often.- Disk totals are calculated by file ownership, not by whether they reside under your home directory. For most users this will not make any difference, but there are circumstances in which this may arise.
- The ls command leaves out files and directories beginning with "." unless the -A option is used.
Quotas on Mail
The quotas described here are different from mail stored on bMail. The campus wide bMail is a part of entirely separate system called bConnected. The bConnected storage was declared "unlimited" on Oct 30, 2014.
(Prior to the conversion to CalMail in 2009 and then to bMail in 2013, the Math department disk quota system
applied also to mail boxes used by incoming mail, but that is no longer
the case. The output of dqstatus still refers to the separate quota
on incoming mail, but that comment is vestigial only.)
Determining Your Disk Usage
There are two programs to aid you in determining your disk usage.
First, there is the program dqstatus (available on the departmental SSH gateway login.math.berkeley.edu). This will print your quota,
your disk usage as of the last time the quota system was run, and
the number of logins you have left (if you are currently restricted).
It may also happen that you have files located in places other than your
home directory. These files or directories are also reported by dqstatus.
The other Unix command du will tell you your current disk usage.
In particular, the command
du -sk ~
will tell you how much space (in kilobytes) you are using in your home directory. At the
beginning of the day this should agree exactly with the output from
dqstatus
(unless there are files in your home directory that have somebody else
as an owner). As the day goes on, however, the numbers will differ since
du reflects the current situation while dqstatus is only updated
once daily. The du command has many other options
which are described in the manual page (man du).
If you want to know what takes space in your current working directory, use the following command:
du -sk `ls -A` | sort -rn | head
The output will list up to ten largest files or directories including names beginning with "." (specified by the output of ls -A command).
