X-Git-Url: https://sipb.mit.edu/gitweb.cgi/wiki.git/blobdiff_plain/7ca43cd4d837c1a16fd0e0053ee7dbbbf9db8123..7f41deb1034c064de31a3e7a056e0b60b0b2f43f:/doc/afs-and-you.html diff --git a/doc/afs-and-you.html b/doc/afs-and-you.html index 3f62fae..d2934c4 100644 --- a/doc/afs-and-you.html +++ b/doc/afs-and-you.html @@ -19,7 +19,7 @@
  • Troubleshooting
      -
    1. I'm trying to access my files, fs la says I should have permissions …
    2. +
    3. I'm trying to access my files, fs la says I should have permissions …
    4. I'd really prefer that not everyone could list my files, how should I stop …
    5. It was around 6am on a Sunday morning and suddenly I couldn't access my …
    6. It isn't Sunday and I can't get to my files
    7. @@ -28,6 +28,8 @@
      1. Putting Software in a Locker
      2. Acquiring a new locker
      3. +
      4. Find out technical + information about my locker
    8. See Also
    @@ -41,30 +43,30 @@ Credit goes to them, blame goes to him.

    What is AFS?

    -The Andrew File System or AFS is a distributed network file system invented at Carnegie Mellon University as part of Project Andrew (approximately their equivalent of MIT's Project Athena). More importantly, it is the file system used to store most files on Athena today. This includes your personal home directory, the data and websites of many living groups and student groups on campus, and probably some of the software you run (if you ever use Athena clusters). (Though all user directories were migrated from NFS in the summer of 1992, some files probably still remain on NFS and, of course, various file systems are used on personal computers and servers.) +AFS (previously the Andrew File System or ) is a distributed network file system invented at Carnegie Mellon University as part of Project Andrew (approximately their equivalent of MIT's Project Athena). More importantly, it is the file system used to store most files on Athena today. This includes your personal home directory, the data and websites of many living groups and student groups on campus, and probably some of the software you run (if you ever use Athena clusters). (Though most user directories were migrated from NFS in the summer of 1992, some files still remain on NFS and, of course, various file systems are used on personal computers and servers.)

    In Short: AFS is probably where some of the files you care about live

    -For the most part, using AFS, particularly at MIT, is well-hidden and can be used like any other UNIX file system. For some things, you will need to know a bit more. Let's start by defining some terms. +For the most part, using AFS, particularly at MIT, is well-abstracted; For most tasks, it can be used like any other UNIX file system. For a few, you will need to know a bit more. Let's start by defining some terms.

    Some MIT/AFS terminology

    locker
    -For practical purposes, a folder. Probably the what you'll care about most of the time. Technically any directory mountable under /mit, regardless of how its stored. Today, most lockers lockers are stored in AFS. +For practical purposes, a folder. Probably what you'll care about most of the time. Technically any directory with a hesiod FILSYS entry, regardless of how its stored. Most lockers, however, are stored in AFS and are usually accessed as directories in /mit.
    tokens
    -Essentially proof to the AFS servers that you are who you say you are, thus allowing you to access files you are supposed to. Analogous to Kerberos tickets. +Proof to the AFS servers that you are who you say you are, thus allowing you to access files you are supposed to. Technically, a special type of Kerberos ticket.
    cell
    -AFS concept of an "administrative domain of authority." Each cell has its own set of users, groups, and administrators. Analogous to a Kerberos realm. Each top-level directory in /afs corresponds to a cell. The cells you are most likely to care about are athena.mit.edu and sipb.mit.edu. +AFS concept of an "administrative domain of authority." Each cell has its own set of users, groups, and administrators. Analogous to a Kerberos realm. Each top-level directory in /afs generally corresponds to a cell. The cells you are most likely to care about are athena.mit.edu and sipb.mit.edu.

    The Basics

    The Layout of a Typical MIT Locker

    -Every Athena user has a locker (their home directory), which mounts at /mit/<username>. From a technical standpoint, it is stored in the volume user.<username> which is located at /afs/athena.mit.edu/user/<first letter>/<second letter>/<user name>. For example, the user joeuser has a home directory that mounts at /mit/joeuser, is volume user.joeuser, and is accessible at /afs/user/j/o/joeuser. Lockers for projects, software, classes, living groups, and student groups are all mounted at /mit/<lockername> and stored at various places in AFS. +Every Athena user has a locker (their home directory), which mounts at /mit/<username>. From a technical standpoint, it is (almost always) stored in the volume user.<username> which is located at /afs/athena.mit.edu/user/<first letter>/<second letter>/<user name>. For example, the user joeuser has a home directory that mounts at /mit/joeuser, is volume user.joeuser, and is accessible at /afs/athena.mit.edu/user/j/o/joeuser. Lockers for projects, software, classes, living groups, and student groups are all mounted at /mit/<lockername> and stored in various places in AFS.

    @@ -73,24 +75,24 @@ Within each locker, there are (by default) 4 special subdirectories you want to

    Public
    By default, this directory can be read by everyone (and I do mean EVERYONE!), so put files here you want to share with the entire world.
    Private
    -By default, this directory can only be read and can only be listed by you (from AFS, not from web) whereas the files in other directories can (by default) only be read by you, but can be listed by anyone. This means that other people can't see the contents of your files, but they can see what files you have outside of Private +By default, this directory can only be read and can only be listed by you (from AFS, not from web) whereas the files in other directories can (by default) only be read by you, but can be listed by anyone. This means that other people can't see the contents of your files, but they can see the names of the files you have outside of Private
    OldFiles
    -This folder is a link to a read-only copy of a backup of your files (created nightly at 3 a.m.). This copy cannot be edited and does not count towards the locker's quota. (From a technical standpoint, this is a separate volume, user.<username>.backup and is stored only as changes against the current copy.) +This folder is a link to a read-only copy of a backup of your files (created nightly around 3 a.m.). This copy cannot be edited and does not count against the locker's quota. From a technical standpoint, this is a separate volume with .backup appended (e.g. user.<username>.backup ) and is stored only as changes against the current copy.
    www
    -Where you should put a website, if you want one. There is very little special about this directory from an AFS standpoint, but it is world-readable (like Public) and is linked directly to http://www.mit.edu/~<lockername> as well as http://web.mit.edu/<lockername>. +Where you should put a website, if you want one. There is very little special about this directory from an AFS standpoint, but it is world-readable (like Public) and is linked directly to http://www.mit.edu/~<lockername> as well as http://web.mit.edu/<lockername>/www/.

    Accessing Lockers

    From Athena

    -On Athena, you can access a locker either as its full AFS path, if you know it (e.g. /afs/athena.mit.edu/course/6/6.01), or under /mit if it is "attached." Yet a third way is to use tilde expansion and simply cd ~~<locker> (e.g. cd ~~6.01) which will take you to the full path. Often, however, you want to attach lockers because it is easier to refer to them outside of the shell (in a script, for example) and software is set up to run with a path under /mit. There are a few ways to attach a locker: +On Athena, you can access a locker either as its full AFS path, if you know it (e.g. /afs/athena.mit.edu/course/6/6.01), or under /mit if it is "attached." Though you can always use the full path, you often want to attach lockers because it is easier to refer to them and software is set up to run with a path under /mit. There are a few ways to attach a locker:

    From the Web

    @@ -99,7 +101,9 @@ Generally any locker that you would access on Athena as /mit/<locker><

    -Also, you may access something in one of the MIT AFS cells by taking its full AFS path after web.mit.edu (http://web.mit.edu/afs/athena.mit.edu/activity/c/chess-club). (That link also shows that if you have a text file named README readable, as a link to Public/README for example, its contents will be displayed below the directory listing). +Also, you may access something in one of the MIT AFS cells by typing its full AFS path after web.mit.edu (http://web.mit.edu/afs/athena.mit.edu/activity/c/chess-club). (That link also shows that if you have a text file named README readable, as a link to Public/README for example, its contents will be displayed below the directory listing). + +Note that when accessed from web.mit.edu (or www.mit.edu), only static files may be shown. If you are interested in serving dynamic content (such as a blog or wiki using PHP, Perl, Python, Ruby, etc.), you should check out SIPB's Scripts dynamic web service. See http://scripts.mit.edu for more information.

    Checking Quota

    @@ -115,7 +119,7 @@ If this information is good enough for you, then you are done. If not, read on.

    Common Tasks

    Controlling Who can Access Files

    -You may be familiar with Unix permissions. Sad to say, but that knowledge is basically useless here. Whereas Unix permissions, are per-file, AFS permissions are controlled by Access Control Lists (ACLs) on a per-directory basis. +You may be familiar with Unix permissions. Sad to say, but that knowledge is more or less useless here. While Unix permissions are per-file, AFS permissions are controlled by Access Control Lists (ACLs) on a per-directory basis. (AFS does however, attend to the execute Unix permission on a file. )

    To view the ACL for a given directory (where you have permission to do so), run fs listacl or fs la, for short. For a typical user locker, the ACL in the top level will look like this @@ -131,23 +135,24 @@ Normal rights: This is a list of users or AFS groups and their permissions in this directory (and subdirectories that don't have their own ACL modifications). In AFS there are seven permissions as follows

    - +
    r read user or members of group can read files in the directory (i.e. see the contents of files) -
    l list user or members of group can list files in the directory (i.e. see the names of files) +
    l list user or members of group can list files in the directory (i.e. see the names of files), they can also list the permissions of the directory.
    i insert user or members of group can create files (or subdirectories) in the directory
    d delete user or members of group can delete files in the directory
    w write user or members of group can modify files in the directory (i.e. change the contents of files)
    k lock user or members of group can lock files in the directory (you will likely never use this) -
    a admin user or members of group can see and change permissions. It does not affect pre-existing subdirectories. -
    +a admin user or members of group can change permissions of this directory (though not existing subdirectories, unless they have the a permission in those too. They will have admin powers in newly created subdirectories.) + +A,B,…,Hsite-specificSite-specific permissions that are meaningless to AFS but may be used by other software (or special versions of AFS).

    To add a user or group to the ACL for a given directory simply run fs setacl or fs sa as follows:

    -
    fs sa <directory> <user or group> <permissions>
    +
    fs sa <directory> <user or group> <permissions> [<user or group> <permissions>]*
     
    <directory>
    can be an absolute or relative path, usually you will want .
    <user or group>
    @@ -164,6 +169,7 @@ For example, if user wants his friends sipbtest and jarand user@host:~/awesome_project$ fs sa . system:anyuser read user@host:~/awesome_project$ fs sa . jarandom write user@host:~/awesome_project$ fs sa . sipbtest write +user@host:~/awesome_project$ #alternatively: fs sa . system:anyuser read jarandom write sipbtest write user@host:~/awesome_project$ fs la Access list for . is Normal rights: @@ -175,18 +181,37 @@ Normal rights: user@host:~/awesome_project$

    -See also: man 1 fs, fs help <command> +See also: man 1 fs, fs help <command>, man fs_listacl. + +There is also such thing as negative permissions to deny rights to certain members of a larger group to which positive permissions are granted. In the words of the fs_setacl manpage, however,

    Setting negative permissions is generally unnecessary and not recommended. Simply omitting a user or group from the "Normal rights" section of the ACL is normally adequate to prevent access. In particular, note that it is futile to deny permissions that are granted to members of the system:anyuser group on the same ACL; the user needs only to issue the unlog command to receive the denied permissions.
    For an example of negative permissions used on Athena run fs la /afs/athena.mit.edu/contrib/games/. + +

    Creating an AFS Group

    -The "normal" way to make an AFS group would be with a command similar to pts creategroup <your user name>:<group name> and then add people with pts adduser <user> <full group name>(e.g. If Donald Guy wanted to created a group for people to edit his www directory (including sipbtest and jflorey, he might use the following chain of commands pts creategroup fawkes:www ; pts adduser sipbtest fawkes:www; pts adduser jflorey fawkes:www; fs sa ~/www fawkes:www write - +The "normal" way to make an AFS group would be with a command similar to pts creategroup <your user name>:<group name> and then add people with pts adduser <user> <full group name>(e.g. If Donald Guy wanted to created a group for people to edit his www directory (including sipbtest and jflorey, he might use the following chain of commands pts creategroup fawkes:www ; pts adduser sipbtest fawkes:www; pts adduser jflorey fawkes:www; fs sa /mit/fawkes/www fawkes:www write +

    +

    +You can see general information about a group by running pts examine <group> and see the membership of a group by running pts membership <group>. In the above example: +

    +fawkes@dr-wily:~$ pts examine fawkes:www
    +Name: fawkes:www, id: -33555072, owner: fawkes, creator: fawkes,
    +  membership: 2, flags: S-M--, group quota: 0.
    +fawkes@dr-wily:~$ pts mem fawkes:www
    +Members of fawkes:www (id: -33555072) are:
    +  jflorey
    +  sipbtest
    +fawkes@dr-wily:~$ 
    +

    +

    -This method will work, but at MIT, it is much more common to use moira lists as a group. To create a new list, use the web interface at https://wserv.mit.edu:444/fcgi-bin/lc? to create a moira list, NOT A MAILMAN LIST and be sure to check the box for "Should this list be an AFS Group?" (to make an already existing moira list into an AFS group simply blanche -G <list>). After the servers update (which may take anywhere between 1 second and 10 minutes depending on the number of similar requests), the AFS group system:<list name> will exist in the athena.mit.edu cell. +This method will work, but at MIT, it is much more common to use moira lists as groups. To create a new list, use the web interface at http://wserv.mit.edu/lc to create a moira list, NOT A MAILMAN LIST and be sure to check the box for "Is this list an AFS group?" (to make an already existing moira list into an AFS group simply run blanche -G <list> on athena). After the servers update (which may take anywhere between 1 second and 10 minutes depending on the number of similar requests), the AFS group system:<list name> will exist in the athena.mit.edu cell.

    +

    To check the membership of a moira list simply run blance <list name>. Note that the name of the list does not have the system: that is in the string. (e.g. to view the membership of the list gnu aka the group system:gnu, run blanche gnu). +

    -This is useful because one often wants the same certain people who can operate on files in a folder to be a mailing list. Thus, for example, it is possible to send mail to gnu@mit.edu and use system:gnu as an AFS group on ACLs. (it is also possible to make moira lists that are AFS groups, but not mailing lists). +This is useful because one often wants the same certain people who can operate on files in a folder to be a mailing list. Thus, for example, it is possible to send mail to gnu@mit.edu and use system:gnu as an AFS group on ACLs. (it is also possible to make moira lists that are AFS groups, but not mailing lists).

    see also: man 1 pts @@ -194,63 +219,60 @@ see also: man 1 pts

    Controlling Access from the Web

    -If you make a directory listable and readable by system:anyuser then it can be viewed by any user on the web via the urls mentioned above +If you make a directory listable and readable by system:anyuser then it can be viewed by any user on the web via the URLs mentioned above

    -Unfortunately, just because you add specific users to an AFS ACL does not mean they can see the folder when the access from the web. IS&T, however, does provide a solution to this. First, make sure that the wanted directory is not readable by system:anyuser. Next fs sa <dir> system:htaccess.mit read . Then create a file named .htaccess.mit in that directory. In that file you can do three things, +Unfortunately, adding specific users to an AFS ACL does not mean they can see the folder when they access it from the web. IS&T, however, does provide a solution to this. First, make sure that the wanted directory is not readable by system:anyuser. Create a file named .htaccess.mit in that directory. In that file you can do one of three things,

    • You can require that the user have valid certificates: -
        <limit GET>
      -  require valid-user
      -  </limit>
      +
      require valid-user
       
    • You can require the reader be (a) specific user(s), for example: -
        <limit GET>
      -  require user fawkes jflorey siptest jarandom
      -  </limit>
      -
      +
      require user fawkes jflorey sipbtest jarandom
       
    • You can require that the reader be a member of one of certain moira groups (notice these are moira groups, there is no "system:". For example: -
       <limit GET>
      -  require group sipb-staff sipb-prospectives
      - </limit>
      +
      require group sipb-staff sipb-prospectives
       

    -There after the users should be able to get to the folders at http'''s'''://web.mit.edu/<locker>/<path to folder> if they have certificates and no one should be able to reach it via http. Make sure to add yourself if you are going to be accessing it. +

    Note that you cannot mix users and groups in the same directory

    . + +

    Finally fs sa <dir> system:htaccess.mit read .

    + +Thereafter, the users should be able to get to the folders at https://web.mit.edu/<locker>/<path to folder> if they have certificates and no one should be able to reach it via http. Make sure to add yourself if you are going to be accessing it.

    -see also: http://web.mit.edu/is/web/reference/web-resources/https.html +see also: http://ist.mit.edu/services/web/reference/web-resources/https

    Troubleshooting

    -

    I'm trying to access my files, fs la says I should have permissions here, but it still says : Permission denied

    +

    I'm trying to access my files, fs la says I should have permissions here, but it still says : Permission denied

    -There are two likely possibilities. First, its likely that your tokens may have expired. To get new tokens, make sure you have valid kerberos tickets and then run aklog. Another possibility is that you have tokens but not for the correct cell. tokens will tell you what tokens you already have. In all likelihood, if you are reading this, you probably want aklog athena sipb. Finally, a third possibility is that your group membership has changed since you acquired tokens. Try running aklog -force +There are two likely possibilities. First, its likely that your tokens may have expired. You can check this by running tokens. If they are, in fact, expired (or missing) get new tokens as follows: first, make sure you have valid kerberos tickets and then run aklog. Another possibility is that you have tokens but not for the correct cell. tokens will tell you what tokens you already have. In all likelihood, if you are reading this, you probably want aklog athena sipb. Finally, a third possibility is that your group membership has changed since you acquired tokens. Try running aklog -force

    I'd really prefer that not everyone could list my files, how should I stop this?

    -What you don't want to do is take away the l permission from system:anyuser because then no one will be able to get to your Public or www which you probably don't want. All told, there is no way to stop people on Athena from listing your files (unless you keep everything under Private or a similarly restricted directory). It is rather trivial to stop web users, however. A solution I recommend, if you have a website in www, is to make a page redirect.html (or similar) with the contents like: +What you don't want to do is take away the l permission from system:anyuser because then no one will be able to get to your Public or www which is probably not what you want want. All told, there is no way to stop people on Athena from listing your files (unless you keep everything under Private or a similarly restricted directory). It is rather trivial to stop web users, however. A solution I recommend, if you have a website in www, is to make a redirect.html (or similar) with the contents like:

    <html>
     <head>
    -  <meta http-equiv="Refresh" content="0; url=http://www.mit.edu/~<lockername>/">
    +  <meta http-equiv="Refresh" content="0; url=http://web.mit.edu/<lockername>/www">
     </head>
     <body>
    -  <p>Please go to my <a href="http://www.mit.edu/~<lockername>/">www</a>!</p>
    +  <p>Please go to my <a href="http://web.mit.edu/<lockername>/www">www/a>!</p>
     
     </body>
     </html>
     

    -and then doing a ln -s www/redirect.html index.html in the top level of the locker. Alternatively you can link index.html to a blank file in your Public or www (you can't simply it in the top level because its not readable there, this will result in a 403 Forbidden error). +and then doing a ln -s www/redirect.html index.html in the top level of the locker. Alternatively you can link index.html to a blank file in your Public or www (you can't simply put it in the top level because it's not readable there; this will result in a 403 Forbidden error).

    It was around 6am on a Sunday morning and suddenly I couldn't access my files

    -Yeah, most AFS servers restart weekly at 6 AM on Sunday. +Most AFS servers restart weekly at 6 AM on Sunday.

    It isn't Sunday and I can't get to my files

    -There may be a non-scheduled AFS outage. Check 3down, hopefully it will be back up soon :-(. +There may be a non-scheduled AFS outage. Check 3down, hopefully it will be back up soon :-(. You can check up on the AFS servers by running fs checkservers (or fs checks). If there is no reported outage and you can't access the AFS servers (but can access the rest of the net), contact OLC.

    Advanced Tasks

    Putting Software in a Locker

    @@ -259,25 +281,22 @@ The Athena environment was designed to allow software to run on several architec

    -The special variable @sys in a path corresponds to the first of these that matches. When a user runs add, /mit/<locker>/arch/@sys/bin is added to their PATH and /mit/<locker>/man or /mit/<locker>/arch/@sys/man is added to their MANPATH. +The special variable @sys in an AFS path corresponds to the first of these that matches. When a user runs add, /mit/<locker>/arch/@sys/bin is added to their PATH and /mit/<locker>/man or /mit/<locker>/arch/@sys/man is added to their MANPATH.

    -Primarily because of this fact, a typical locker is set up with the following sort of layout: +Primarily because of this fact, a typical locker might be set up with the following sort of layout:

    user@host:/mit/<some locker>$ tree -dL 2
     .
     |-- arch
    -|   |-- i386_linux22
    -|   |-- i386_linux24 -> i386_rhel4/
    +|   |-- i386_linux24
     |   |-- i386_rh9 -> i386_linux24
    -|   |-- i386_rhel3 -> i386_rhel4
     |   |-- i386_rhel4
     |   |-- sgi_65
    -|   |-- sun4x_510
    -|   |-- sun4x_57
    -|   |-- sun4x_58 -> sun4x_510/
    -|   `-- sun4x_59 -> sun4x_58
    +|   |-- sun4x_58
    +|   |-- sun4x_59 -> sun4x_518/
    +|   `-- sun4x_510 -> sun4x_59
     |-- bin -> arch/@sys/bin
     |-- include -> arch/@sys/include
     |-- info
    @@ -286,8 +305,8 @@ Primarily because of this fact, a typical locker is set up with the following so
        `-- man1
     
     

    -Please note that the @sys variable should really be used ONLY in these convenience in symlinks. To make the folder in arch you usually want to either pick a more general architecture and set it up yourself or simply use the following command in the top level of the locker: -mkdir -p arch/$ATHENA_SYS/{bin,lib,include}. After making this folder, make the shortcut symlinks with commands similar to ln -s arch/@sys/bin bin (again, this is the ONLY time you should use @sys yourself). Other than that, its mainly just a matter of making sure to run configure with options like --prefix=/mit/<lockername> and --with-manpath=/mit/<lockername>/man. +Please note that the @sys variable should really be used ONLY in these convenience symlinks. To make the folder in arch you usually want to either pick a more general architecture and set it up yourself or simply use the following command in the top level of the locker: +mkdir -p arch/$ATHENA_SYS/{bin,lib,include}. After making this folder, make the shortcut symlinks with commands similar to ln -s arch/@sys/bin bin (again, this is the ONLY time you should use @sys yourself). Other than that, it's mainly just a matter of making sure to run configure with options like --prefix=/mit/<lockername> and --with-manpath=/mit/<lockername>/man.

    @@ -299,6 +318,27 @@ See also: man lockers While it is easily possible to make an AFS group for yourself, it is harder to get a new locker. If you want a locker for something official like a student group or a piece of software, send an email to accounts@mit.edu explaining what you need it for and someone at IS&T will let you know. If you feel it is something less official or you would just rather consider it a SIPB project (perhaps get other SIPB people involved) send an email to sipb-afsreq@mit.edu instead. In either case, if a mount point under /mit fails to get set up for you, let hesreq@mit.edu know.

    +

    Find out technical information about my locker

    +

    Figure out the volume name of the locker. One way to do this is to run fs lq . in the directory and look in the left column. Once you have the volume name, run vos examine <volume name>. This will tell you information such as what server it is located on, its ID numbers, when it was last accessed, when it was last backed up, etc. For example: +

    +$ vos examine user.sipbtest
    +user.sipbtest                     537058147 RW      69785 K  On-line
    +    PHLEGETHON.MIT.EDU /vicepa 
    +    RWrite  537058147 ROnly          0 Backup  537058149 
    +    MaxQuota    2000000 K 
    +    Creation    Sat Nov  2 21:29:46 1996
    +    Copy        Fri Dec  4 19:38:29 2009
    +    Backup      Sat Mar 20 02:07:55 2010
    +    Last Update Wed Mar  3 00:22:34 2010
    +    35 accesses in the past day (i.e., vnode references)
    +
    +    RWrite: 537058147     Backup: 537058149 
    +    number of sites -> 1
    +       server PHLEGETHON.MIT.EDU partition /vicepa RW Site
    +
    + +vos has many other commands, but most of them are only useful to administrators. +

    See Also

    SIPB's older guide, Inessential AFS
    OpenAFS documentation at http://www.openafs.org/