mu2epro

• Introduction
• Kerberos
• Certificate
• Setup scripts
• Jobsub and ifdh
• Cron

Introduction

ssh -k mu2epro@nodename (-k is important!)

Kerberos

In 8/2016 we asked the service desk to reset the keytab passwords and then ran the procedure /usr/krb5/config/make-cron-keytab This moves the keytab to a secure system location under /var. This solves the security problem of using the keytab from a network-mounted disk, exposing the kerberos data to the network.

To get a new ticket, you run

/usr/krb5/bin/kcron

A cron job on all of the interactive machines runs every 4 hours, runs a kcron command and writes the ticket to /tmp/krb5cc_44592_auto. As long as you use this ticket location, you will always have a valid ticket.

Using the keytab with

kinit -k -t $HOME/keytab/mu2egpvm01.keytab mu2epro/cron/mu2egpvm01.fnal.gov@FNAL.GOV

It is important to setup how the ticket is located. If you do not have the KRB5CCNAME environmental set when you run kcron, the ticket will go to /tmp/krb5cc_44592cronXXXXX and no other utility will find it, not even klist, so you will always want to have KRB5CCNAME set. What follows is how it gets set.

Ideally users should

ssh -k mu2epro@nodename

If you forward your ticket (not recommended):

ssh mu2epro@nodename

KRB5CCNAME=FILE:/tmp/krb5cc_44592_XXXXXXXXXX

Anyone can get their personal principal back with

kinit your_user_name@FNAL.GOV

Bottom line, just ssh -k and only use the default mu2epro automatic tickets!

Andrei has an ssh config setup that will effectively set "-k" for you automatically. Add this text block to $HOME/.ssh/config

#----------------
Host mu2epro
User mu2epro
HostName mu2egpvm01.fnal.gov
ForwardAgent no
GSSAPIDelegateCredentials no
#----------------

ssh mu2epro

Certificate

The cert has been copied to a Computing Division server. There, it is converted to a VOMS proxy every 8 hours and that proxy is copied in a secure way to

/opt/mu2epro/mu2epro.Production.proxy

Andrei has installed the actual certificate in the /opt area too. We believe the /opt area is a local disk so this will not expose the cert over the network. This will allow scripts to authenticate https communication with the SAM server. There may be other methods to authenticate, but the concern is that they will time-out.

Setup scripts

The login file .bash_profile defines KRB5CCNAME to point to the mu2epro automatic kerberos ticket. It also makes sure /usr/krb5/bin and ~/bin are in the path, defines some environmentals which allow you to use the mu2epro authentication for commands, and aliases for setting further things up.

.bashrc defines aliases for personalization for operators and for projects (see next). It also runs

/cvmfs/fermilab.opensciencegrid.org/products/common/etc/setups

After logging in, the user can chose their environment. You can set your preferences through a personal setup script alias such as "setup_rlc". Please see

~mu2epro/bin/setup_alias.sh

export OPERATOR=rlc

In addition, you can run the setup for a project, such as "setup_upload". These should set the environmental MUE2PRO_PROJECT to a unique value. This script should then setup all environment needed for the project. This should be complete, but not include any enviroment that is not necessary to run the project. Necessary setups might include "setup mu2e", offline version setup, jobsub, ifdhc and useful environmentals, such as pointers to scratch disk directories.

If the environment for a project must be strictly controlled, then you would only setup the project, and this setup should overwrite the value of OPERATOR to stop personalized .bashrc behaviour.

You should write project scripts which only depend on setup_project and with alias-defeating /bin/mv etc to avoid differences in preferences. A final useful convention would be to keep project scripts in ~/projectname.

There are also hooks in .bashrc to add behaviour as needed.

source ~/bin/setup_alias.sh
[ "$OPERATOR" == "rlc"      ] && source ~rlc/bin/setup_rlc.sh rc
...
[ "$MUE2PRO_PROJECT" == "upload" ] && source ~/bin/setup_upload.sh rc

cron jobs should do all their setup explicitly, which should include sourcing these setup scripts.

Jobsub and ifdh

ifdhc with version less than 1_7 may not work to transfer files, it may generate an incorrect gridftp url resulting in globus errors. If you write to bluearc, you will need to force the transfer through gridftp to get the permissions to work.

ifdh cp --force=gridftp file /mu2e/data/..

ifdh rm /mu2e/data/.. --force=gridftp

jobsub needs to be pointed to the OSG cert to work.

export X509_USER_PROXY=/tmp/x509up_u44592

setup_mu2epro

As of jobsub_client v1_1_0_2, the --role switch is needed for submit, rm, and fetchlog. It is not needed for q and history, which list the jobs as user mu2epro.

Note that when a user becomes mu2epro, they may still have their personal kerberos ticket active, and that may be used accidentally to authenticate the submission, so the user should kdestroy their ticket. Please see the discussion of kerberos and the setup commands below. If there is no user kerberos ticket, the job will be authenticated by the proxy.

jobsub_submit -G mu2e --role=Production -N 1 \
  --generate-email-summary --email-to=rlc@fnal.gov  --OS=SL6 \
  --resource-provides=usage_model=DEDICATED  \
  file:///mu2e/app/users/rlc/jobs/remote.bash

"ifdh cp --force=expftp" will not work with the service cert, i.e. interactively. It works on the grid because a different grid-based cert is used there.

Cron jobs

Backup

Security, Privacy, Legal