Scratch Dcache

• Introduction
• Quick Start
• More on dCache
• Access Protocols

Introduction

There is also a tape-backed dCache pool, not addressed in this page, which is written through an upload process and read through SAM

Quick Start

mkdir /pnfs/mu2e/scratch/users/$USER

setup mu2e
setup ifdhc
ifdh cp local-file /pnfs/mu2e/scratch/users/$USER/new-file-name

ifdh cp local-file /pnfs/mu2e/scratch/users/$USER/

ifdh cp /pnfs/mu2e/scratch/users/$USER/file .

Wildcards are not allowed - move one file per command. Try to keep the number of files in a directory under 1000 and to avoid excessive numbers of small files, or frequent renaming of files.

/pnfs is not a directory, it is an interface to a database implemented as an nfs server, and there are restrictions. You can use the following commands in /pnfs: ls, rm, mv, mkdir, rmdir, chmod, cp, cat, more, less, etc. On SL5 (mu2egpvm04) you can't use the following commands: cp, cat, more, less, etc. You shouldn't run commands which make large demands on the database: "find .", "ls -lr *"

More on dCache

You can see basic metadata about the file in dCache though the /pnfs/mu2e/scratch filesystem. All files in dCache will appear here. /pnfs looks like a file system but is actually an nfs server with the database of dCache files as a backend. It is mounted on many lab machines, but only as needed. There are many subdirectories possible under /pnfs, but only a selected few are mounted on any particular machine, so we only see /pnfs/mu2e.

dCache instances can come in read, write, or read/write forms. Write-only is usually used to get raw data to tape. Read-only might be used to read fixed datasets. The most common is read/write, like /pnfs/mu2e/scratch which acts like a scratch disk. There are also tape-backed versions, where all files written to the dCache are migrated to tape and can be migrated off tape and into the dCache again on demand.

When you issue commands to read or write data on dcache you have to use one of several protocols. We will usually use the ifdh script and allow it to pick the most logical protocol. In any protocol, the procedure starts with contacting a head node with the request in the form of a reading or writing a /pnfs file spec. The head node is a load-balancer which directs you to a "door" process on a server. The door looks up your request and determines how to satisfy it. The door passes your request to a queue on a disk server node to actually serve or receive the file. Your transfer protocol request will hang until it gets through the queue and complates the transfer. Since the transfer is coming from the disk server node to your machine, dCache is providing the throughput capability of all the disk server nodes, typically dozens of Gb connections. A dCache can be tuned in the number and size of servers, the number and of type of door and queue and other parameters.

dCache has a file access load balancing system. If it detects that some files are frequently requested, it can spread those files to many severs to greatly increase the overall throughput for these files. It can keep multiple copies of a file to help with frequent access or as a planned backup mechanism. dCache regularly checks all file checksums and can gracefully handle local hardware failures without stopping the rest of the system.

Access Protocols

• dcap the native protocol
  

  dccp /pnfs/mu2e/scratch/users/$USER/filename .
  

  dccp is installed on mu2egpvm, but it may have to be installed or setup (the product is named dcap) on other nodes
  
  
• root protocol

  on SL5:
  export LD_PRELOAD=/usr/lib64/libpdcap.so.1
  root [0] ff = TFile::Open("/pnfs/mu2e/scratch/users/$USER/file")
  on SL6:
  root [0] ff = TFile::Open("/pnfs/mu2e/scratch/users/$USER/file")
  

  There are other versions of this access, through different plugins which trigger different authentication, protocols and transfer queues.
  
  
• webdav There is CS-doc-5050reference
  

  curl -1 -L  --cacert $X509_USER_PROXY  --capath /etc/grid-security/certificates  --cert /tmp/x509up_u1311  https://fndca1.fnal.gov:2880/pnfs/fnal.gov/usr/mu2e/scratch/users/rlc/s1
  
  (not currently working)
  

• gridFtp
  

  kinit
  getcert
  export X509_USER_CERT=/tmp/x509up_u`id -u`
  export X509_USER_KEY=$X509_USER_CERT
  export X509_USER_PROXY=$X509_USER_CERT
  grid-proxy-init
  voms-proxy-init -noregen -rfc -voms fermilab:/fermilab/mu2e/Role=Analysis
  globus-url-copy  gsiftp://fndca1.fnal.gov:2811/scratch/users/$USER/file-name file:///$PWD/file-name
  
  

Security, Privacy, Legal