Mu2e Home
Scratch Dcache


Scratch dCache is a system of disks aggregated across many servers, and accessed by several possible protocols through an interface which looks liek a file system. The disk space is many TB, shared between the IF experiments and must be treated as scratch - least-used files are purged if the space approaches full. It is possible to prevent a file from being purged by "pinning" it, but only in special cases.

There is also a tape-backed dCache pool, not addressed in this page, which is written through an upload process and read through SAM

Quick Start

On detsim and mu2egpvm*, you can create your area in the scratch dCache:
mkdir /pnfs/mu2e/scratch/users/$USER
and copy files in (also from grid nodes). You will need a kerberos ticket.
setup mu2e
setup ifdhc
ifdh cp local-file /pnfs/mu2e/scratch/users/$USER/new-file-name
or, to leave the file name the same (trailing slash required)
ifdh cp local-file /pnfs/mu2e/scratch/users/$USER/
You can copy files out:
ifdh cp /pnfs/mu2e/scratch/users/$USER/file .

Wildcards are not allowed - move one file per command. Try to keep the number of files in a directory under 1000 and to avoid excessive numbers of small files, or frequent renaming of files.

/pnfs is not a directory, it is an interface to a database implemented as an nfs server, and there are restrictions. You can use the following commands in /pnfs: ls, rm, mv, mkdir, rmdir, chmod, cp, cat, more, less, etc. On SL5 (mu2egpvm04) you can't use the following commands: cp, cat, more, less, etc. You shouldn't run commands which make large demands on the database: "find .", "ls -lr *"

More on dCache

dCache is a distributed disk system developed orginally at DESY, now maintained by Fermilab and DESY. It is used at many HEP sites around the world. It has organization home page and a monitor for our instance.

You can see basic metadata about the file in dCache though the /pnfs/mu2e/scratch filesystem. All files in dCache will appear here. /pnfs looks like a file system but is actually an nfs server with the database of dCache files as a backend. It is mounted on many lab machines, but only as needed. There are many subdirectories possible under /pnfs, but only a selected few are mounted on any particular machine, so we only see /pnfs/mu2e.

dCache instances can come in read, write, or read/write forms. Write-only is usually used to get raw data to tape. Read-only might be used to read fixed datasets. The most common is read/write, like /pnfs/mu2e/scratch which acts like a scratch disk. There are also tape-backed versions, where all files written to the dCache are migrated to tape and can be migrated off tape and into the dCache again on demand.

When you issue commands to read or write data on dcache you have to use one of several protocols. We will usually use the ifdh script and allow it to pick the most logical protocol. In any protocol, the procedure starts with contacting a head node with the request in the form of a reading or writing a /pnfs file spec. The head node is a load-balancer which directs you to a "door" process on a server. The door looks up your request and determines how to satisfy it. The door passes your request to a queue on a disk server node to actually serve or receive the file. Your transfer protocol request will hang until it gets through the queue and complates the transfer. Since the transfer is coming from the disk server node to your machine, dCache is providing the throughput capability of all the disk server nodes, typically dozens of Gb connections. A dCache can be tuned in the number and size of servers, the number and of type of door and queue and other parameters.

dCache has a file access load balancing system. If it detects that some files are frequently requested, it can spread those files to many severs to greatly increase the overall throughput for these files. It can keep multiple copies of a file to help with frequent access or as a planned backup mechanism. dCache regularly checks all file checksums and can gracefully handle local hardware failures without stopping the rest of the system.

Access Protocols

While we will typically use ifdh to read or write files, that script will use one of the dCache supported protocols. Each protocol may several have authentication options. Not all protocols have the same file transfer capacity. Any parallel access, such as from a grid job, must use "ifdh cp" to avoid overloads! Some of the more interesting protocols are:

Fermilab at Work ]  [ Mu2e Home ]  [ Mu2e @ Work ]  [ Mu2e DocDB ]  [ Mu2e Search ]

For web related questions:
For content related questions:
This file last modified Thursday, 07-May-2015 16:20:30 CDT
Security, Privacy, Legal Fermi National Accelerator Laboratory