The directory structure and symlinking

The first thing that VDAT performs when launched is to symlink the files from the raw directory to the redux directory, changing the way the files are organised. First we describe how the files are (should be) organized in the raw directories and how VDAT reorganises them in the redux directories. Then we will describe how the symlinking works and how the user can interact with it.

Raw directory structure

The path to the raw directory can be provided via the configuration option rawdir in the [general] section of the main VDAT configuration file or via the vdat positional command line arguments.

VDAT assumes that the raw directory has the following structure and that rawdir points to the directory just containing the dates:

├── 20120301                                                 # Date of the start of the night
│   ├── virus                                                # instrument name
│   │   ├── virus0003000                                     # shot ID
│   │   │   ├── exp01                                        # science dithers 1/3
│   │   │   │   └── virus
│   │   │   │       ├── 20120301T002848_014LL_sci.fits
│   │   │   │       ├── [...]
│   │   │   │       └── 20120301T002848_065RU_sci.fits
│   │   │   ├── [...]
│   │   │   └── exp03                                        # science dithers 3/3
│   │   │       └── virus
│   │   │           ├── 20120301T004248_014LL_sci.fits
│   │   │           ├── [...]
│   │   │           └── 20120301T004248_065RU_sci.fits
│   │   ├── virus0003001                                     # shot ID
│   │   │   ├── exp01                                        # bias frames
│   │   │   │   └── virus
│   │   │   │       ├── 20120301T000000_014LL_zro.fits
│   │   │   │       ├── [...]
│   │   │   │       └── 20120301T000000_065RU_zro.fits
│   │   │   ├── [...]
│   │   │   └── expNN                                        # NN exposures
│   │   │       └── virus
│   │   │           ├── 20120301T001000_014LL_zro.fits
│   │   │           ├── [...]
│   │   │           └── 20120301T001000_065RU_zro.fits
│   │   ├── virus0003002                                     # shot ID
│   │   │   ├── exp01                                        # flat frames
│   │   │   │   └── virus
│   │   │   │       ├── 20120301T001200_014LL_flt.fits
│   │   │   │       ├── [...]
│   │   │   │       └── 20120301T001200_065RU_flt.fits
│   │   │   └── expNN                                        # NN exposures
│   │   │       └── virus
│   │   │           ├── 20120301T002000_014LL_flt.fits
│   │   │           ├── [...]
│   │   │           └── 20120301T002000_065RU_flt.fits
│   │   └── virus0003002                                     # shot ID
│   │       ├── exp01                                        # comp/arc frames
│   │       │   └── virus
│   │       │       ├── 20120301T002200_014LL_cmp.fits
│   │       │       ├── [...]
│   │       │       └── 20120301T002200_065RU_cmp.fits
│   │       ├── [...]
│   │       └── expNN                                        # NN exposures
│   │           └── virus
│   │               ├── 20120301T003000_014LL_cmp.fits
│   │               ├── [...]
│   │               └── 20120301T003000_065RU_cmp.fits
│   ├── gc1                                                  # guide probe data, ignored
│   │   └── [...]
│   ├── gc2                                                  # guide probe data, ignored
│   │   └── [...]
│   ├── wf1                                                  # wave front sensor, ignored
│   │   └── [...]
│   └── wf2                                                  # wave front sensor, ignored
│       └── [...]
└── 20120302                                                 # Date of the following night
    └── [...]                                                # as before

Early versions of the data might miss the virus directory, the instrument name, just below the nights. See Configuration for more info.

Redux directory structure

The name of the redux directory is taken from the redux_dir option in the [general] section of the main VDAT configuration file.

The structure of the redux directory is the following:

├── vdat.db
├── 20120301
│   ├── cal
│   │   └── 20151025_122009
│   │        ├── 20120301T001200_014LL_flt.fits
│   │        ├── [...]
│   │        ├── 20120301T001200_065RU_flt.fits
│   │        ├── 20120301T002000_014LL_flt.fits
│   │        ├── [...]
│   │        ├── 20120301T002000_065RU_flt.fits
│   │        ├── 20120301T002200_014LL_cmp.fits
│   │        ├── [...]
│   │        ├── 20120301T002200_065RU_cmp.fits
│   │        ├── 20120301T003000_014LL_cmp.fits
│   │        ├── [...]
│   │        └── 20120301T003000_065RU_cmp.fits
│   ├── sci
│   │   └── M37SIM-00003-obs-1
│   │       ├── 20120301T002848_014LL_sci.fits
│   │       ├── [...]
│   │       ├── 20120301T002848_065RU_sci.fits
│   │       ├── 20120301T004248_014LL_sci.fits
│   │       ├── [...]
│   │       └── 20120301T004248_065RU_sci.fits
│   └── zro
│       └── 20151025_120252
│           ├── 20120301T000000_014LL_zro.fits
│           ├── [...]
│           ├── 20120301T000000_065RU_zro.fits
│           ├── 20120301T001000_014LL_zro.fits
│           ├── [...]
│           └── 20120301T001000_065RU_zro.fits
└── 20120301

Symlinking

In this phase the files from Raw directory structure are symlinked into Redux directory structure and a database, used to drive the GUI, is created.

As in the raw directory, the data are divided per night, but inside each night they are reorganized by their type.

  • The science frames belonging to the same virus shot are organized in a subdirectory of the sci directory, called as the value of the OBJECT header keyword, with white spaces replaced by underscores. If the keyword is not present or holds multiple values for the files belonging to one shot the symlinking will fail. If the OBJECT keyword is not populated, the directory name 'no_object' is used. If multiple shots in the same night have the same OBJECT, the directory name is the value of the keyword followed by "_sym" and a three digit counter.
  • The bias (a.k.a. zero) frames belonging to one shot are grouped together in a subdirectory of zro, whose name is the average time stamp of all of the files.
  • The calibration frames, namely flats and arcs (a.k.a. comps), are grouped together in pairs in subdirectories of cal. The name of each subdirectory is the average time stamp of the flats or arc/comp files it contains. The rationale for the grouping is that the flat and the arcs need to be combined to compute distortion and fiber model.

All the files are symlinked in the above subdirectories without replicating the exp??/virus structure.

When symlinking the files, two metadata files are created in each directory: shot_name.txt and exposure_names.txt. The former one contains information about the type of files and the original shot directories; the latter contains a mapping between exposure number.

If a redux directory is found, VDAT uses these files to rebuild the internal database, thus they should never be removed without removing the directory too. In this way is possible to re-run VDAT without performing the initial symlink.

After the symlink is finished, VDAT attempts to find a zro and a cal directory for each sci directory. This is used as a source of calibration frames for the reduction. As a default VDAT chooses the closest directories in time, but the zro and cal directories can also be chosen by the user via the GUI.

Configuration

It’s possible for the user to customize some part of the symlinking, e.g. providing new directory name matches, via the main VDAT configuration file, vdat_setting.cfg.

In the rest of the section we will describe the name of the available options, divided per section in the file.

[general]

  • rawdir: path to the raw directory/ies or to the night(s) to symlink; it is overridden by the vdat positional arguments.
  • redux_dir: path to the redux directory.
  • is_rawdir_night: if true vdat interprets the rawdir entries as being already night directories. If not found defaults to no. Can be overridden with the -N/--is-rawsir-night command line option.

[symlink]

  • night: generic name of the night directories. If, e.g. the night directory name is a eight digit number, the value would be: nights = */[0-9]{8}; ignored if is_rawdir_night is given.
  • is_night_regex: if the value is any of yes/true/on/1 the value of night is interpreted as a regex expression, if it’s any of no/false/off/0 or not present, the night is interpreted as described in fnmatch.
  • virus_instrument: name of the instrument; it is the name of the directory between the night and the shot; if not found, defaults to ‘virus’. If it’s given but empty, the directory structure becomes night/shot. Can be overridden with the -i/--virus-instrument command line option.
  • virus_shot: generic name for the virus shot directories. Very likely will be virus_shot = *virus[0-9]*.
  • is_virus_shot_regex: as is_night_regex but for the virus shot entry.
  • virus_fits_files: name of the fits file to search; e.g.: */virus/*.fits
  • is_virus_fits_files_regex: as is_night_regex but for the virus fits file
  • relative_symlink: decide whether to do an absolute or a relative symlink
  • replace_symlink: if a symlink exists, remove and recreate it
  • image_type_pattern: regex pattern to use to extract the image type from the file name in the form of pattern, replacement. For example .*_(\w{3})\.fits, \1 extract the last three letters before the file
  • datetime_pattern: regex pattern to use to extract the date and time from the file name in the form of pattern, replacement. For example .*/(\d{8}T\d{6})_.*\.fits, \1 extract eight number, a T and other six number at the beginning of the file name
  • datetime_fmt: format of the date and time extracted wit the above pattern; e.g.: %Y%m%dT%H%M%S
  • object_key: name of the header keyword to use to extract the object for the science frames; default OBJECT
  • cal_types: list of comma separated image types that should be collected together; cal_types = flt, cmp collects flat and arc frames, while cal_types = twi, cmp collects twilights and arc frames. The option can be overridden with the --cal-types command line option.
  • cal_type_header_typ: when dealing with the calibration type typ get the value of the header keyword cal_type_header_typ and use it as object for that type. This allows the collection multiple shots of the same typ in the same calibration directory as long as the header keyword has different values. The cmp types have different lamp types, whose names are saved in the OBJECT header keyword, so by default VDAT uses cal_type_header_cmp = OBJECT
  • max_delta_cal: a flat and an arc per lamp_type shot are collected in the same redux directory if their average time stamps differs less second than:

Unknown or nonstandard types can be also symlinked, as long as the operation can be carried out as if the types were sci or zro. At the moment there is no support for nonstandard cal like types. To declare that an unknown type, e.g. drk can be symlinked as if it were a known one, e.g. zro add to the [symlink] section the following option:

drk_symlink_as = zro