The directory structure and symlinking ************************************** The first thing that VDAT performs when launched is to symlink the files from the ``raw`` directory to the ``redux`` directory, changing the way the files are organised. First we describe how the files are (should be) organized in the :ref:`raw directories ` and how VDAT reorganises them in the :ref:`redux directories `. Then we will describe how the :ref:`symlinking ` works and how the user can interact with it. .. _raw_dir: Raw directory structure ======================= The path to the raw directory can be provided via the configuration option ``rawdir`` in the ``[general]`` section of the main VDAT configuration file or via the ``vdat`` positional command line arguments. VDAT assumes that the raw directory has the following structure and that ``rawdir`` points to the directory just containing the dates:: ├── 20120301 # Date of the start of the night │ ├── virus # instrument name │ │ ├── virus0003000 # shot ID │ │ │ ├── exp01 # science dithers 1/3 │ │ │ │ └── virus │ │ │ │ ├── 20120301T002848_014LL_sci.fits │ │ │ │ ├── [...] │ │ │ │ └── 20120301T002848_065RU_sci.fits │ │ │ ├── [...] │ │ │ └── exp03 # science dithers 3/3 │ │ │ └── virus │ │ │ ├── 20120301T004248_014LL_sci.fits │ │ │ ├── [...] │ │ │ └── 20120301T004248_065RU_sci.fits │ │ ├── virus0003001 # shot ID │ │ │ ├── exp01 # bias frames │ │ │ │ └── virus │ │ │ │ ├── 20120301T000000_014LL_zro.fits │ │ │ │ ├── [...] │ │ │ │ └── 20120301T000000_065RU_zro.fits │ │ │ ├── [...] │ │ │ └── expNN # NN exposures │ │ │ └── virus │ │ │ ├── 20120301T001000_014LL_zro.fits │ │ │ ├── [...] │ │ │ └── 20120301T001000_065RU_zro.fits │ │ ├── virus0003002 # shot ID │ │ │ ├── exp01 # flat frames │ │ │ │ └── virus │ │ │ │ ├── 20120301T001200_014LL_flt.fits │ │ │ │ ├── [...] │ │ │ │ └── 20120301T001200_065RU_flt.fits │ │ │ └── expNN # NN exposures │ │ │ └── virus │ │ │ ├── 20120301T002000_014LL_flt.fits │ │ │ ├── [...] │ │ │ └── 20120301T002000_065RU_flt.fits │ │ └── virus0003002 # shot ID │ │ ├── exp01 # comp/arc frames │ │ │ └── virus │ │ │ ├── 20120301T002200_014LL_cmp.fits │ │ │ ├── [...] │ │ │ └── 20120301T002200_065RU_cmp.fits │ │ ├── [...] │ │ └── expNN # NN exposures │ │ └── virus │ │ ├── 20120301T003000_014LL_cmp.fits │ │ ├── [...] │ │ └── 20120301T003000_065RU_cmp.fits │ ├── gc1 # guide probe data, ignored │ │ └── [...] │ ├── gc2 # guide probe data, ignored │ │ └── [...] │ ├── wf1 # wave front sensor, ignored │ │ └── [...] │ └── wf2 # wave front sensor, ignored │ └── [...] └── 20120302 # Date of the following night └── [...] # as before Early versions of the data might miss the ``virus`` directory, the instrument name, just below the nights. See :ref:`symlink_conf` for more info. .. _redux_dir: Redux directory structure ========================= The name of the redux directory is taken from the ``redux_dir`` option in the ``[general]`` section of the main VDAT configuration file. The structure of the redux directory is the following:: ├── vdat.db ├── 20120301 │ ├── cal │ │   └── 20151025_122009 │ │ ├── 20120301T001200_014LL_flt.fits │ │ ├── [...] │ │ ├── 20120301T001200_065RU_flt.fits │ │ ├── 20120301T002000_014LL_flt.fits │ │ ├── [...] │ │ ├── 20120301T002000_065RU_flt.fits │ │ ├── 20120301T002200_014LL_cmp.fits │ │ ├── [...] │ │ ├── 20120301T002200_065RU_cmp.fits │ │ ├── 20120301T003000_014LL_cmp.fits │ │ ├── [...] │ │ └── 20120301T003000_065RU_cmp.fits │ ├── sci │ │   └── M37SIM-00003-obs-1 │ │ ├── 20120301T002848_014LL_sci.fits │ │ ├── [...] │ │ ├── 20120301T002848_065RU_sci.fits │ │ ├── 20120301T004248_014LL_sci.fits │ │ ├── [...] │ │ └── 20120301T004248_065RU_sci.fits │ └── zro │ └── 20151025_120252 │ ├── 20120301T000000_014LL_zro.fits │ ├── [...] │ ├── 20120301T000000_065RU_zro.fits │ ├── 20120301T001000_014LL_zro.fits │ ├── [...] │ └── 20120301T001000_065RU_zro.fits └── 20120301 .. _symlink: Symlinking ========== In this phase the files from :ref:`raw_dir` are symlinked into :ref:`redux_dir` and a database, used to drive the GUI, is created. As in the raw directory, the data are divided per night, but inside each night they are reorganized by their type. * The science frames belonging to the same virus shot are organized in a subdirectory of the ``sci`` directory, called as the value of the ``OBJECT`` header keyword, with white spaces replaced by underscores. If the keyword is not present or holds multiple values for the files belonging to one shot the symlinking will fail. If the ``OBJECT`` keyword is not populated, the directory name ``'no_object'`` is used. If multiple shots in the same night have the same ``OBJECT``, the directory name is the value of the keyword followed by ``"_sym"`` and a three digit counter. * The bias (a.k.a. zero) frames belonging to one shot are grouped together in a subdirectory of ``zro``, whose name is the average time stamp of all of the files. * The calibration frames, namely flats and arcs (a.k.a. comps), are grouped together in pairs in subdirectories of ``cal``. The name of each subdirectory is the average time stamp of the flats or arc/comp files it contains. The rationale for the grouping is that the flat and the arcs need to be combined to compute distortion and fiber model. All the files are symlinked in the above subdirectories without replicating the ``exp??/virus`` structure. When symlinking the files, two metadata files are created in each directory: ``shot_name.txt`` and ``exposure_names.txt``. The former one contains information about the type of files and the original shot directories; the latter contains a mapping between exposure number. If a ``redux`` directory is found, VDAT uses these files to rebuild the internal database, thus they **should never** be removed without removing the directory too. In this way is possible to re-run VDAT without performing the initial symlink. After the symlink is finished, VDAT attempts to find a ``zro`` and a ``cal`` directory for each ``sci`` directory. This is used as a source of calibration frames for the reduction. As a default VDAT chooses the closest directories in time, but the ``zro`` and ``cal`` directories can also be chosen by the user via the GUI. .. _symlink_conf: Configuration ------------- It's possible for the user to customize some part of the symlinking, e.g. providing new directory name matches, via the main VDAT configuration file, ``vdat_setting.cfg``. In the rest of the section we will describe the name of the available options, divided per section in the file. ``[general]`` +++++++++++++ * ``rawdir``: path to the raw directory/ies or to the night(s) to symlink; it is overridden by the ``vdat`` positional arguments. * ``redux_dir``: path to the redux directory. * ``is_rawdir_night``: if true ``vdat`` interprets the ``rawdir`` entries as being already night directories. If not found defaults to ``no``. Can be overridden with the ``-N/--is-rawsir-night`` command line option. ``[symlink]`` +++++++++++++ * ``night``: generic name of the night directories. If, e.g. the night directory name is a eight digit number, the value would be: ``nights = */[0-9]{8}``; ignored if ``is_rawdir_night`` is given. * ``is_night_regex``: if the value is any of ``yes/true/on/1`` the value of ``night`` is interpreted as a regex expression, if it's any of ``no/false/off/0`` or not present, the ``night`` is interpreted as described in :mod:`fnmatch`. * ``virus_instrument``: name of the instrument; it is the name of the directory between the night and the shot; if not found, defaults to 'virus'. If it's given but empty, the directory structure becomes night/shot. Can be overridden with the ``-i/--virus-instrument`` command line option. * ``virus_shot``: generic name for the virus shot directories. Very likely will be ``virus_shot = *virus[0-9]*``. * ``is_virus_shot_regex``: as ``is_night_regex`` but for the virus shot entry. * ``virus_fits_files``: name of the fits file to search; e.g.: ``*/virus/*.fits`` * ``is_virus_fits_files_regex``: as ``is_night_regex`` but for the virus fits file * ``relative_symlink``: decide whether to do an absolute or a relative symlink * ``replace_symlink``: if a symlink exists, remove and recreate it * ``image_type_pattern``: regex pattern to use to extract the image type from the file name in the form of ``pattern, replacement``. For example ``.*_(\w{3})\.fits, \1`` extract the last three letters before the file * ``datetime_pattern``: regex pattern to use to extract the date and time from the file name in the form of ``pattern, replacement``. For example ``.*/(\d{8}T\d{6})_.*\.fits, \1`` extract eight number, a ``T`` and other six number at the beginning of the file name * ``datetime_fmt``: format of the date and time extracted wit the above pattern; e.g.: ``%Y%m%dT%H%M%S`` * ``object_key``: name of the header keyword to use to extract the object for the science frames; default ``OBJECT`` * ``cal_types``: list of comma separated image types that should be collected together; ``cal_types = flt, cmp`` collects flat and arc frames, while ``cal_types = twi, cmp`` collects twilights and arc frames. The option can be overridden with the ``--cal-types`` command line option. * ``cal_type_header_typ``: when dealing with the calibration type ``typ`` get the value of the header keyword ``cal_type_header_typ`` and use it as ``object`` for that type. This allows the collection multiple shots of the same ``typ`` in the same calibration directory as long as the header keyword has different values. The ``cmp`` types have different lamp types, whose names are saved in the ``OBJECT`` header keyword, so by default VDAT uses ``cal_type_header_cmp = OBJECT`` * ``max_delta_cal``: a flat and an arc per ``lamp_type`` shot are collected in the same redux directory if their average time stamps differs less second than: Unknown or nonstandard types can be also symlinked, as long as the operation can be carried out as if the types were ``sci`` or ``zro``. At the moment there is no support for nonstandard ``cal`` like types. To declare that an unknown type, e.g. ``drk`` can be symlinked as if it were a known one, e.g. ``zro`` add to the ``[symlink]`` section the following option:: drk_symlink_as = zro .. The text below is a repeat of what is written above .. After the symlinking is finished, for each redux directory a reference zero and a calibration directory are searched and registered; the reference directory is the nearest one in time.