Use VDAT: how to **************** This document describes how to start using VDAT and introduces the configuration files needed to run the software. .. note:: All the executables shipped with VDAT accept the ``-h/--help`` command line option to show the help text. We suggest our users read them to get acquainted with the available options. Also, this documentation might not directly cover all of them. Get the configuration files =========================== VDAT needs a handful of configuration files to run. A set of essential files are shipped together with the code and can be recovered with the ``vdat_config`` executable. Issuing:: vdat_config copy copies the following files in the current directory:: ├── extra_files │   ├── defaultL.dist │   ├── defaultR.dist │   ├── dither_positions.txt │   ├── IFUcen_HETDEX.txt │   ├── lines_L.par │   └── lines_R.par ├── fplane.txt ├── tasks.yml ├── vdat_commands.yml └── vdat_setting.cfg If any of the files already exists, the user is asked whether she/he wants to overwrite existing files. When issuing the ``-f/--force`` keyword existing files are overwritten without asking. Alternatively, it is possible to backup existing files before copying with the ``-b/--backup`` option: this will rename the files adding ``.bkp`` to the end of their names. The files are: * ``vdat_setting.cfg``: this is is the main configuration file that contains all of the relevant options to make vdat work. The options in the file are commented and the relevant parts of the file are described in some section of this documentation. VDAT expects to find this file in the directory from which ``vdat`` is launched; alternatively it can be given with the ``-s/--setting`` command-line argument. This file is versioned. * ``tasks.yml``: contains the instructions used to build the central panel in the GUI. The entries in the file are described in :doc:`gui/main_panel`. Alternative or multiple file names can be provided either through the configuration option ``tasks_config`` in the ``[general]`` section or via the ``-t/--tasks-config`` option of the ``vdat`` executable. The order in which options are provided is important: sections in a file might be overridden by sections with the same name in subsequent files. This file is versioned. * ``vdat_commands.yml``: contains the definition of the commands that can be run; see :doc:`command_interpreter` for more info. Alternative or multiple file names can be provided either through the configuration option ``command_config`` in the ``[general]`` section or via the ``-c/--command-config`` option of the ``vdat`` executable. The order in which options are provided is important: sections in a file might be overridden by sections with the same name in subsequent files. This file is versioned. * ``fplane.txt``: it tells the GUI where the IFUs are located in the focal plane and connects the various IDs together. The name of the focal plane is passed to VDAT via the ``fp_filename`` option in the ``[fplane]`` section of the main configuration files. * ``extra_files/default*.dist``: reference distortion files used by the quick reconstruction. Their names is passed to VDAT via the ``default_dist_r`` and ``default_dist_l`` options of the ``[reconstruction]`` section. * ``extra_files/dither_positions.txt``: list of dither positions per slot position. Used in the ``mkdither`` command in ``vdat_commands.yml``. * ``extra_files/IFUcen_HETDEX.txt``: reference IFU-center files, used both in the quick reconstruction and in the definition of various commands, like ``detect``. In the former case, the name is passed to VDAT via the ``ifucen`` option in the ``[general]`` section. * ``extra_files/lines_*.par``: reference line files used in the ``deformer`` command. Check the configuration files ============================= As the code evolves, so do the configuration files. Some of the changes are essential for the VDAT inner working while some are minor modifications and improvements, we add to the three main configuration files, ``vdat_setting.cfg``, ``vdat_commands.yml`` and ``tasks.yml``, a version number. Also to help users track changes and decide how to deal with them, the ``vdat_config`` command has also ``compare`` subcommand, that does some basic comparison between the files shipped with VDAT and the ones of the users. By default the command checks that all the files exist and that the version numbers match. Optionally it is possible to see the line diffs (``-d/--diff``) or to try to load the user files to check for inconsistencies (``-l/--load``). By default those two checks apply only to the :attr:`files ` essential to run VDAT. It is possible to run them on all the files with the ``-a/--all`` switch. Launch VDAT =========== Once you have all the configuration in place, you can run VDAT using the ``vdat`` executable. However VDAT needs to know where the raw files are. To do so you can point ``vdat`` to one or more directories with the ``rawdir`` option in the ``[general]`` section of ``vdat_setting.cfg`` file or with the positional arguments of ``vdat``. The directory structure expected is described in :ref:`raw_dir`. It is also possible to point ``vdat`` to one or more nights directly: to do so it is also necessary to set the ``is_rawdir_night`` option in ``[general]`` section or the ``-N/--is-rawdir-night`` command line option to ``yes``. At first vdat executes the symlinking as described in :doc:`dirstruct`, then starts the GUI, the various elements of which are described in :doc:`gui/index`. Within the GUI buttons are available that execute tasks in the background, either in single- or multi-processor mode. Multiprocessing and the number of processors can be set using the ``use_multiprocessing`` and ``n_processors`` options in the ``[general]`` section of the ``vdat_setting.cfg`` file or via the ``-m/--use-multiprocessing`` and ``-n/--n-processors`` command line options. We suggest using, at maximum, two fewer processors than available. This way one ensures leaving enough power for the GUI event loop and the secondary threads that we run for some intensive tasks, e.g. when creating thumbnails. The VDAT database ================= When symlinking, VDAT in each reduction directory two metadata files, :attr:`~vdat.utilities.SHOT_FILE` and :attr:`~vdat.utilities.EXPS_FILE`. They contain information about the type of files in the directory and where they originally came from, as well as allow mapping file names to exposures. If found, these files are loaded into VDAT and used to rebuild the internal database, without need to redo the symlinking. Given the importance of these files, we provide an executable to check and if possible repair the metadata files: ``vdat_db``. It has two subcommands: 1) ``check``: it checks that all the metadata file found have the correct version and can be correctly loaded into the database. 2) ``update``: it tries to update the metadata files to the current version and to load them to the database. Before beginning the checks, the old metadata are moved into a backup file and if the update fails they are not moved back. This way the offending directories do not make ``vdat`` fail on the next run Success or errors are reported to the console either in a condensed way. More details can be obtained using the ``-v\--verbose`` option.