Use VDAT: how to

This document describes how to start using VDAT and introduces the configuration files needed to run the software.

Note

All the executables shipped with VDAT accept the -h/--help command line option to show the help text. We suggest our users read them to get acquainted with the available options. Also, this documentation might not directly cover all of them.

Get the configuration files

VDAT needs a handful of configuration files to run. A set of essential files are shipped together with the code and can be recovered with the vdat_config executable. Issuing:

vdat_config copy

copies the following files in the current directory:

├── extra_files
│   ├── defaultL.dist
│   ├── defaultR.dist
│   ├── dither_positions.txt
│   ├── IFUcen_HETDEX.txt
│   ├── lines_L.par
│   └── lines_R.par
├── fplane.txt
├── tasks.yml
├── vdat_commands.yml
└── vdat_setting.cfg

If any of the files already exists, the user is asked whether she/he wants to overwrite existing files. When issuing the -f/--force keyword existing files are overwritten without asking. Alternatively, it is possible to backup existing files before copying with the -b/--backup option: this will rename the files adding .bkp to the end of their names.

The files are:

  • vdat_setting.cfg: this is is the main configuration file that contains all of the relevant options to make vdat work. The options in the file are commented and the relevant parts of the file are described in some section of this documentation. VDAT expects to find this file in the directory from which vdat is launched; alternatively it can be given with the -s/--setting command-line argument. This file is versioned.
  • tasks.yml: contains the instructions used to build the central panel in the GUI. The entries in the file are described in Main Panel. Alternative or multiple file names can be provided either through the configuration option tasks_config in the [general] section or via the -t/--tasks-config option of the vdat executable. The order in which options are provided is important: sections in a file might be overridden by sections with the same name in subsequent files. This file is versioned.
  • vdat_commands.yml: contains the definition of the commands that can be run; see The Command Interpreter Tool for more info. Alternative or multiple file names can be provided either through the configuration option command_config in the [general] section or via the -c/--command-config option of the vdat executable. The order in which options are provided is important: sections in a file might be overridden by sections with the same name in subsequent files. This file is versioned.
  • fplane.txt: it tells the GUI where the IFUs are located in the focal plane and connects the various IDs together. The name of the focal plane is passed to VDAT via the fp_filename option in the [fplane] section of the main configuration files.
  • extra_files/default*.dist: reference distortion files used by the quick reconstruction. Their names is passed to VDAT via the default_dist_r and default_dist_l options of the [reconstruction] section.
  • extra_files/dither_positions.txt: list of dither positions per slot position. Used in the mkdither command in vdat_commands.yml.
  • extra_files/IFUcen_HETDEX.txt: reference IFU-center files, used both in the quick reconstruction and in the definition of various commands, like detect. In the former case, the name is passed to VDAT via the ifucen option in the [general] section.
  • extra_files/lines_*.par: reference line files used in the deformer command.

Check the configuration files

As the code evolves, so do the configuration files. Some of the changes are essential for the VDAT inner working while some are minor modifications and improvements, we add to the three main configuration files, vdat_setting.cfg, vdat_commands.yml and tasks.yml, a version number. Also to help users track changes and decide how to deal with them, the vdat_config command has also compare subcommand, that does some basic comparison between the files shipped with VDAT and the ones of the users.

By default the command checks that all the files exist and that the version numbers match. Optionally it is possible to see the line diffs (-d/--diff) or to try to load the user files to check for inconsistencies (-l/--load). By default those two checks apply only to the files essential to run VDAT. It is possible to run them on all the files with the -a/--all switch.

Launch VDAT

Once you have all the configuration in place, you can run VDAT using the vdat executable.

However VDAT needs to know where the raw files are. To do so you can point vdat to one or more directories with the rawdir option in the [general] section of vdat_setting.cfg file or with the positional arguments of vdat. The directory structure expected is described in Raw directory structure. It is also possible to point vdat to one or more nights directly: to do so it is also necessary to set the is_rawdir_night option in [general] section or the -N/--is-rawdir-night command line option to yes.

At first vdat executes the symlinking as described in The directory structure and symlinking, then starts the GUI, the various elements of which are described in The Graphical User Interface.

Within the GUI buttons are available that execute tasks in the background, either in single- or multi-processor mode. Multiprocessing and the number of processors can be set using the use_multiprocessing and n_processors options in the [general] section of the vdat_setting.cfg file or via the -m/--use-multiprocessing and -n/--n-processors command line options. We suggest using, at maximum, two fewer processors than available. This way one ensures leaving enough power for the GUI event loop and the secondary threads that we run for some intensive tasks, e.g. when creating thumbnails.

The VDAT database

When symlinking, VDAT in each reduction directory two metadata files, SHOT_FILE and EXPS_FILE. They contain information about the type of files in the directory and where they originally came from, as well as allow mapping file names to exposures.

If found, these files are loaded into VDAT and used to rebuild the internal database, without need to redo the symlinking.

Given the importance of these files, we provide an executable to check and if possible repair the metadata files: vdat_db. It has two subcommands:

  1. check: it checks that all the metadata file found have the correct version and can be correctly loaded into the database.
  2. update: it tries to update the metadata files to the current version and to load them to the database. Before beginning the checks, the old metadata are moved into a backup file and if the update fails they are not moved back. This way the offending directories do not make vdat fail on the next run

Success or errors are reported to the console either in a condensed way. More details can be obtained using the -v\--verbose option.