vdat.libvdat – Stuff that isn’t GUI

Symlinking

Symlink raw files into a redux directory

Multiprocessing is disabled for the following reasons:

• Calibration and science frames need locking to correctly deal with grouping and renaming if multiple shots have the same name from same objects;
• a lot of peewee.OperationalError: database is locked errors are raised
• symlinking is going to run only now and then and is always going to be much faster that any of the reduction steps
• if the symlink is run from the gui, there is no risk that interferes with reduction steps running

vdat.libvdat.symlink.FMT_DATE_DIR = '%Y%m%d_%H%M%S'

Format for converting a datetime instance into as string used as directory name

vdat.libvdat.symlink.symlink(log_message)

Symlink shots in the directory ‘path’ to an output redux directory. Format of redux directory follows issue 820 <https://luna.mpe.mpg.de/redmine/issues/820>

Parameters:

log_message : string

Message to log as info before starting

vdat.libvdat.symlink._scan_dirs(redux_dir)

Scan the redux directories and fill the database with entries from existing ‘shot_name.txt’ and ‘exposure_names.txt’ files. It also updates the ‘shot_name.txt’ file if the redux directory has changed.

Parameters:

redux_dir : string

name of the redux directory

Returns:

int, int

Number of VDATDir and VDATExposures entries added to the database

vdat.libvdat.symlink.do_symlink(raw_dir, redux_dir)

Run the symlinking from the raw to the redux directory

Parameters:

raw_dir : list of strings

name of the raw or the night directories

redux_dir : string

name of the raw and the redux directory

vdat.libvdat.symlink._symlink_shot(shot_dir, redux_dir)

Create the redux directory for the shot and symlink all the files from the shot directory

Parameters:

shot_dir : string

name of the shot directory

redux_dir : string

name of the directory where the new directory and symlink must go

Raises:

:class:`~vdat.utilities.VDATFitsTypeError`

if the image type or object are not consistent or unknown

vdat.libvdat.symlink._save_exposures(shot_dir, fits_files, vdat_dir)

Give the list of files belonging to shot and the VDATDir database entry vdat_dir, takes the first file in each exposure and create the necessary VDATExposures entries.

Parameters:

shot_dir : string

name of the shot directory

fits_files : list of strings

symlinked files

vdat_dir : vdat.database.VDATDir instance

database entry related to the input files

vdat.libvdat.symlink._get_imagetype_datetime(conf, log, shot_dir, fnames)

Extract the image type and the date from the files

Parameters:

conf: configuration object

can be either a ConfigParser or a dictionary(-like) object

log : logging object

shot_dir : string

name of the shot directory

fnames : list

file names to scan

Returns:

image_type : string

type of the images

avg_date : datetime

average timestamp

avg_date_string : string

string representation of the average time stamp

Raises:

VDATFitsParseError

if it fails parsing the file names

VDATFitsTypeError

if there are more than one image types

vdat.libvdat.symlink._save_and_symlink(log, fits_files, vdat_dir, write_shot_file, append_to_shot_file, vdat_dir_shot_file=None)

Fail safe symlinking:

• add vdat_dir to the database; if it fails, abort
• symlink the fits_files to vdat_dir.path; if it fails removes the symlinked files and vdat_dir from the database
• add vdat_dir to the shot file; if it fails undo the previous steps

Parameters:

log : logging object

fits_files : list of strings

as in the input

vdat_dir : vdat.database.VDATDir instance

database entry to save

write_shot_file, append_to_shot_file : bool

whether the entry needs to be written into the shot file and, in such case, whether it must be appended

vdat_dir_shot_file : vdat.database.VDATDir instance

if given this entry, not vdat_dir, is added to the shot file

vdat.libvdat.symlink._get_unique_keyword(fits_files, keyword, shot)

Get the header keyword from all the fits files and check that only one exists.

Parameters:

fits_files : list of strings

names of the fits files

keyword : string

name of the keyword to extract

shot : string

name of the shot

Returns:

string

value of the header keyword

Raises:

VDATFitsParseError

if the header keyword does not exist

VDATFitsTypeError

if there are more than one values for the header keywords

vdat.libvdat.symlink._symlink_sci(fits_files, vdat_dir)

Symlink the science shots

Parameters:

fits_files : list of strings

names of the fits files

vdat_dir : vdat.database.VDATDir instance

contains all the relevant information for the symlinking

Returns:

out_vdat_dir : vdat.database.VDATDir instance

database entry to representing the current data

Raises:

VDATSymlinkError

if there is a mismatch between objects and shots (e.g. the same shot in the same night has a different object name)

vdat.libvdat.symlink._symlink_zro(fits_files, vdat_dir)

Symlink the bias shots.

Parameters:

fits_files : list of strings

names of the fits files

vdat_dir : vdat.database.VDATDir instance

contains all the relevant information for the symlinking

Returns:

out_vdat_dir : vdat.database.VDATDir instance

database entry to representing the current data

vdat.libvdat.symlink._symlink_cal(fits_files, vdat_dir)

Symlink the calibration, flat and arc, shots.

Flats and arcs taken together goes into the same directory.

1. If the shot is already symlinked, reuse the directory

2. If not look for directories with different original type or object,

   order them at increasing time distance and take the nearest one: if it’s within a maximum time distance, symlink into that directory

3. Otherwise create a new directory and symlink into it

Parameters:

fits_files : list of strings

names of the fits files

vdat_dir : vdat.database.VDATDir instance

contains all the relevant information for the symlinking

Returns:

out_vdat_dir : vdat.database.VDATDir instance

database entry to representing the current data

vdat.libvdat.symlink._mkdir(dirname, log, failsafe=True)

Create the directory.

If it exists, log it as error and, if failsafe is False, re-raise the exception

Parameters:

dirname : string

name of the directory to create

log : logging.Logging instance

log messages to this logger

safe : bool, optional

if true silently ignores OSError` due to existing directories

Raises:

:class:`~vdat.utilities.VDATDirError` if the creation fails with a

:class:`OSError` and ``failsafe`` is False

vdat.libvdat.symlink._symlink_file(file_list, target_dir, log, failsafe=True)

Symlink the files into the target directory.

If it exists, log it as error and, if failsafe is False, re-raise the exception

Parameters:

file_list : list of strings

names of the fits files to symlink

target_dir : string

name of the directory where to do the symlink

log : logging.Logging instance

log messages to this logger

safe : bool, optional

if true ignores OSError` due to existing files

Returns:

symlinked_list : list of strings

list of files in the target directory successfully symlinked

Raises:

:class:`~vdat.utilities.VDATSymlinkError` if the symlink creation fails

with a :class:`OSError` and ``failsafe`` is False

vdat.libvdat.symlink._average_timestamps(dates, infmt, outfmt='%Y%m%d_%H%M%S')

Average the list of timestamps.

Parameters:

dates : list of strings

strings containing timestamps

infmt : strings

format of dates

outfmt : string, optional

format of the output time stamp

Returns:

avg_timestamp : datetime.datetime instance

average time

string

avg_timestamp formatted according to outfmt

Raises:

:class:`~vdat.utilities.VDATDateError`

if it fails to parse dates from the fits headers

vdat.libvdat.symlink._find_nearest(q, timestamp, n_nearest=1, nearest_then=None)

Go through the list of query results, order them according to the absolute distance from timestamp and return the n_nearest.

Parameters:

q : peewee.SelectQuery

query to use

timestamp : datetime instance

timestamp to use as reference

n_nearest : int, optional

maximum number of directories returned; set it to negative to return all

nearest_then : timedelta instance

if not None, don’t consider any directory whose delta time is larger than nearest_then; applied after n_nearest

Returns:

sorted_q : list of query results

ordered with respect to the timestamp

vdat.libvdat.symlink.db_create_references()

search reference zero and calibration directories and add them to the database

Logging

Deal with loggers

vdat.libvdat.loggers.setup_main_logger(conf, name='logger')

Setup the main vdat logger.

Add to the logger called name a standard output and a file logger

Parameters:

conf : ConfigParser instance

configuration options,

vdat.libvdat.loggers.make_logger(logger, conf, section=None)

Create a handler using the instructions in the configuration section and add it to the logger.

It looks for the following options:

• remove_old: if True remove all the previous handlers before adding the new one, default False
• remove_std: if True remove all the previous StreamHandler`, default False
• the ones used by make_handler()

Parameters:

logger : Logger instance

logger

conf : ConfigParser instance

configuration options

section : string, default

name of the section containing the information to build the logger; defaults to logging.name, where name is the logger name

vdat.libvdat.loggers.astropy_handlers(conf)

Tweak the astropy logger removing the existing (stream) handlers and setting a new one if required

Parameters:

conf : ConfigParser instance

configuration options,

vdat.libvdat.loggers.ginga_handlers(conf)

Create/update the ginga logger removing existing handlers setting a new one if required

Parameters:

conf : ConfigParser instance

configuration options,

vdat.libvdat.loggers.setup_command_loggers(main_config)

Set up the loggers for the commands connected or connectible to buttons

Parameters:

main_config : ConfigParser instance

configuration options,

vdat.libvdat.loggers.add_parent(conf, section, from_parent=['logdir', 'logger_level'])

Extract the section and, if from_parent is not an empty list, extend the section with those entry from the parent, without overwriting existing entries in section

The parent section of e.g. section=logging.main.stdout is logging.

Parameters:

conf : ConfigParser instance

configuration options,

section : string

name of the section in the configuration where to find all the info

Returns:

:class:`~pyhetdex.tools.configuration.ConfigParser` section

vdat.libvdat.loggers.make_handler(conf)

Create a handler following the instructions contained in section

It looks for the following options:

• enabled: if False, doesn’t create a handler and returns None; default False
• file_name: if not found, a StreamHandler is created, if exists, a FileHandler is created
• logdir: directory where the log file goes; default ‘.’
• format: formatting of the message, default %(levelname)s: %(message)s
• level: level of the handler, default INFO

Parameters:

conf : ConfigParser section

configuration options for the handler; it can be created using add_parent()

Returns:

None, if enabled is ``False``

:class:`~logging.StreamHandler`: if not file name is given

:class:`~logging.FileHandler`: otherwise

vdat.libvdat.loggers._to_log_level(level, default=20)

Try to convert level to int or to extract the level level from the logging module. If both fail, return the default.

See here for more info about logging levels.

Parameters:

level : string

numerical or literal value of the level

Returns:

int

handler level