vdat.utilities – Generic utilities

Utilities

vdat.utilities.ISOTIME_FMT = '%Y-%m-%dT%H:%M:%S.%f'

Date time formatting in the json

vdat.utilities.SHOT_FILE = 'shot_name.txt'

The file contains basic information about the type of files and the original directory and is used to rebuild the database on subsequent runs of vdat

vdat.utilities.EXPS_FILE = 'exposure_names.txt'

Maps the base name of each virus fits file (basically the time stamp), with the exposure number

class vdat.utilities.DatetimeEncoder(*args, **kwargs)

Encodes datetime.date, datetime.time or datetime.datetime as dictionary:

{“__datetime__”: True, “date”: formatted datetime,

“type”: datetime type, “fmt”: format}

with datetime type is one of: “datetime”, “date”, “time”

Parameters:

*args, **kwargs : same as json.JSONEncoder

dt_formatter : string

formatter used to encode the datetime; defaults to ISOTIME_FMT

d_formatter : string

formatter used to encode the date; defaults to the part before “T” in ISOTIME_FMT

t_formatter : string

formatter used to encode the time; defaults to the part after “T” in ISOTIME_FMT

default(obj)

Decode object obj. If it’s not a date/time/datetime instance delegate to the parent class default

Parameters:

obj : object to be serialised

Returns:

serialised object

_datetime_dic(date, type_, fmt)

Create the dictionary to feed to the encoder

vdat.utilities.decode_datetime(dct)

If the input dictionary has a __datetime__ key set to true, uses the keys “date”, “type” and “fmt” to decode the datetime encoded by DatetimeEncoder

Parameters:

dct : dictionaries

object to decode

Returns:

decoded object

vdat.utilities.json_dumps(obj)

Serialise obj into a json using DatetimeEncoder and %Y-%m-%dT%H:%M:%S.%f formatting for the date

Parameters:

obj : object to serialise

Returns:

string

serialised json

vdat.utilities.json_loads(s)

Deserialize the string s into a python object, undoing the datetime encoding done by DatetimeEncoder

Parameters:

s : string to deserialize

Returns:

python object

vdat.utilities.read_json_file(fname, decode=True)

Read the content of the file and, if decode is True decode each line as a json entry

Parameters:

fname : string

name of the file to read

decode : bool, optional

decode each line in the file as a json

Returns:

list of string or of objects

lines of the file

vdat.utilities.write_to_json_file(fname, append=True, **kwargs)

Serialize the keyword arguments and write them as a single line to fname.

Parameters:

fname : string

name of the file where to write

append : bool, optional

if true append to the file, if false write to it

kwargs : dictionary

line to write

vdat.utilities._read_file(fname)

Wrap read_json_file() to pass only the directory name

vdat.utilities._write_file(fname)

Wrap write_to_json_file() to pass only the directory name

vdat.utilities.read_shot_file(dir_, decode=True)

Read the content of the shot file ‘shot_name.txt’ in dir_.

Parameters:

dir_ : string

name of the directory where the file is located

decode : bool, optional

decode each line in the file as a json

Returns:

list of string or of objects

lines of the file

vdat.utilities.read_exps_file(dir_, decode=True)

Read the content of the shot file ‘exposure_names.txt’ in dir_.

Parameters:

dir_ : string

name of the directory where the file is located

decode : bool, optional

decode each line in the file as a json

Returns:

list of string or of objects

lines of the file

vdat.utilities.write_to_shot_file(dir_, append=True, **kwargs)

Serialize the keyword arguments as a single line to the shot_name.txt file in directory dir_.

Parameters:

dir_ : string

name of the directory where the file is located

append : bool, optional

if true append to the file, if false write to it

kwargs : dictionary

line to write

vdat.utilities.write_to_exps_file(dir_, append=True, **kwargs)

Serialize the keyword arguments as a single line to the exposure_names.txt file in directory dir_.

Parameters:

dir_ : string

name of the directory where the file is located

append : bool, optional

if true append to the file, if false write to it

kwargs : dictionary

line to write

vdat.utilities.merge_dicts(dicts, exclude=[])

Merge the dictionaries into one

Unique entries are copied verbatim. For repeated entries: * if string: join them with “, ” * if date or datetime: average them * if bool: all is used: so is True only if all the entries are True

Parameters:

dicts : list of dictionaries

dictionaries to merge

exclude : list of strings

exclude entries from out_dict.

Returns:

out_dict : dictionary

merged dictionaries

Raises:

VDATUnknownDictEntry

if it doesn’t know what to do how to merge the entry

vdat.utilities.collect_metadata(redux_dir, skip_empty=False, repair_redux=False, merge_shot=False, yield_dir=False)

Recursively scan the redux_dir directory looking for shot_name.txt and exposure_names.txt files and yield their content.

Parameters:

redux_dir : string

name of the directory to scan

skip_empty : bool, optional

if any of the two files is empty or does not exist, skip the directory

repair_redux : bool, optional

if the redux_dir entry in the shot_name.txt files is different from the input one, update it

merge_shot : bool, optional

if the file shot_name.txt contains multiple lines, merge them into one

yield_dir : bool, optional

if True, yields also the name of the directory containing the files

Yields:

shot_file, exps_file : list of dict

content of the shot_name.txt and the exposure_names.txt files

vdat.utilities.grouper(iterable, n, fillvalue=None)

Collect data into fixed-length chunks or blocks.

From https://docs.python.org/3/library/itertools.html#itertools-recipes

Parameters:

iterable :

iterable to split in chunks

n : int

size of the chunks

fillvalue : anything, optional

if the size the interable isn’t a multiple of n, fill the last chunk with fillvalue

Returns:

iterable

chunk of size n of the input iterable

Examples

>>> list(grouper('ABCDEFG', 3, 'x'))
[('A', 'B', 'C'), ('D', 'E', 'F'), ('G', None, None)]

exception vdat.utilities.VDATError

Generic vdat error

exception vdat.utilities.VDATDirError

Error raised when trying to create directories

exception vdat.utilities.VDATSymlinkError

Generic error raised when performing the symlinking

exception vdat.utilities.VDATFitsParseError

Exception raised when the parsing of the fits file name or headers to extract information during the symlinking fails

exception vdat.utilities.VDATFitsTypeError

Error raised when the type of the fits files is wrong or unknown

exception vdat.utilities.VDATDateError

Error raised when failing to parse dates

exception vdat.utilities.VDATUnknownDictEntry

Error raised when the shot file is malformed or contains unexpected entries

exception vdat.utilities.VDATDatabaseError

Database related errors

exception vdat.utilities.VDATDatabaseUniquenessError

The entry in the database is not unique