Scratches¶
A scratch is a copy of the images a program goes back to, such as the particles of a dataset read once per epoch. A loader made with one reads what the scratch holds from it, and takes in what it does not hold yet.
- class vitrio.ImageScratch¶
A copy of the images a program goes back to, in RAM or in a file.
Make one with create_host_image_scratch, create_mapped_file_image_scratch or create_directory_image_scratch, and read through it with vitrio.loader(scratch=…) or vitrio.reader_provider(scratch=…). What it holds is read from it, and what it does not yet hold is read from the file and taken in.
- vitrio.create_host_image_scratch(locations, run_length, max_size=None, files=None)¶
Create a scratch that keeps its copies in main memory.
A scratch holds copies of the images a program keeps going back to, such as the particles of a dataset read once per epoch. The memory is as large as the images of locations need, and the scratch holds them all unless that is more than max_size. It then holds what fits, file by file in ascending order of key, and reads the rest from the files.
The scratch starts empty. An image is taken in the first time it is read through the scratch, or ahead of that by prefetch_scratch_async. The memory is released once nothing holds the scratch, the providers and loaders made with it included.
Every file is opened, twice, before this returns.
- Parameters:
locations – The images to hold, as an ImageLocationGrouping or as any iterable of ImageLocation. One without an index in a stack names the whole file.
run_length – How many held images of a file are taken in together. Reading any of them reads them all, in one read of the file.
max_size – The most bytes to allocate. Defaults to no maximum.
files – Where the files become readers. Defaults to a provider that opens the file at the path each key is.
- Returns:
The scratch.
- Return type:
- Raises:
ValueError – If run_length is zero, or if there is nothing to hold: locations names no image, or max_size has no room for one.
IndexError – If a location has an index its stack does not have.
- vitrio.create_mapped_file_image_scratch(locations, path, run_length, max_size=None, files=None)¶
Create a scratch that keeps its copies in a file mapped into memory.
Holds what create_host_image_scratch would, with the same arguments, but in a file at path. The file may be larger than the main memory, and it outlives the scratch: a scratch created later over the same file, for the same images, resumes from what was taken in before, except for the images whose files have been modified since. Programs that hold the same images may use the file at the same time, and share it.
A file at path that holds anything else is overwritten, and the scratch starts empty.
vitrio never removes the file. Remove it once no program is using it any more. On Windows it can be removed only once nothing holds the scratch, the providers and loaders made with it included.
- Parameters:
locations – The images to hold, as create_host_image_scratch takes them.
path – The file to keep the copies in. It is created if it is missing.
run_length – How many held images of a file are taken in together.
max_size – The largest the file may be, in bytes. Defaults to no maximum.
files – Where the image files become readers. Defaults to a provider that opens the file at the path each key is.
- Returns:
The scratch.
- Return type:
- Raises:
ValueError – If run_length is zero, or if there is nothing to hold. No file is created then.
IndexError – If a location has an index its stack does not have.
- vitrio.create_directory_image_scratch(locations, directory, run_length, budget, files=None)¶
Create a scratch that keeps its copies in a file of a directory kept within a budget.
Holds what create_mapped_file_image_scratch would, in a file of directory named after the images and how they are laid out. A later scratch of the same images, with the same run_length, finds that file and resumes from it, and programs that hold the same images may share it. So the directory can serve every program of a machine.
A new file is made room for by removing the files of other scratches, the one used least recently first. A file is used when a scratch is created over it. If the images do not fit in budget, the scratch holds those that do, as create_host_image_scratch describes, and every other file that can be removed is.
Only the files vitrio names count against the budget and are removed: those ending in .vitrio-scratch. Anything else in the directory is left alone.
A file that is in use elsewhere can not be removed on Windows, and keeps its room. On Linux and macOS it is removed, but its space on the disk is freed only once nothing uses it. The disk must have room for the budget: the space is taken as images are taken in, and a program that finds the disk full then is ended.
- Parameters:
locations – The images to hold, as create_host_image_scratch takes them.
directory – Where the files of scratches are kept. It is created if it is missing.
run_length – How many held images of a file are taken in together.
budget – The most bytes the files of scratches in directory may take together.
files – Where the image files become readers. Defaults to a provider that opens the file at the path each key is.
- Returns:
The scratch.
- Return type:
- Raises:
ValueError – If run_length is zero, or if there is nothing to hold: locations names no image, or the budget has no room for one. No file is created then.
IndexError – If a location has an index its stack does not have.
- vitrio.prefetch_scratch_async(scratch, locations, executor, files=None)¶
Take images into a scratch ahead of the reads that need them.
Returns before the images are in. Each file is one task on executor, which reads what locations names of it in one go. A file the scratch holds nothing of is passed over. The scratch may be read through in the meantime, and a read takes in what it needs itself.
The executor is not defaulted: one made for the call would be destroyed when the call returns, and that waits for its work.
- Parameters:
scratch – The scratch to fill.
locations – The images to take in, as an ImageLocationGrouping or as any iterable of ImageLocation. Usually those the scratch was created with.
executor – Where the files are read.
files – Where the files become readers. It has to read the files themselves and not through scratch. Defaults to a provider that opens the file at the path each key is.
- Returns:
Ready once every file has been taken in or has failed, and it raises the first failure.
- Return type: