Cloud Storage Input/Output

cloudreg.scripts.download_raw_data.download_raw_data(in_bucket_path, channel, outdir)[source]

Download COLM raw data from S3 to local storage

Parameters

  • in_bucket_path (str) – Name of S3 bucket where raw dadta live at

  • channel (int) – Channel number to process

  • outdir (str) – Local path to store raw data

cloudreg.scripts.download_raw_data.download_tile(s3, raw_tile_bucket, raw_tile_path, outdir, bias=None)[source]

Download single raw data image file from S3 to local directory

Parameters

  • s3 (S3.Resource) – A Boto3 S3 resource

  • raw_tile_bucket (str) – Name of bucket with raw data

  • raw_tile_path (str) – Path to raw data file in S3 bucket

  • outdir (str) – Local path to store raw data

  • bias (np.ndarray, optional) – Bias correction multiplied by image before saving. Must be same size as image Defaults to None.

cloudreg.scripts.download_raw_data.download_tiles(tiles, raw_tile_bucket, outdir)[source]

Download a chunk of tiles from S3 to local storage

Parameters

  • tiles (list of str) – S3 paths to raw data files to download

  • raw_tile_bucket (str) – Name of bucket where raw data live

  • outdir (str) – Local path to store raw data at

cloudreg.scripts.download_raw_data.get_all_s3_objects(s3, **base_kwargs)[source]

Get all s3 objects with base_kwargs

Parameters

s3 (boto3.S3.client) – an active S3 Client.

Yields

dict – Response object with keys to objects if there are any.

cloudreg.scripts.download_raw_data.get_list_of_files_to_process(in_bucket_name, prefix, channel)[source]

Get paths of all raw data files for a given channel.

Parameters

  • in_bucket_name (str) – S3 bucket in which raw data live

  • prefix (str) – Prefix for the S3 path at which raw data live

  • channel (int) – Channel number to process

Returns

List of S3 paths for all raw data files

Return type

list of str

cloudreg.scripts.download_raw_data.get_out_path(in_path, outdir)[source]

Get output path for given tile, maintaining folder structure for Terastitcher

Parameters

  • in_path (str) – S3 key to raw tile

  • outdir (str) – Path to local directory to store raw data

Returns

Path to store raw tile at.

Return type

str

cloudreg.scripts.create_precomputed_volume.create_cloud_volume(precomputed_path, img_size, voxel_size, num_mips, chunk_size, parallel=False, layer_type='image', dtype='uint16')[source]

Create Neuroglancer precomputed volume S3

Parameters

  • precomputed_path (str) – S3 Path to location where precomputed layer will be stored

  • img_size (list of int) – Size of the image (in 3D) to be uploaded

  • voxel_size ([type]) – Voxel size in nanometers

  • num_mips (int, optional) – Number of downsampling levels in X and Y. Defaults to 6.

  • chunk_size (list, optional) – Size of each chunk stored on S3. Defaults to [1024, 1024, 1].

  • parallel (bool, optional) – Whether or not the returned CloudVlue object will use parallel threads. Defaults to False.

  • layer_type (str, optional) – Neuroglancer type of layer. Can be image or segmentation. Defaults to “image”.

  • dtype (str, optional) – Datatype of precomputed volume. Defaults to “uint16”.

Returns

CloudVolume object associated with this precomputed volume

Return type

cloudvolume.CloudVolume

cloudreg.scripts.create_precomputed_volume.create_precomputed_volume(input_path, voxel_size, precomputed_path, num_procs=None, extension='tif')[source]

Create precomputed volume on S3 from 2D TIF series

Parameters

  • input_path (str) – Local path to 2D TIF series

  • voxel_size (np.ndarray) – Voxel size of image in X,Y,Z in microns

  • precomputed_path (str) – S3 path where precomputed volume will be stored

  • extension (str, optional) – Extension for image files. Defaults to “tif”.

cloudreg.scripts.create_precomputed_volume.get_image_dims(files)[source]

Get X,Y,Z dimensions of images based on list of files

Parameters

files (list of str) – Path to 2D tif series

Returns

X,Y,Z size of image in files

Return type

list of int

cloudreg.scripts.create_precomputed_volume.process(z, file_path, layer_path, num_mips)[source]

Upload single slice to S3 as precomputed

Parameters

  • z (int) – Z slice number to upload

  • file_path (str) – Path to z-th slice

  • layer_path (str) – S3 path to store data at

  • num_mips (int) – Number of 2x2 downsampling levels in X,Y

cloudreg.scripts.download_data.download_data(s3_path, outfile, desired_resolution, resample_isotropic=False, return_size=False)[source]

Download whole precomputed volume from S3 at desired resolution and optionally resample data to be isotropic

Parameters

  • s3_path (str) – S3 path to precomputed volume

  • outfile (str) – Path to output file

  • desired_resolution (int) – Lowest resolution (in nanometers) at which to download data if desired_resolution isnt available.

  • resample_isotropic (bool, optional) – If true, resample data to be isotropic at desired_resolution.

Returns

Resoluton of downloaded data in microns

Return type

resolution

cloudreg.scripts.download_data.get_mip_at_res(vol, resolution)[source]

Find the mip that is at least a given resolution

Parameters

  • vol (cloudvolume.CloudVoluem) – CloudVolume object for desired precomputed volume

  • resolution (int) – Desired resolution in nanometers

Returns

mip and resolution at that mip

Return type

tuple