Cloud Storage Input/Output

cloudreg.scripts.download_raw_data.download_raw_data(in_bucket_path, channel, outdir)

Download COLM raw data from S3 to local storage

Parameters

• in_bucket_path (str) – Name of S3 bucket where raw dadta live at

• channel (int) – Channel number to process

• outdir (str) – Local path to store raw data

cloudreg.scripts.download_raw_data.download_tile(s3, raw_tile_bucket, raw_tile_path, outdir, bias=None)

Download single raw data image file from S3 to local directory

Parameters

• s3 (S3.Resource) – A Boto3 S3 resource

• raw_tile_bucket (str) – Name of bucket with raw data

• raw_tile_path (str) – Path to raw data file in S3 bucket

• outdir (str) – Local path to store raw data

• bias (np.ndarray, optional) – Bias correction multiplied by image before saving. Must be same size as image Defaults to None.

cloudreg.scripts.download_raw_data.download_tiles(tiles, raw_tile_bucket, outdir)

Download a chunk of tiles from S3 to local storage

Parameters

• tiles (list of str) – S3 paths to raw data files to download

• raw_tile_bucket (str) – Name of bucket where raw data live

• outdir (str) – Local path to store raw data at

cloudreg.scripts.download_raw_data.get_all_s3_objects(s3, **base_kwargs)

Get all s3 objects with base_kwargs

Parameters

s3 (boto3.S3.client) – an active S3 Client.

Yields

dict – Response object with keys to objects if there are any.

cloudreg.scripts.download_raw_data.get_list_of_files_to_process(in_bucket_name, prefix, channel)

Get paths of all raw data files for a given channel.

Parameters

• in_bucket_name (str) – S3 bucket in which raw data live

• prefix (str) – Prefix for the S3 path at which raw data live

• channel (int) – Channel number to process

Returns

List of S3 paths for all raw data files

Return type

list of str

cloudreg.scripts.download_raw_data.get_out_path(in_path, outdir)

Get output path for given tile, maintaining folder structure for Terastitcher

Parameters

• in_path (str) – S3 key to raw tile

• outdir (str) – Path to local directory to store raw data

Returns

Path to store raw tile at.

Return type

str

cloudreg.scripts.download_data.download_data(s3_path, outfile, desired_resolution, resample_isotropic=False, return_size=False)

Download whole precomputed volume from S3 at desired resolution and optionally resample data to be isotropic

Parameters

• s3_path (str) – S3 path to precomputed volume

• outfile (str) – Path to output file

• desired_resolution (int) – Lowest resolution (in nanometers) at which to download data if desired_resolution isnt available.

• resample_isotropic (bool, optional) – If true, resample data to be isotropic at desired_resolution.

Returns

Resoluton of downloaded data in microns

Return type

resolution

cloudreg.scripts.download_data.get_mip_at_res(vol, resolution)

Find the mip that is at least a given resolution

Parameters

• vol (cloudvolume.CloudVoluem) – CloudVolume object for desired precomputed volume

• resolution (int) – Desired resolution in nanometers

Returns

mip and resolution at that mip

Return type

tuple