Cloud Storage Input/Output¶
-
cloudreg.scripts.download_raw_data.
download_raw_data
(in_bucket_path, channel, outdir)[source]¶ Download COLM raw data from S3 to local storage
- Parameters
in_bucket_path (str) – Name of S3 bucket where raw dadta live at
channel (int) – Channel number to process
outdir (str) – Local path to store raw data
-
cloudreg.scripts.download_raw_data.
download_tile
(s3, raw_tile_bucket, raw_tile_path, outdir, bias=None)[source]¶ Download single raw data image file from S3 to local directory
- Parameters
s3 (S3.Resource) – A Boto3 S3 resource
raw_tile_bucket (str) – Name of bucket with raw data
raw_tile_path (str) – Path to raw data file in S3 bucket
outdir (str) – Local path to store raw data
bias (np.ndarray, optional) – Bias correction multiplied by image before saving. Must be same size as image Defaults to None.
-
cloudreg.scripts.download_raw_data.
download_tiles
(tiles, raw_tile_bucket, outdir)[source]¶ Download a chunk of tiles from S3 to local storage
- Parameters
tiles (list of str) – S3 paths to raw data files to download
raw_tile_bucket (str) – Name of bucket where raw data live
outdir (str) – Local path to store raw data at
-
cloudreg.scripts.download_raw_data.
get_all_s3_objects
(s3, **base_kwargs)[source]¶ Get all s3 objects with base_kwargs
- Parameters
s3 (boto3.S3.client) – an active S3 Client.
- Yields
dict – Response object with keys to objects if there are any.
-
cloudreg.scripts.download_raw_data.
get_list_of_files_to_process
(in_bucket_name, prefix, channel)[source]¶ Get paths of all raw data files for a given channel.
- Parameters
in_bucket_name (str) – S3 bucket in which raw data live
prefix (str) – Prefix for the S3 path at which raw data live
channel (int) – Channel number to process
- Returns
List of S3 paths for all raw data files
- Return type
list of str
-
cloudreg.scripts.download_raw_data.
get_out_path
(in_path, outdir)[source]¶ Get output path for given tile, maintaining folder structure for Terastitcher
- Parameters
in_path (str) – S3 key to raw tile
outdir (str) – Path to local directory to store raw data
- Returns
Path to store raw tile at.
- Return type
str
-
cloudreg.scripts.create_precomputed_volume.
create_cloud_volume
(precomputed_path, img_size, voxel_size, num_mips, chunk_size, parallel=False, layer_type='image', dtype='uint16')[source]¶ Create Neuroglancer precomputed volume S3
- Parameters
precomputed_path (str) – S3 Path to location where precomputed layer will be stored
img_size (list of int) – Size of the image (in 3D) to be uploaded
voxel_size ([type]) – Voxel size in nanometers
num_mips (int, optional) – Number of downsampling levels in X and Y. Defaults to 6.
chunk_size (list, optional) – Size of each chunk stored on S3. Defaults to [1024, 1024, 1].
parallel (bool, optional) – Whether or not the returned CloudVlue object will use parallel threads. Defaults to False.
layer_type (str, optional) – Neuroglancer type of layer. Can be image or segmentation. Defaults to “image”.
dtype (str, optional) – Datatype of precomputed volume. Defaults to “uint16”.
- Returns
CloudVolume object associated with this precomputed volume
- Return type
cloudvolume.CloudVolume
-
cloudreg.scripts.create_precomputed_volume.
create_precomputed_volume
(input_path, voxel_size, precomputed_path, extension='tif')[source]¶ Create precomputed volume on S3 from 2D TIF series
- Parameters
input_path (str) – Local path to 2D TIF series
voxel_size (np.ndarray) – Voxel size of image in X,Y,Z in microns
precomputed_path (str) – S3 path where precomputed volume will be stored
extension (str, optional) – Extension for image files. Defaults to “tif”.
-
cloudreg.scripts.create_precomputed_volume.
get_image_dims
(files)[source]¶ Get X,Y,Z dimensions of images based on list of files
- Parameters
files (list of str) – Path to 2D tif series
- Returns
X,Y,Z size of image in files
- Return type
list of int
-
cloudreg.scripts.create_precomputed_volume.
process
(z, file_path, layer_path, num_mips)[source]¶ Upload single slice to S3 as precomputed
- Parameters
z (int) – Z slice number to upload
file_path (str) – Path to z-th slice
layer_path (str) – S3 path to store data at
num_mips (int) – Number of 2x2 downsampling levels in X,Y
-
cloudreg.scripts.download_data.
download_data
(s3_path, outfile, desired_resolution=15000, resample_isotropic=False, return_size=False)[source]¶ Download whole precomputed volume from S3 at desired resolution and optionally resample data to be isotropic
- Parameters
s3_path (str) – S3 path to precomputed volume
outfile (str) – Path to output file
desired_resolution (int, optional) – Lowest resolution (in nanometers) at which to download data if desired res isnt available. Defaults to 15000.
resample_isotropic (bool, optional) – If true, resample data to be isotropic.
- Returns
Resoluton of downloaded data in microns
- Return type
resolution
-
cloudreg.scripts.download_data.
get_mip_at_res
(vol, resolution)[source]¶ Find the mip that is at least a given resolution
- Parameters
vol (cloudvolume.CloudVoluem) – CloudVolume object for desired precomputed volume
resolution (int) – Desired resolution in nanometers
- Returns
mip and resolution at that mip
- Return type
tuple