getsentinel package

Submodules

getsentinel.gs_config module

Handles getsentinel configuration

Provides the paths for all the internal workings of getsentinel.

class getsentinel.gs_config.UserConfig

Bases: object

Holder class for configuration info.

ESA_USERNAME

str – The ESA SciHub username for the user.

ESA_PASSWORD

str – The ESA SciHub password for the user.

SEN2COR_ROOT_PATH

str – Contains the aboslute filepath to the sen2cor installation.

GPT_ROOT_PATH

str – The absolute filepath to the Sentinel Toolbox gpt utility.

DATA_PATH

str – The relative or absolute filepath to the data storage directory.

QUICKLOOKS_PATH

str – The relative or absolute filepath to the quicklooks storage directory.

DATA_PATH

ESA_PASSWORD

ESA_USERNAME

GPT_ROOT_PATH

QUICKLOOKS_PATH

SEN2COR_ROOT_PATH

get_property(property_name)

getsentinel.gs_config.set_userinfo(info_dict=False)

Get the path and account info from the user.

Note

This function can be called to reset/change the user config info or the user can just delete the gs_config.json file in the working directory.

Parameters:info_dict (dict of str, optional) – If passed, must contain a dict of format contained in the global variable USER_INFO_DICT Returns: Return type:None

getsentinel.gs_downloader module

getsentinel gs_downloader.py

Script to locate and auto-download products from Copernicus Open Access Hub which contains ESA Sentinel Satellite products. https://scihub.copernicus.eu/ Queries and downloads are based on dates, coordinates, and product type.

Example

Basic usage example:

import datetime
from getsentinel import gs_downloader

shape_file = 'path/to/shapefile.shp'
start = datetime.date(2018, 1, 1)
end = datetime.date(2018, 6, 1)

# initialise the query object
query = gs_downloader.Query('S1', start, end, shape_file)
query.product_details('L1', 'GRD', 'IW', 'VV VH')

# initialise the hub connection
hub = gs_downloader.CopernicusHubConnection()
total_products, product_list = hub.submit_query(query)

# optionally filter overlapping products from a given region
# query.ROI contains a polygon generated from the input coordinates
product_list = gs_downloader.filter_overlaps(product_list, query.ROI)

# optinally download the corresponding product quicklooks
hub.download_quicklooks(product_list)
hub.download_products(product_list)

exception getsentinel.gs_downloader.ChecksumError

Bases: Exception

Checksum Exception for when checksums do not match in downloading.

class getsentinel.gs_downloader.CopernicusHubConnection

Bases: object

Handles queries and product downloads to and from the ESA SciHub.

config

gs_config.UserConfig – Class container for user configuation info.

username

str – The user’s ESA account username.

password

str – The user’s ESA account password.

download_products(products, verify=False)

Downloads the products product_list to the downloadpath directory.

Parameters:

• productlist (dict) – Contains all the product returned from the query, keyed by their product UUID
• verify (bool) – If true, downloads are checked using MD5 checksum

Returns:

Return type:

None

download_quicklooks(productlist, downloadpath=None)

Downloads the quicklooks of products to a specified directory.

Note

If no quicklook is available for a product, HTML status code 500 is returned. In this case, the ESA placeholder ‘No Quicklook’ image is downloaded.

Parameters:

• product_list (dict) – Contains all the products whose quicklooks will be downlaoded, keyed by their product UUID
• downloadpath (str, optional) – Path to the directory where the quicklooks should be downloaded. Default is the QUICKLOOKS_PATH default from gs_config.

Returns:

Return type:

None

raw_query(query)

Queries the ESA SciHub with a pre-formatted query.

Note

This is mainly used by the gs_localmanager module and should not generally be used for queries. Use the submit_query method instead.

Parameters:query (str) – Pre-forammted search query string. Returns:

• total_results (int) – Number of results returned from the query
• product_list (dict) – Contains all the products returned from the query, keyed by their product UUID

submit_query(parameters)

Formats and submits a query to the ESA scihub via requests.

Note

The working of this function relies heavily on the format of the XML returned by the ESA remaining constant. Returns the number of results and a dict contain the results UUIDs and information.

Parameters:parameters (Query) – Returns:

• num_results (int) – Number of results returned from the query
• product_list (dict) – Contains all the products returned from the query, keyed by their product UUID

class getsentinel.gs_downloader.Query(satellite, start_date, end_date=False, ROI=False)

Bases: object

Holds the query parameters use in an ESA hub query.

Parameters:

• sat (str) – Current Sentinel satellites supported: ‘S1’, ‘S2’
• start_date (datetime.date) – The start date for the date range of the ESA query
• end_date (datetime.date, optional) – The end date for the date range of the ESA query
• ROI (str or list, optional) – Can be a str containing the path to a geo-referenced .geojson or shapefile or a list of coordinates, see the set_coordinates and coords_from_file methods for more information.

dates

tuple – Contains the acquisition start and end datetime.date objects that define the query search time period.

coordinates

list – A list containing the coordinates of the boundary of the region of interest (ROI).

tiles

tuple – Contains a list of the ESA defined Sentinel-2 tiles that the ROI traverses and the tile which the ROI overlaps the most with in the format (overlapped_tiles, majority_tile).

ROI

shapely.geometry.Polygon – A shapely object defining the border of the region of interst using in the query.

satellite

str – Defines which Sentinel satellite the query is for, e.g. ‘S1’, ‘S2’

proclevel

str – The processing level of the products desired from the query

details

dict – Contains optional parameters for the query

acquisition_date_range(acqstart, acqend=False)

Set the date range for the query.

This method will raise an error if a anything but datetime.date objects are passed to it or if acqstart is a date after acqend.

Note

This is used to specify the Sensing Start Time search criteria. If no end date is specific, the 24 hour period for the given start date is used. Any specified end date is considered inclusive.

Parameters:

• acqstart (datetime.date) – The start date for the date range of the ESA query
• acqend (datetime.date, optional) – The end date for the date range of the ESA query

Returns:

Return type:

None

coords_from_file(filepath)

Loads in the coordinates of a region of interest from a shapefile or geojson file.

Uses the osgeo module to extract the coordinate reference system from the file and reprojects it to WGS84.

Note

Only geojson and shapefiles are currently supported.

Parameters:filepath (str) – The path to the shapefile or geojson. Returns: Return type:None

product_details(proclevel=False, producttype=False, mode=False, polarisation=False, orbitdirection=False, resolution=False, cloudcoverlimit=False)

Sets product search the satellite type and details.

Note

Some combinations will always produce no results, eg. GRD products do note have ‘HH’ polarisations (as of 25th July 2018) and so queries using GRD and HH with return 0 results. If in doubt, broaden the parameters and always consult the ESA docs. See https://sentinel.esa.int/web/sentinel/user-guides for reference.

Parameters:

• proclevel (str, optional) – Current S1 processing levels supported: ‘L0’, ‘L1’, ‘L2’, ‘ALL’ Current S2 processing levels supported: ‘L1C’, ‘L2A’, ‘BEST’, ‘ALL’ ‘BEST’ searches for the highest level processed product from available Sentinel-2 data for the given co-ordinates.
• producttype (str, optional) – Current product type supported for S1: ‘RAW’, ‘SLC’, ‘GRD’, ‘OCN’ Not supported for S2
• mode (str, optional) – Current S1 modes identifiers supported: ‘SM’, ‘IW’, ‘EW’, ‘WV’ Not supported for S2
• polarisation (str, optional) – Current S1 polarisations supported: ‘HH’, ‘VV’, ‘HV’, ‘VH’, ‘HH HV’, ‘VV VH’ Not supported for S2
• orbitdirection (str, optional) – S1 parameter for orbit direction: ‘Ascending’ or ‘Descending’ Not supported for S2
• resolution (str, optional) – Current S1 resolutions supported: F, H, M Not supported for S2
• cloudcoverlimit (int, optional) – Not supported for S1 Integer threshold, products with percentage cloud cover higher than the threshold with be excluded from the query results.

set_coordinates(coordlist)

Stores the passed coordinates and generates ROI boundary polygon.

Stores the passed coordinates list and generates a shapely object describing the region of interest. Also uses the gs_gridtest module to find the corresponding ESA defined Sentinel-2 product tiles that the ROI overlaps.

Note

coordlist must be in the format [(lon1, lat1), (lon2, lat2), … ] First and last co-ordinates given in coordlist must be the same to complete the described area.

Parameters:coordlist (list) – List containing the coordinates describing the boundary of the region of interest. Returns: Return type:None

getsentinel.gs_downloader.filter_overlaps(product_list, ROI, external_list=False)

Filters out any overlapping products

If the ROI coordinates are completely encompassed by two products and the sensing time for both products is the same, the products will have identical data in the overlapping regions. Thus one of them can be removed from the downloads required.

Parameters:

• product_list (dict) – Contains all the products returned from the query, keyed by their product UUID
• ROI (shapely.geometry.Polygon) –
• external_list (dict) – A list of products provided that represents the products already present in the inventory. Prevents filtering from removing one product over another when the first is already present in the inventory.

getsentinel.gs_localmanager module

Downloaded product inventory manager.

This module provides an up-to-date product inventory and checks the integrity of any saved inventory against the actual contents of the DATA_PATH provided by the gs_config.

Products are stored as dictionaries containing all the necessary product info. See below for example product info format.

Example

current_inventory = gs_localmanager.get_product_inventory()

new_downloaded_products = {uuid: product, uuid: product, ...}

gs_localmanager.add_new_products(new_downloaded_products)

Example Product Info Format

Below is an example format, as stored in the product_inventory.json, of an unprocessed Sentinel-1 product. The keys tileid and userprocessed are added by the gs_downloader module. The former denotes the Senntinel-2 MGRS tiles that the product traverses, the latter indicates whether the file has been processed. For processed products a further field, origin, is added which contains the UUID of the unprocessed product from which the processed product originates.

{'acquisitiontype': 'NOMINAL',
 'beginposition': '2015-08-24T06:22:07.95Z',
 'downloadlink': "https://scihub.copernicus.eu/dhus/odata/v1/Products ... "
 'endposition': '2015-08-24T06:22:32.949Z',
 'filename': 'S1A_IW_GRDH_1SDV_20150824T062207_20150824T062232_007401 ... '
 'footprint': 'POLYGON ((-0.059059 53.057114,-3.867077 53.461575)) '
 'format': 'SAFE',
 'gmlfootprint': '<gml:Polygon footprint ...'
 'identifier': 'S1A_IW_GRDH_1SDV_20150824T062207_20150824T062232_ ... '
 'ingestiondate': '2015-08-24T15:55:01.836Z',
 'instrumentname': 'Synthetic Aperture Radar (C-band)',
 'instrumentshortname': 'SAR-C SAR',
 'lastorbitnumber': '7401',
 'lastrelativeorbitnumber': '154',
 'missiondatatakeid': '41709',
 'orbitdirection': 'DESCENDING',
 'orbitnumber': '7401',
 'origin': 'd42cc6fe-5f81-4753-bf61-dc081af8fa68',
 'platformidentifier': '0000-000A',
 'platformname': 'Sentinel-1',
 'polarisationmode': 'VV VH',
 'productclass': 'S',
 'producttype': 'GRD',
 'quicklookdownload': "https://scihub.copernicus.eu/dhus/ ... "
 'relativeorbitnumber': '154',
 'sensoroperationalmode': 'IW',
 'size': '1 GB',
 'slicenumber': '6',
 'status': 'ARCHIVED',
 'swathidentifier': 'IW',
 'tileid': [['30UVE', ...]]
 'userprocessed': False}

getsentinel.gs_localmanager.add_new_products(new_products: dict)

Adds new products to the inventory.

Note

This function should be used to add processed products that have been manually added by the user. This function does not need to be called when you are downloading products via gs_downloader as they are added to the inventory automatically.

Returns:added_uuids – List of strings containing the UUIDs of the products that have been successfully added to the inventory. Return type:list

getsentinel.gs_localmanager.check_integrity()

Checks the integrity of the current inventory.

Adds any products that were manually added to the DATA_PATH by the user since the last check. Removes any missing products.

Note

Any manually added products that have already been processed must be explicitly added via add_new_products.

Returns:

True if successful, will throw an error otherwise.

Return type:

bool

Raises:

• RuntimeError – If no unique match for a user added product can be found.
• RuntimeError – If the user added product filename does not start with ‘S1’ or ‘S2’
• UserWarning – If a processed product is manually added to the DATA_PATH without having been added to the inventory by called add_new_products.

getsentinel.gs_localmanager.get_product_inventory()

Returns the product inventory as a dictionary keyed by product UUIDs.

Returns:product_inventory – Dictionary of products that are in the DATA_PATH directory, keyed by their UUIDs. Return type:dict

getsentinel.gs_processor module

Channels under-processed products to the relevant ESA processing tools.

This modules takes an individual downloaded product UUID and processing the corresponding product. The resultant processed product is stored in the product inventory and the new UUID under which it is stored is returned.

Note

Sentinel-2 products that are not Level-1C will be skipped and their current uuids and product info returned.

Example

from getsentinel import gs_process

level1c_s2products = {uuid: info, uuid: info}

for uuid in level1c_s2products:
    level2a_s2products = gs_process.batch_process(uuid)

getsentinel.gs_processor.batch_process(product_inventory, gpt_graph=False)

Processes a batch of product and returns the new processed products uuids and info.

Parameters:

• product_inventory (dict) – Contains products as supplied in {uuid: info, uuid: info} format.
• gpt_graph (str, optional) – The path to the gpt tool graph to be used to process any Sentinel-1 products supplied.

Returns:

Contains the new processed product uuids and info.

Return type:

dict

getsentinel.gs_processor.process(uuid, gpt_graph=False)

Channels a product through the corresponding ESA processing tool.

Parameters:

• uuid (str) – UUID of the product to be processed.
• gpt_graph (str, optional) – The path to the gpt tool graph to be used to process a Sentinel-1 product.

Returns:

• str – The UUID of the newly generated processed product as it appears in the product inventory.
• dict – Contains the new product info with parameters changed to reflect the processing that has occured.

getsentinel.gs_stacker module

Extracts data from Sentinel products for given regions of interest.

Postage stamp ROIs out of given rasters file in batch. After processing the Stacker object acts as a dictionary from which numpy arrays containing all the multi-temporal data layers.

Handles both Sentinel-1 and Sentinel-2 rasters over multiple time frames.

Example

from getsentinel import gs_localmanager, gs_stacker

products = gs_localmanager.get_product_inventory()

roi = 'path/to/test_field.geojson'
start = datetime.date(2018, 5, 6)
end = datetime.date(2018, 5, 7)

stacker = gs_stacker.Stacker(products, roi, start, end)

# get the True Colour Image at resolution 10m
stacker.set_bands(s2_band_list=['TCI'], s2_resolution=10)

data = stacker.generate_stacks()

class getsentinel.gs_stacker.Stack

Bases: numpy.ndarray

Used to add an attribute to an existing numpy array. adapted from: https://docs.scipy.org/doc/numpy-1.12.0/user/basics.subclassing.html

Also masks all the zero values out of the array.

Parameters:

• input_array (numpy.ndarray) – An existing array that is to be converted to a Stack.
• info (list, str) – list of str or just str that contains info on each layer such as UUID of origin, platform, date of capture, band, and processing level.
• tranform (affline.Affline) – the out_transform output from the mask function of the rasterio library.
• name (str, optional) – The name of the Stack. Used in the final array stacks to name the stacks with the str of their corresponding shapefile names.

info

list, str – Copy of parameter info.

tranform

affline.Affline – Copy of parameter transform.

name

str, optional – Copy of paramter name.

class getsentinel.gs_stacker.Stacker(product_list, geo_files, start_date, end_date)

Bases: object

Creates stacks of arrays from a product list and a group of shape files.

Arrays contains the raster data each passed shapefile over Sentinel products covering different dates/times.

Note

The user does not need to concern themselves with checking that the products passed contain data for for all the shapefiles. This class will sort products to their corresponding shapefiles and filter out all null data.

Parameters:

• product_list (dict) – Contains the Sentinel products containing data relevant to the area coverered by the passed shapefiles or geojsons.
• geo_files (list or str) – A list of str containing the paths to all of the relevant shapefiles or geojsons or a single str.
• start_date (datetime.date) – The start of the time period used for data extraction. Any Sentinel products passed to the Stacker but generated outside of this time period will be excluded from the extraction process.
• end_date (datetime.date) – The end of the time period used for data extraction. Inclusive.

config

gs_config.UserConfig – Class container for user configuation info.

products

dict – Filtered copy of product_list passed to the object containing only products generated between start_date and end_date.

product_boundaries

dict – Contains shapely.geometry.Polygon objects describing the boundaries of each of the products in products.

geo_files

list – Contains a copy of geo_files passed to the object.

start_date

datetime.date – A copy of start_date passed to the object.

end_date

datetime.date – A copy of the end_date passed to the object.

job_list

dict – Contains keys of all the uuids of products passed to the object, with their values being the shapefiles that are within the boundaries of each product.

stack_list

dict – After the run method is called, this dict is populated with keys that are the names of the shapefiles passed, with their values being the corresponding processed numpy arrays.

band_list

list – list of str containing the product bands that are to be extracted from the products.

check_weather(cloud=False, snow=False)

Set a threshold for the probability of cloud or snow cover in the region of interest.

If the probability cover provided by the ESA in the automated cloud and snow masks is higher than the user-defined threshold, a warning is thrown during stacking.

Parameters:

• cloud (int, optional) – The percentrage threshold cloud probability for regions of interest, above which the stacker throws a warning.
• snow (int, optional) – The percentrage threshold snow probability for regions of interest, above which the stacker throws a warning.

Returns:

Return type:

None

generate_stacks()

Runs the data layer extraction and stacking process.

Returns:

Contains all the generate Stack objects keyed by the names of their corresponding shapefiles.

Return type:

dict

Raises:

• NotImplementedError – If S2 products have been passed to Stacker.
• ValueError – If set_bands has not been called before calling run.
• ValueError – If any products other than S1 or S2 products are passed to Stacker.

set_bands(s1_band_list=[], s2_band_list=[], s2_resolution=False)

Sets the attribute band_list to the passed bands.

Valid bands for S1 GRD products there are: ‘vv’, ‘vh’. Valid bands for S2 L2A products are:

‘10’: [‘AOT’, ‘B02’, ‘B03’, ‘B04’, ‘B08’, ‘TCI’,

‘WVP’],

‘20’: [‘AOT’, ‘B02’, ‘B03’, ‘B04’, ‘B05’, ‘B06’,

‘B07’, ‘B8A’, ‘B11’, ‘B12’, ‘SCL’, ‘TCI’, ‘WVP’],

‘60’: [‘AOT’, ‘B02’, ‘B03’, ‘B04’, ‘B05’, ‘B06’,

‘B07’, ‘B8A’, ‘B09’, ‘B11’, ‘B12’, ‘SCL’, ‘TCI’, ‘WVP’]}

Parameters:

• s1_band_list (list) – list of str containing the S1 bands the user wants extract from the S1 products passed to Stacker. Required form : [‘band1’, ‘band2’, … ]
• s2_band_list (list) – list of str containing the S2 bands the user wants extract from the S2 products passed to Stacker. Required form : [‘band1’, ‘band2’, … ]
• s2_resolution (int, optional) – Choose the band resolution of the Sentinel-2 bands. Can be 10, 20, or 60.

Returns:

Return type:

None

Raises:

ValueError – If the band strings passed are not valid bands.

Module contents