🌎 Modules

Fetchez comes builtin with 70+ different modules to access geospatial data from various remote apis and local file-systems.

Data Modules

Fetchez includes a Module System in its ModuleRegistry that allows you to access various geospatial data sources locally or from around the world.

Modules come with their own arguments to set different data types, modify outputs, set credentials, etc. Hooks can be used to modify or manage modules before, during or after fetching; allowing for full ETL processing workflows using disparate sets of data modules.

Modules define a specific dataset, either a full data collection from a government API, a simple REST service that distributes daily tides or a single file located on your hard-drive.

Module Bundles

Bundles are YAML configuration files that define a group of Modules, possibly with preset arguments and hooks and can be used in the same ways as standard modules. You can make your own, or use a pre-configured Bundle from Fetchez or it’s extensions.

Example

Define your bundle

Put this in your ~/.fetchez/modules/bundles/ plugin folder

name: grav_and_bath
description: >
  Some fast bathymetry data sources.
modules:
  - module: margrav
    args:
      weight: .01
  - module: nos_hydro
    args:
      datatype: "xyz"
      weight: .35
    hooks:
      - name: unzip
      - name: set_datatype
        args:
          data_type: "nos_xyz"
  - module: charts
    args:
      weight: .15
    hooks:
      - name: unzip
      - name: filename_filter
        args:
          match: ".000"
          stage: "file"
      - name: set_datatype
        args:
          data_type: "charts_000"

• Run it: Your new grav_and_bath bundle is now registrered in the BundleRegistry and available in the fetchez cli:

fetchez run -R loc:"portland, me" grav_and_bath

• Add it: You can also use the bundle as a module in recipes or can be referenced by other Bundles.

project:
  name: "my_harbor"
  region: loc:"portland, me"
  modules:
  - bundle: grav_and_bath
    args: {weight: 1.0}

Extending Bunldes (Plugins and Extensions)

Fetchez is generic. If you are building a custom tool and want to create a custom module or bundle your own custom set of modules, you can register them either in your project or in the ~.fetchez configuration directory and they will be discoverable with the fetchez.registry

To create an extension where your bundles can be installed and used by fetchez, make a directory called ‘modules’ in your project; add any python source files defining the module there or make a directory called modules/bundles add any YAML bundle definitions there and register them with fetchez in your pyproject.toml:

Modules

[project.entry-points."fetchez.modules"]
my_project_modules = "my_project.modules"

Bundles

[project.entry-points."fetchez.modules.bundles"]
my_project_bundles = "my_project.modules.bundles"