Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create classes for common harvest transforms: Filters, Excludes, Overrides, Defaults #250

Open
janette opened this issue Nov 12, 2019 · 0 comments

Comments

@janette
Copy link
Member

janette commented Nov 12, 2019

Transforms

Add common transform classes to allow user to fine tune the results of the harvest.

Filters
Filters restrict the datasets imported by a particular field. For instance, if you are harvesting a data.json source and want only to harvest health-related datasets, you might add a filter with "keyword":"heatlh"

Excludes
Excludes are the inverse of filters. For example, if you know there is one publisher listed on the source whose datasets you do not want to bring into your data portal, you might add "publisher.name":"Agency of Untidy Data"

Overrides
Overrides will replace values from the source when you harvest. For instance, if you want to take responsibility for the datasets once harvested and add your agency's name as the publisher, you might add "publisher.name":"My agency". This would replace any existing value.

Defaults
Defaults work the same as overrides, BUT will only be applied if the relevant field is empty in the source

Bonus points

Build a UI for the user to create a harvest.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant