The data and pipeline to build the XXX dataset.
- collection/source.csv — the list of data sources by organisation, see specification/source
- collection/endpoint.csv — the list of endpoint URLs for the collection, see specification/endpoint
- collection/resource/ — collected resources
These files are now stored in AWS S3:
- collection/log/ — individual log JSON files, created by the collection process
- collection/log.csv — a collection log assembled from the individual log files, see specification/log
- collection/resource.csv — a list of collected resources, see specification/resource
We recommend working in virtual environment before installing the python requirements, makerules and other dependencies. Requires Make v4.0 or above.
$ make makerules
$ make init
$ make collect
The collected files can then be converted into a national dataset:
$ make
The software in this project is open source and covered by the LICENSE file.
Individual datasets copied into this repository may have specific copyright and licensing, otherwise all content and data in this repository is © Crown copyright and available under the terms of the Open Government 3.0 licence.