Skip to content

ashulmanWeWork/marquez

 
 

Repository files navigation

Marquez

CircleCI codecov status Gitter license docker Known Vulnerabilities

Marquez is an open source metadata service for the collection, aggregation, and visualization of a data ecosystem's metadata. It maintains the provenance of how datasets are consumed and produced, provides global visibility into job runtime and frequency of dataset access, centralization of dataset lifecycle management, and much more.

Status

This project is under active development at WeWork and Stitch Fix (in collaboration with many others organizations).

Documentation

The Marquez design is being actively updated and is open for comments.

Requirements

Building

To build the entire project run:

$ ./gradlew shadowJar

The executable can be found under build/libs/

Configuration

To run Marquez, you will have to define config.yml. The configuration file is passed to the application and used to specify your database connection. When creating your database, we recommend calling it marquez. Please copy config.example.yml:

$ cp config.example.yml config.yml

You will then need to set the following environment variables (we recommend adding them to your .bashrc): POSTGRES_DB, POSTGRES_USER, and POSTGRES_PASSWORD.

Note: All environment variables in config.yml are accessed with ${VAR_NAME}.

Running the Application

$ ./gradlew run --args 'server config.yml'

Then browse to the admin interface: http://localhost:8081

Running with Docker

$ docker-compose up

Getting involved

Contributing

See CONTRIBUTING.md for more details about how to contribute.

About

Collect, aggregate, and visualize a data ecosystem's metadata

Resources

License

Code of conduct

Stars

Watchers

Forks

Packages

No packages published

Languages

  • Java 99.3%
  • Other 0.7%