Added command line switch that allows saving to proto format #49
Conversation
| */ | ||
| public class AtlasProtoOutputFormat extends AbstractFileOutputFormat<Atlas> | ||
| { | ||
|
|
| } | ||
| return this.format.getRecordWriter(fileSystem, job, name, progress); | ||
| } | ||
|
|
There was a problem hiding this comment.
Good catch thanks!
|
I changed the switch so the generation is proto by default. Now you must provide a switch |
There was a problem hiding this comment.
Thanks @lucaspcram! Any way those changes could be unit-tested (or integration tested)? The other ones are not, but we need to find new ways!
|
@matthieun I would imagine that the best way to (integration) test this would be to kick off a local Spark job, create a single atlas file (maybe we need a test PBF file in this repo?), and then verify its format using Or are you thinking of something simpler that could fit in a unit test? |
|
Added #51 |
An optional command line switch
useJavaFormat=truecan be provided to generate atlas files using the old java serialization format. When this switch is not provided it will be set to false by default, and the new protobuf format will be used.