Write script to update all data for Scribe updates

[andrewtavis]

Currently updating Scribe's data for a release is time consuming, with this process being one that could easily be converted to a single Python script. The goal of this issue is to make a script that uses a Wikidata tool to run all of the query*.sparql scripts in the Scribe-Data/data directory, and then runs all the format_*.py scripts to put the new information into the respective Data directories for the keyboards. The script would be placed in the root of Scribe-Data/data.

• As of now it would likely be best if translation scripts were not included as these are not being updated until Deleted: Add translation data to Wikidata Scribe-Data#17 is finished

Another positive is that this feature will provide many good first issues for the future where someone can update the data and thus be integrated into the contribution process.

[andrewtavis]

The script written to complete this issue should also produce a .txt/.json file that has the differences in data for the most recent run. The structure of this file should allow it to be directly pasted into CHANGELOG.md, thus turning the data update process into two steps (run script and paste results).

[andrewtavis]

It looks like WikidataIntegrator would be an appropriate tool to complete the SPARQL query portion of this task, as shown in this Jupyter notebook.

[andrewtavis]

Current progress is a script that can find the necessary queries, picks the first to run, formats it for use with WikidataIntegrator, runs the query and prints the results of the query.

Next steps:

• Handle errors from WikidataIntegrator when the query doesn't work
• Allow Scribe-Data/data to take arguments for certain languages and data types
• Save results that are currently being printed into a JSON
• Run coinciding formatting script over the saved file to update Scribe's data
• Compare and save results in Scribe-Data/data/_update_files/total_data.json
• Update Scribe-Data/data/_update_files/data_updates.txt

[andrewtavis]

Labels for properties do not appear to be being queried with WikidataIntegrator, but this can be checked for in formatting scripts where the string will start with Q and be followed by an int. In this case these values can be translated to their labels prior to being assigned.

Example: map_genders of format_nouns.py scripts can also check for QIDs and then convert them to abbreviations.

[andrewtavis]

The final step is to allow the individual formatting files to detect when they're being ran by Scribe-Data/data/update_data.py by checking the Python paths via sys.argv[0].

[andrewtavis]

Note that the result of this issue is now scribe_data/load/update_data.py, and all referenced files have been moved to Scribe-Data.