By Manuel Villa.
This repository shows the methodology used to obtain data for The Seattle Times story published on 5 April 2020, How the coronavirus overwhelmed Washington state’s early efforts to contain it .
The results are summarized in the splendid vizualization by Emily Eng you can see at the bottom.
Staff reporter Dan Gilbert obtained the Washington state Military Department's database of requests from healthcare organizations and government agencies for resources to respond to the coronavirus outbreak. We obtained the 70 PDF files saved down in the folder source_data/pdf. The objective was to convert these files into structured data.
Not having time nor the will to reinvent the wheel, we followed the least-resistance approach: to use Tabula. This marvelous tool efficiently transforms tables containd inside PDFs into structured data formats, such as CSV or JSON. The tradeoff for such convenience, however, is that those PDF tables need to be in pristine shape for Tabula to efficiently recognize and convert them. That was not the case with the records we obtained. So to use it, we would first need to overhaul the PDFs to Tabula's taste.
There are two scripts in the repository:
The PDFs had a lot of information that we did not need. Not only that, but that information's layout in the PDF kept throwing Tabula off. This script cropped away areas from all pages that constituted nothing but noise. Then we produced new PDFs that did not include those unnecessary areas and fed them to Tabula.
After feeding the cropped PDFs to Tabula, it very kindly produced CSV files that contained the data it scraped off from them. This was still very raw data that needed a lot of polishing and standardizing. This script did that.
Both scripts are populated with comments that we hope will make them easy to follow and to understand the rather simple structure of the repository.
Questions or comments are always welcome.
Cheers,
Manuel Villa
