Supporting gs urls for running Toil on Slurm

[mobinasri]

I have a test WDL that outputs the sizes of the input files. It takes an array of file urls and outputs a text file. I'm running this on a Slurm machine. It works fine when I pass the public url of an object in a public google bucket. But when I pass the gs url it says it does not exist. Isn't Toil supposed to work with gs urls?
I checked my access to the gs url:

gsutil ls -lh gs://masri/flagger/v0.3.0/chm13v2.0.censat.bed
218.93 KiB  2023-05-14T22:15:02Z  gs://masri/flagger/v0.3.0/chm13v2.0.censat.bed
TOTAL: 1 objects, 224187 bytes (218.93 KiB)

Here is the command I'm using for running Toil on Slurm

rm -Rf logdump && rm -Rf ./trash/tree && mkdir logdump && time SINGULARITY_CACHEDIR=`pwd`/trash/cache/.singularity/cache MINIWDL__SINGULARITY__IMAGE_CACHE=`pwd`/trash/cache/.cache/miniwdl toil-wdl-runner -e SINGULARITY_CACHEDIR -e MINIWDL__SINGULARITY__IMAGE_CACHE --disableProgress --logDebug --retryCount 0 --jobStore ./trash/tree --outputDirectory ./trash/out --outputFile ./trash/out.json --batchSystem slurm --batchLogsDir `pwd`/logdump `pwd`/get_sizes.wdl `pwd`/inputs_public_url.json 2>&1 | tee log_public_url.txt

I've attached the WDL, input json files and the log files I get for each json file; one with gs url and one with public url.
test_gs_url.zip

┆Issue is synchronized with this Jira Story
┆Issue Number: TOIL-1337

[adamnovak]

OK, I checked this and I think the real problem is that Toil is not installed with the [google] extra, which is required for it to understand gs:// URLs. Just having gsutil available on the PATH isn't enough.

@mobinasri Can you reinstall Toil with google in the list of extras (along with wdl) and try again?

I have a PR now to change the error reporting so that it will suggest that you install a missing extra if it finds a URL it can't understand.

[mobinasri]

It is now working fine with gs uris for me. Thanks!