Travis times out on cloning sample models on Mac build

[bwasty]

Had two failed builds in my project (got a dependecy on the wrapper branch) and saw it also happened once here:
https://travis-ci.org/alteous/gltf/jobs/242077737

Perhaps it would be an alternative to download the zip file (https:/KhronosGroup/glTF-Sample-Models/archive/master.zip). The repo is a 1GB after cloning, the .zip only 540MB and downloads in ~2min for me.

[alteous]

That would be a good alternative. This could be merged with https:/alteous/gltf/issues/39.

[alteous]

N.B. the download link is actually https://codeload.github.com/KhronosGroup/glTF-Sample-Models/zip/master.

[alteous]

Since glTF-Sample-Models was made a 'shallow' submodule (i.e. only the last commits are cloned) this issue seems to have stopping occuring. I hope this will continue to be the case in the future.

At the moment I'm not experienced enough with version control systems to figure out a better solution. Downloading the zipped models worked on my own OS (Kubuntu) but there seems to be no cross-platform way without depending on significantly more crates.

Has the download time improved for you since this change was made?

[bwasty]

Actually I had timeouts again, though I'm not sure I was on the correct branch (didn't find that commit in the history)...let me try again.

[bwasty]

Hmm, no luck: https://travis-ci.org/bwasty/gltf-viewer/jobs/252109361
It's pretty weird, considering that on Linux the whole build takes under 4 minutes.
I don't have a good idea at the moment (except commenting out the OS X build while using gltf from git). There is https://docs.travis-ci.com/user/common-build-problems/#Build-times-out-because-no-output-was-received, but this requires some more changes to the travis config I guess (since there is no explicit command one could put travis_wait before).

[bwasty]

Idea I had while once again waiting for the samples submodule (...happens locally for each cargo update):
extract the samples to a dev dependency somehow. Not sure about the details though.

[alteous]

extract the samples to a dev dependency somehow

How might this help?

[alteous]

What would be really great is if we could package up pre-compiled crates. I have a slightly crazy idea of using include_bytes! to bake the models into a library. That way we only package the models we need and without any git metadata.

[bwasty]

extract the samples to a dev dependency somehow
How might this help?

It would only help dependent crates, so it wouldn't be a full solution (... These dependencies are not propagated to other packages which depend on this package.)

I have a slightly crazy idea of using include_bytes! to bake the models into a library.

I'd prefer to keep it file-based because I'll probably also need to add the samples to the viewer eventually for integration tests.

Did you see KhronosGroup/glTF-Sample-Models#26 (comment)?
GitZip might be worth a try. And otherwise, a package made by the sample models CI would probably preferable to something Rust-specific.

[bwasty]

Actually, going back to the zip idea...why not add reqwest and zip as dev dependencies and use them in the test setup to download and unzip? Should be simple and cross-platform.

Also, I tried out S3 for fun - here's a 330MB zip with only the 2.0 samples:
https://s3.eu-central-1.amazonaws.com/gltf-samples-models-2.0/2.0.zip (did a manual upload; the idea was that travis could to that automatically).

[bwasty]

I've had a jab at KhronosGroup/glTF-Sample-Models#77:
KhronosGroup/glTF-Sample-Models@master...bwasty:generate_index.
With the model index file, the download could be more fine-granular than with a zip and wouldn't require S3 or the like (can just use raw.githubusercontent.com for individual files).
I realised though that the names of binary/image aren't included - one way to deal with that would be to add an http Source that caches results on the filesystem (so they're not downloaded on every test run).

What do you think?

[alteous]

The index generator script is great!

can just use raw.githubusercontent.com for individual files

How do you do this? I keep getting 404 for anything other than the whole project.

caches results on the filesystem

That would be useful for local builds but I'm not sure how useful it would be for travIs et al.

[bwasty]

How do you do this? I keep getting 404 for anything other than the whole project.

It's what the Raw button points to, e.g.
https://raw.githubusercontent.com/bwasty/glTF-Sample-Models/generate_index/2.0/model-index.json
https://raw.githubusercontent.com/KhronosGroup/glTF-Sample-Models/master/2.0/Box/glTF/Box.gltf

That would be useful for local builds but I'm not sure how useful it would be for travIs et al.

Yes, that's mainly for local builds. Although trust is set up to cache+reuse "Cargo artifacts" between builds, and it looks like custom directories could also be cached: https://docs.travis-ci.com/user/caching/#Fetching-and-storing-caches.

[alteous]

https://travis-ci.org/alteous/gltf/builds

(╯°□°）╯︵ ┻━┻

[bwasty]

The errors occur because the submodule is one commit behind master. I'd propose removing shallow=true again (I guess that's the cause).

[alteous]

Travis CI allows for caching crates with cargo. If we made a gltf-sample-models crate then we might be able to have this in the cache.

[alteous]

I'm a bit fed up with this issue. Until caching is feasible, I suggest we run the import tests locally. I'm currently doing this with an environment variable GLTF_SAMPLE_MODELS=$HOME/glTF-Sample-Models.

[bwasty]

I see #94 still had shallow = true. Without it builds are fine, at least on Linux...

[alteous]

Oh, I forgot about that, thanks for reminding me.

[alteous]

An OS X build passed!

Does this warrant closing of this issue?

[bwasty]

Looking at the times, it seems just under the timeout limit, so it's probably random chance if it will succeed. I noticed the travis config changed so I could copy-paste the solution used in the viewer: #107.

[alteous]

The Travis build doesn't time out often anymore. I'd like to resolve and close this issue as 'good enough'.

[alteous]

Resolution: 'Good enough'.