Help: call for testing projects

[xd009642]

I'm going to be working on some substantial changes to tarpaulin (described here: #516 (comment)) which should:

1. Significantly speed it up for projects with a lot of dependencies or quite heavy dependencies
2. Fix numerous linker issues from FFI code
3. Improve accuracy of coverage.

So all in all this should impact the following issues:

#126 #340 #351 #496 #516 #517

But because of this big change I'm also looking for a variety of projects which:

1. Demonstrate usage of different language features (stable or nightly)
2. Don't currently work with tarpaulin
3. Are slow with tarpaulin
4. Have varying levels of dependencies or tests
5. Work on Linux and x64

I'll be using these to work out differences in performance and accuracy and preventing any regressions as the search space of potential issues is fairly large. If people could suggest projects that would be a great help.

So far I'll be using:

• https:/jonhoo/faktory
• https:/tokio-rs/tokio
• https:/rust-ndarray/ndarray
• https:/rust-ndarray/ndarray-linalg
• https:/clap-rs/clap
• https:/simd-lite/simd-json
• https:/czbiohub/bcl2fastr
• https:/serde-rs/serde
• https:/Kixiron/lasso
• https:/jonhoo/openssh-rs/
• https:/fadeevab/cocoon
• https:/rust-git/rsgit
• https:/stouset/secrets

[agersant]

Not sure if this one can help, I have a small library which:

• Works on Linux x64
• Is small (somewhere between 1k LOC and 2k LOC)
• Has high test coverage
• Showcases various false negatives in tarpaulin

Links:

• Source
• Codecov report

Of note, it has false positives for lines defining type alias within functions, which I don't see listed as a known issue in #351.

[xd009642]

Yeah that's good, small makes it easier for me to look at all the source 👍

[AngelOnFira]

I'd be happy to help get Veloren in on this 😄

• Works on Linux x64
• Is very large (100k+ loc)
• Only 5% coverage 😢
• 500+ deps

[xd009642]

Oh veloren should be good for gauging performance change between the two, cloned and in the testing folder ❤️

[ralpha]

I came across this because of #517. (which can be closed I think, and I don't think is really related to this issue)
The code base I'm working on could also be used for testing.

• Works on Linux x86-64
• Already use Tarpaulin in build pipeline
• Builds with nightly
• Has a fair amount of dependencies. (Clap, Rocket, Serde, Diesel, ...)
• Build times are long (because of database, Diesel, query checks), about 33min on single core.
• Coverage at ~20% and tests take about 12min (single core), docker containers are set up (and included in repo).

Code is opensource and on GitLab: https://gitlab.com/df_storyteller/df-storyteller
Rocket might also be a good candidate.

[xd009642]

This has been stalled slightly. I implemented my approach and found that without replicating the partial evaluation steps in the compiler I couldn't reliably mark lines that "are able to be hit". But removing -C link-dead-code did fix linker issues on a variety of projects.

My next step was to try and get it working with cargo rustc as that lets me apply linker flags to just the crate being compiled. But it only works on the package level not the workspace level (there's an issue for that now rust-lang/cargo#8651). I tried some other stuff anyone interested can see in the issue but to no avail.

Now I was ultimately hoping this could work alongside my ptrace based coverage or #549 and be part of the gradual mitigation as until llvm coverage is stable. But it might just be a case of waiting for a bit and being patient

[jhpratt]

The time crate (time-rs/time) has decent coverage (~95% on the main crate), and is roughly 5k lines of source code. Its dependencies are minimal ("standard" proc macro dependencies, a couple of macros for progressive enhancement, and system APIs). It definitely works on x86!

From manually looking at the coverage, it looks like there's a fair number of false negatives.

[silvanshade]

Another project you could use is wasm-lsp/wasm-language-server.

The code base isn't too large yet but the project is fairly complex and coverage is high (~99.5%). There were several false negatives earlier having to do with multi-line method chaining but I just reformatted the code to resolve those.

In order to check coverage you would want to invoke:

cargo xtask tarpaulin --rest='--features wasm-language-server/corpus --features wasm-language-server-parsers/corpus'

[rubik]

@xd009642 I can submit a recent crate of mine for testing:

rubik/lobster - v0.2.0 on crates.io

• it has no dependencies
• it's a small library (about 500 lines of codes)
• it has a TON of Tarpaulin false negatives (e.g. see here: https://coveralls.io/builds/33629732/source?filename=src/orderbook.rs). A lot of lines which are obviously run are marked as uncovered.

Let me know if there's more I can do to help, Tarpaulin is a great and very useful project. Thanks for your time.

[xd009642]

So I'm just going to close this for now, to aid with testing I've created tater and added most of the repos here into the repo list as well as some other projects I got via some list of important rust projects and other users.

I'll be aiming to use this to spot out regressions and changes in coverage behaviour. Although the laptop I've been using it with can't quite handle it so will have to wait till I have access to my desktop again or consider some cloud compute instances 😅