Idempotent runs by locally tracking successful uploads (😘) #311
Conversation
…oaded the way yt-dlp keep track of downloaded items. - Amend bibanon#19 by adding optional idempotency between runs: While concurrent-instances can still rely on a "clean" download directory (bibanon#19) a single upload node can use --use-upload-archive can avoid source files removal. Subsequent runs of `tubeup` will omit files already uploaded. - Fixes bibanon#23 (and part of bibanon#233) - NB: Ability for upload_ia() to return None, paves the way to fix bibanon#36 or bibanon#109
|
#233 isn't a bug it's his inability to symlink. No. Also he needs to learn to use yt-dlp. No local saving capability. I'll tease out the command and put it in README.md so all of Jasons little minions need to save the the videos locally. But no we're not doing local saving. Ever. Do all of these needs to be in the same PR? |
|
Break the bug fixes out - and local no downloads aren't a bug, use yt-dlp - and I'll merge bug fixes. |
|
oh wait this is all one PR. Split and I'll merge piecemeal |
Do you mean, the tool must keep trying to bypass user control by forbidding any caching behavior or intelligent idempotency? Regarding split, it's not clear which parts you want in or out. Do you mind providing more details about:
Thank you! |
No. Learn to use yt-dlp. One can learn to use yt-dlps flags to download videos and metadata. I've offered to put a example command on the front page. No one wants to take the offer they want Tubeup to change. Alternatively use Tubearchivist. Of course the best is to just learn yt-dlps flags. We are not here to prevent or especially enable abuse of archive.org resources. Period. I do not on a personal level like seeing Archive.orgs servers filled with for example, Logan Paul or Linus Tech Tips videos. They can ban abusive people. And I'nm going to tip my hand here a bit and say I think I know where the desire for local saving comes from. We're not Archive Team. Learn to use yt-dlp. #19 - Fixed when the flag for delete on successful uploading was implemented YEARS ago - old and dead not changing.
This feature already exists and is on by default. VideoID are one by one searched for on Archive.org before rip, and anything existing is skipped. Thank you for your time on this PR. It's features either exist or aren't desired. Stick around you're needed for other things. Thank you. |
... the same way
yt-dlpkeep track of downloaded items.Amend Garbage collection without interference #19 by adding optional idempotency between runs: While concurrent instances can still rely on a "clean"
downloaddirectory (Garbage collection without interference #19) single-instance nodes can now use--use-upload-archiveto avoid source files removal. Subsequent runs oftubeupwill politely omit files already uploaded.Fixes Fixes #19 -- local files are deleted after successful upload to archive.org #23 (also requested in Change directory and no delete commands #233)
NB:
upload_ia()may now returnNoneinstead ofraising(paves the way to fix Script fails an entire channel upload if one video has already been uploaded by someone else. #36 or Add option to skip existing items #109)NB: most of the diff is tests and argument handling. Only
upload_iabehavior is actually change (if the new option is passed)