Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Garbage collection without interference #19

Closed
vxbinaca opened this issue Jul 17, 2016 · 3 comments
Closed

Garbage collection without interference #19

vxbinaca opened this issue Jul 17, 2016 · 3 comments
Labels
feature request help wanted

Comments

@vxbinaca
Copy link
Collaborator

vxbinaca commented Jul 17, 2016 •
edited
Loading

REGEX needs to be used to do this.

Goals:

  • Clean video and metadata from disk on upload
  • Don't interfere with another instance of tubeups work. EX: one process is uploading a channel while another simultaneous instance is downloading it (max bandwidth utilization). So no quick and easy rm ~/.tubeup/downloads/* on exit.
@jjjake
Copy link
Collaborator

jjjake commented Jul 22, 2016

@vxbinaca I think we could use the delete parameter in internetarchive.upload() to deal with this (if I'm understanding it correctly).

When delete is set to True in the upload() call, the local files are deleted after the file has been successfully uploaded. Successfully uploaded means that the local md5 matches the md5 calculated by IA-S3 after the file has been handed off, and a 200 status is returned.

@vxbinaca
Copy link
Collaborator Author

@jjjake Yes that sounds like it would work quite well.

My fiddling is literally done. I've cleaned up the comments and testing opts from when it was a script. So when you're ready I'll make a new version tag.

But then again, if you click on the link in your email, you can do that yourself if you'd like. You'll have write access to the repo.

@jjjake
Copy link
Collaborator

jjjake commented Jul 22, 2016

Awesome, sounds great @vxbinaca.

@jjjake jjjake closed this as completed in 6d1b46c Jul 22, 2016
antonizoon pushed a commit that referenced this issue Jan 22, 2018
@jjjake
Merge pull request #23 from jjjake/master
b5c3e99 
Fixes #19 -- local files are deleted after successful upload to archive.org
drzraf added a commit to drzraf/tubeup that referenced this issue Sep 17, 2023
@drzraf
Idempotency: Keep a local "archive" (index) of files successfully upl…
feaa124 
…oaded the way

  yt-dlp keep track of downloaded items.

- Amend bibanon#19 by adding optional idempotency between runs:
  While concurrent-instances can still rely on a "clean" download directory (bibanon#19)
  a single upload node can use --use-upload-archive can avoid source files removal.
  Subsequent runs of `tubeup` will omit files already uploaded.

- Fixes bibanon#23 (and part of bibanon#233)
- NB: Ability for upload_ia() to return None, paves the way to fix bibanon#36 or bibanon#109
@drzraf drzraf mentioned this issue Sep 17, 2023
Closed
drzraf added a commit to drzraf/tubeup that referenced this issue Sep 19, 2023
@drzraf
Idempotency: Keep a local "archive" (index) of files successfully upl…
d0b9df6 
…oaded the way

  yt-dlp keep track of downloaded items.

- Amend bibanon#19 by adding optional idempotency between runs:
  While concurrent-instances can still rely on a "clean" download directory (bibanon#19)
  a single upload node can use --use-upload-archive can avoid source files removal.
  Subsequent runs of `tubeup` will omit files already uploaded.

- Fixes bibanon#23 (and part of bibanon#233)
- NB: Ability for upload_ia() to return None, paves the way to fix bibanon#36 or bibanon#109
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
feature request help wanted
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jjjake @vxbinaca