-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Bound for memory usage #81
Comments
Hi, it's me again :) Or there's something I'm missing :) (Actually, there's a post on stackoverflow on this very same issue...) |
@raspooti there is no way around this at the moment.
Yeah, things can be improved but what do you expect, this tool has features to downloads tons of articles along with their related data, it's bound to consume a lot of memory. This is especially true if you keep everything in python and keep growing the memory used. Until a better solution comes up you can wrap a new shell script for every 1000 articles or something and run them on cron (not in parallel) |
First, thanks a lot for the great tool. I've been trying it out, and seems magic (except for some corner cases, websites for which it doesn't work, etc) but really cool :)
However, I tried it in a setting with scarce ressources (1G of RAM), and I have the impression that the memory keeps growing build after build until ... memory error. I deactivated the memoize articles, tried to empty the articles, dereference the sources, but looks like a bunch of other things are also memoized, and kept in memory, with no means to deactivate them. What is the best way to handle this? How does newspaper handle the increase of memory usage build after build? Is there a limit?
Thanks again for the magic tool :)
raspooti
The text was updated successfully, but these errors were encountered: