Cannot complete jobs any longer

[twhiting9275]

Seeing a LOT of this lately, indicating that there's just too much going on when queue is run, and it's just not completing successfully

| HTTPConnectionPool(host='192.168.68.86', port=32400): Max retries exceeded with url: /library/sections/2/all?type=18&X-Plex-Container-Start=0&X-Plex-Container-Size=100 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x395F9070>: Failed to establish a new connection: [WinError 10055] An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full')) |

[meisnate12]

I'm at a loss on alot of network errors. The Plex Server can handle a ton of calls and if one of the calls fail it should retry after 10 seconds up to 6 tries all while it makes no other calls so i doubt its because of too many calls but truthfully im not sure.

[twhiting9275]

So, basically, "I don't know, and I'm not going to look at it".
Your program is broken at this point. It needs to be fixed

[meisnate12]

its more like thats a generic network error that I cant reproduce on my end therefore its near impossible for me to have any idea whats happening. I don't know anything about how your network is setup, how your running PMM, how your running Plex. PMM runs fine on many huge libraries so the problem has something to do with the setup not the code.

[twhiting9275]

Except it's not a 'network error'. This is a script error. There are far too many sockets opened at once with this script here.

A bit of research shows this is actually a socket error, and why, specifically this is going on. The best resource I could find on this, right here

You are clogging your TCP stack with outgoing data and all the connection establishment and termination packets.

Testing this shows that this is 100% correct
Running a netstat -nb when this script is running reveals the following results. Keep in mind , this is just after running for 5-10 minutes. The script is just getting warmed up
pmm.txt

That's just the stuff that's open when PMM runs. Better than 400 sockets open at the same time.

This isn't a network error. This is a script not utilizing network efficiently error, and it's a pretty big thing, especially if you're going to utilize larger collections.

[meisnate12]

when i run netstat -nb im not having that many sockets opening during my runs which makes this incredibly hard to figure out whats going on. Also how my script connects to Plex is handled by PlexAPI. If you send me the full traceback i can try and bring it up with them

[twhiting9275]

The Plex connection isn't the problem. It's every other connection that is being made. IMDB, TMDB, thetvdb, trakt, even Sonarr (which isn't even configured). 500+ Sonarr connections in the latest netstat pull.

Somehow, these connections need to be optimized and utilized more efficiently. It's amazing that with this many open connections that whoever owns these IP's hasn't banned mine yet

Like I mentioned, these connections are only open when this is running, and I have a very sneaking suspicion that it's IMDB causing this , more specifically the queries to IMDB's database when trying something like https://www.imdb.com/search/title/?sort=moviemeter&title_type=tv_series&year=2021 , or more importantly, how those are processed.

[twhiting9275]

More cons. Meant to add them in the last reply
pmm2.txt

[meisnate12]

More cons. Meant to add them in the last reply
pmm2.txt

I think the link is bad here

[twhiting9275]

downloaded fine for me, tried twice

[meisnate12]

try uploading to gist whenever i go to that link it just a blank page

so im running the imdb list you had above and now im seeing much more sockets open and i think i know why. when i kind a proper conversion for an IMDb ID to TMDb ID i save it in a local cache so i dont have to make that call again but everytime PMM cant find an ID conversion in its cache it will attempt to use TMDb and Trakt.tv to convert it. A vast number of movies in the list you mentioned do not have their IDs linked to TMDb or Trakt yet. Although even with that when i ran that list it completed just fine and closed all the sockets. I'll look into how to better manage network connections to see what i can do.

Are you able to make it through building that list or does it fail at some point during it everytime?

[twhiting9275]

Just ran into this, and I managed to get a couple of quick traces. The relevant stuff
1
2
Can't figure out how to create multiple gists, so 2 is on pastebin.

The first shows around 700 con's to 8989 (Sonarr) directly. The second shows over 850. The same things were happening this afternoon, based on the logging. Hundreds of calls to sonarr, despite sonarr configs literally being commented out (in config itself)

As far as the list:
No, that collection is never built. Looking through, it's definitely showing a ton of convert errors, titles unfound, etc, which means it's trying to do something there, and that shouldn't be using a separate connection each time. With thousands of titles, it's not surprising it's killing the connection buffers. This needs to be better optimized

[meisnate12]

if sonarr isnt configured in your config file PMM is not making calls to it how would PMM even know where to call? Do you have sonarr on the same machine as PMM? or do you have other apps that may be talking to sonarr.

Ultimately I'm just using the standard requests module where my calls are nothing more then requests.get(URL_HERE) same as the TMDb, Trakt, Arr, Plex API wrappers that PMM uses. They all do requests in the same way. Trying to optimize that is something well outside the scope of this project.

[meisnate12]

So i did some digging and found something that may be able to help. Check out the newest version of Develop. All requests now use use a Session object so it should handle connections much better and reuse sockets. The only parts that dont use Session ATM is TMDB, Trakt, Radarr, and Sonarr. I've updated the TMDb Wrapper and Arr Wrapper to use a Session just waiting for it to get approved. Trakt Im having a little bit more of an issue with but im working on it. Let me know if this helps out the issues you were having

[twhiting9275]

Running a job now, will update when that's done, or if it keeps going. Sessions should absolutely help keep those connections down, for sure.

One thing I noticed when updating (been a while since I did) is that I had to reimport modules (pip). Might want to make a note of that when people download so they aren't caught off guard. Either a new module dependency was added, or something. Took about 2 seconds to fix, obviously, just saying , many would be perplexed.

[meisnate12]

let me know how it goes! also i put a note in the release notes of every release that includes dependency changes for example https:/meisnate12/Plex-Meta-Manager/releases/tag/v1.11.0

Is there a different place you would suggest me mentioning it as well?

[meisnate12]

so Trakt TMDb and the Arrs should now all be using the same Session as well. hopefully this will help you run the program better let me know if youve got any issues

[twhiting9275]

Latest develop fails to start:

| Connecting to TMDb... | | __init__() got an unexpected keyword argument 'session'

I mean, that's ONE way to solve a problem ? ;)

[meisnate12]

Sorry you need to install requirements again I updated my arr and tmdb modules I don't have any dev release notes so there's no where for me to really say that at lol

[twhiting9275]

So, this is still failing . Definitely an IMDB thing
I've attached the template that's causing this.
If you leave the
'&countries=us&language=en'
bit in, it handles things fine
if you take that off, it will stop around 2013 or so, because there's 6k+ entries to process in those lists
Unfortunately with those two filters in, the system leaves out a handful of noticeable TV shows.

The decade entries are handled well. It's just the individual years, where thousands of entries are attempted at the same time.

MyTV.txt