Unable to push images > 545MB with https #287
Comments
|
The sregistry tool I'm eventually deprecating in favor of the singularity client to push images, so could you please post the full logs for pushing with singularity library:// endpoint? Thanks! |
nginx logsuwsgi logsThanks for your help! |
|
I can only really help if I can reproduce your issue (which I cannot). If it helps, it looks a little like this: https:/sylabs/singularity/pull/4850#issuecomment-563346984. it also seems that you are getting the same error for both singularity and sregistry, so the client is not relevant. The 443 message seems relevant though to be permissions, and perhaps this is SSL related, something about the version of SSL that you are using on your host (the bit I can't reproduce). See dirkjanm/PrivExchange#13 (comment). Anyway, not sure I can help. Good luck! |
|
hey @n8wood-brown could you please provide me a recipe to build / pull an image of this size? I just tested with 1.3 GB and it worked okay - during development I was testing with an 8GB image and that was working too. I'm thinking that this is specific to your internet connection or Singularity, but I want to be absolutely sure by trying the push myself. |
|
I'm still not getting anywhere with this. The |
|
I definitely think so - the way that I usually test and develop is on localhost, and I compile Singularity to allow for http:// for the remote endpoint (the link is provided in the docs on this page) https://singularityhub.github.io/sregistry/docs/client#singularity-push. Then you can use 127.0.0.1 as the remote, and see if the error reproduces there. If that works, then it's some issue with your setup. |
|
@vsoch - I recompiled Singularity and am still seeing the error. I ran a packet capture on the server to confirm that there were no connections over port 443. Here's the image file I was testing with (~4.4GB): Thanks! |
|
hey @n8wood-brown - I think this might be a memory issue? The reason is because when I first tried (not with having a lot of memory free) it started to upload, and then I got an EOF error. Once I freed up some memory and tried again, I didn't have any issue (although it took a while for the upload to finalize). vanessa@vanessa-ThinkPad-T490s:~/Downloads$ singularity push -U fmriprep-20.0.0.simg library://vsoch/test/fmriprep:20.0.0
WARNING: Skipping container verifying
4.5GiB / 4.5GiB [==============================================================================] 100 % 1.3 GiB/s 0sYou can see the image in my collection folder: $ tree images/test/
images/test/
├── fmriprep-sha256.92278b7c046c0acf0952b3e1663b8abb819c260e8a96705bad90833d87ca0874.sif
└── trax_centos-sha256.72a69eb0b4963f472058412477cb6b28d074627c409a0f1de8a144f276804c6f.sif
0 directories, 2 filesAnd my collection interface: I'm sorry that I'm not able to reproduce your error, but I suspect it has something to do with the size of the image (and what Singularity is doing to send it to the registry, note that I'm not privy to how this works but since they added multipart I suspect it's done in one streamed chunk) and possibly your memory or setup of your host. If there is something else you'd like me to test to try and reproduce I'd be happy to, but I am afraid I cannot help if I am unable to reproduce. |
|
Let me know if you'd like any information about my host, in case that helps. |
|
@vsoch - OK I started over from scratch and didn't use the Ansible set up this time around. I'm at least back to where I started. Pushing with http works again but https still does not. How do you test locally with a self-signed cert? I'm getting this: |
|
For allowing http, please follow the instructions that I linked here https://singularityhub.github.io/sregistry/docs/client#singularity-push. There is a link to the exact line in Singularity that you'll need to change before recompiling to allow for http. You shouldn't need to use a certificate, I'm not actually privy to the checks that Singularity does for them but it looks like it's not happy with self signed. |
|
I'm not sure that it makes a difference, but I also used 127.0.0.1 instead of localhost. |
|
I actually did compile a version of singularity with that change and it does work witth http. But I'm hoping to test locally with https and a self-signed cert so I can see if there's anything wrong with my real cert. I guess that's not an option. |
|
Ah okay. I haven't done anything with self signed certs (I usually go from http to certbot signed ones) so I'm afraid I don't have insight there. But I do think it would make sense (for a sanity check) to try the setup and push with just http. |
|
I'm planning to test this with Certbot once I have access to an Internet accessible system that can be verified by Let's Encrypt. |
|
ah, good luck! I guess I'm lucky, I generally deploy to a cloud instance so the hardest thing is navigating the slippery white interfaces to find networking / DNS configuration. Keep me updated! |
|
No luck on a new server set up with Certbot, uploads still die around 545MB. I'm still not sure if this a problem on the client side or server side. If you still have that large container I posted, would you mind seeing if you have any success? It's configured with only Google oauth at the moment. https://pshubcit.services.brown.edu |
|
@n8wood-brown please see #298 this should greatly help your issue! I wanted to get it done asap because I remembered you were having trouble. I just created an account, added the host as a remote, verified my token, and I'm pushing a large container (4.5GB): $ singularity push -U big.sif library://vsochat/bigcontainers/chonker:latest
WARNING: Skipping container verifying
544.5MiB / 4.5GiB [========>---------------------------------------------------------------------] 12 % 1.9 MiB/s 36m21s
FATAL: Unable to push image to library: error uploading image: Put https://pshubcit.services.brown.edu/v2/push/imagefile/3/d10a7ad8-8cb6-482d-9e3c-7a1fb771861c: write tcp 10.0.0.196:39952->128.148.254.126:443: write: connection reset by peerI reproduced what you are seeing, and I would suspect this is an issue due to the size of your server - if the memory limit cannot hold the entire file, it would be logical that it would fail. I've also tested sregistry many times recently on my current host, and I don't run into this issue (so it must be an issue of memory where it is hosted). Do you want to try #298 with multipart uploads to see if it helps? It would mean signed URLs, the uploads being done in parts, and if absolutely needed, you could host or scale the Minio for storage separately (and not burden the uwsgi container). I am hopeful that this PR will fix this issue for you, sorry it's been such a trouble! |
|
@vsoch - I just doubled the memory to 32GB and that didn't help. I'll look into the mulipart uploads link you posted. Thank you so much for the attention you've put into this! |
|
I wasn't able to figure this out so we ended up allowing http instead. Thank you @vsoch for your help! |
|
Alrighty, but please try the new Minio backend if you want https, it works remarkably better than my original implementation. |
Hi, I'm having trouble pushing larger images through the GUI, sregistry-cli, and singularity when https is enabled. I see there have been similar issues resolved in the past, I've gone through those threads multiple times to try out different suggestions but haven't had any success. I'm not sure if this is a bug or a misconfiguration on my part. When I change
SREGISTRY_REGISTRY_BASEto use http everything works.singularity version: 3.5.3
sregistry version: 1.1.22
sregistry-cli version:0.2.35
Success with an image < 545MB
Failure with a large image
The upload dies when it reaches around 554MB
nginx logs
uwsgi logs
config.py
nginx.conf
The text was updated successfully, but these errors were encountered: