-
Notifications
You must be signed in to change notification settings - Fork 25
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Include Production WSGI (Gunicorn) to replace Flask Default Server #263
base: main
Are you sure you want to change the base?
Conversation
Add Gunicorn to Dockerfile Signed-off-by: Kaiyi <[email protected]>
@KaiyiLiu1234 https://developers.redhat.com/articles/2023/08/17/how-deploy-flask-application-python-gunicorn#containerization is good reference to follow
I am not sure if the recommendation applies to containers / k8s.
The part applies to containers is don't run as root. However, binding to port 80/443 isn't needed for containers.
Again, this is fine for a containerized env. I feel following the article linked should be good enough to fix our issue. |
Included Gunicorn Instances for Model Server, Online Trainer (inactive), and Offline Trainer. All three of these applications make use of one Flask App, so three Gunicorn Instances should be set up to manage them all. Signed-off-by: Kaiyi <[email protected]>
Incorporate Gunicorn Server setup for Model Server and Offline Trainer. The setup is currently placed in a dockerfile test as changes need to be made in using the Kepler Model Server image with gunicorn rather than with python. Signed-off-by: Kaiyi <[email protected]>
@sthaha @sunya-ch I incorporated the Gunicorn changes into a test dockerfile for now. How we interact with the image will change now in docker composes and kubernetes resources (for instance command should not be python model_server.py - it should instead use gunicorn or no command as the CMD for the dockerfile can handle launching the gunicorn servers for model server, offline trainer and online trainer). I can also include in the makefile commands to run the new gunicorn test dockerfile in this PR or in a future PR? |
Added Makefile Tests which will easily build, run, and clean Model Server with Gunicorn Servers for Model Server and Offline Trainer. Signed-off-by: Kaiyi <[email protected]>
Makefile
Outdated
@@ -51,27 +55,42 @@ run-model-server: | |||
$(CTR_CMD) run -d --platform linux/amd64 -e "MODEL_TOPURL=http://localhost:8110" -v ${MODEL_PATH}:/mnt/models -p 8100:8100 --name model-server $(TEST_IMAGE) /bin/bash -c "python3.8 tests/http_server.py & sleep 10 && python3.8 src/server/model_server.py" | |||
while ! docker logs model-server | grep -q Serving; do echo "waiting for model-server to serve"; sleep 5; done | |||
|
|||
run-model-server-gunicorn-complete: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
run-model-server-gunicorn-complete: | |
run-model-server-prod: |
Makefile
Outdated
@@ -51,27 +55,42 @@ run-model-server: | |||
$(CTR_CMD) run -d --platform linux/amd64 -e "MODEL_TOPURL=http://localhost:8110" -v ${MODEL_PATH}:/mnt/models -p 8100:8100 --name model-server $(TEST_IMAGE) /bin/bash -c "python3.8 tests/http_server.py & sleep 10 && python3.8 src/server/model_server.py" | |||
while ! docker logs model-server | grep -q Serving; do echo "waiting for model-server to serve"; sleep 5; done | |||
|
|||
run-model-server-gunicorn-complete: | |||
$(CTR_CMD) run -d --platform linux/amd64 -e "MODEL_TOPURL=http://localhost:8110" -v ${MODEL_PATH}:/mnt/models -p 8105:8105 -p 9109:9109 --name model-server-gunicorn-complete $(GUNICORN_TEST_IMAGE) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
$(CTR_CMD) run -d --platform linux/amd64 -e "MODEL_TOPURL=http://localhost:8110" -v ${MODEL_PATH}:/mnt/models -p 8105:8105 -p 9109:9109 --name model-server-gunicorn-complete $(GUNICORN_TEST_IMAGE) | |
$(CTR_CMD) run -d \ | |
--platform linux/amd64 \ | |
-e "MODEL_TOPURL=http://localhost:8110" \ | |
-v ${MODEL_PATH}:/mnt/models \ | |
-p 8105:8105 -p 9109:9109 \ | |
--name model-server-prod \ | |
$(GUNICORN_TEST_IMAGE) |
cmd/base_gunicorn_config.py
Outdated
workers = int(os.environ.get('GUNICORN_PROCESSES', '2')) | ||
|
||
threads = int(os.environ.get('GUNICORN_THREADS', '4')) | ||
|
||
port = os.environ.get('GUNICORN_PORT', '8100') | ||
|
||
bind = os.environ.get('GUNICORN_BIND', '0.0.0.0:' + port) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
workers = int(os.environ.get('GUNICORN_PROCESSES', '2')) | |
threads = int(os.environ.get('GUNICORN_THREADS', '4')) | |
port = os.environ.get('GUNICORN_PORT', '8100') | |
bind = os.environ.get('GUNICORN_BIND', '0.0.0.0:' + port) | |
workers = int(os.environ.get('GUNICORN_PROCESSES', '2')) | |
threads = int(os.environ.get('GUNICORN_THREADS', '4')) | |
port = os.environ.get('GUNICORN_PORT', '8100') | |
bind = os.environ.get('GUNICORN_BIND', '0.0.0.0:' + port) |
cmd/base_gunicorn_config.py
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
how about we call it gunicorn_config.py ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lets keep config under a config
dir
dockerfiles/Dockerfile.test-gunicorn
Outdated
# ENTRYPOINT ["python3.8", "cmd/main.py"] | ||
#ENTRYPOINT [ "python3.8", "-u", "src/server/model_server.py" ] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
# ENTRYPOINT ["python3.8", "cmd/main.py"] | |
#ENTRYPOINT [ "python3.8", "-u", "src/server/model_server.py" ] |
dockerfiles/Dockerfile.test-gunicorn
Outdated
COPY src/estimate src/estimate | ||
COPY src/server src/server | ||
COPY src/train src/train | ||
COPY src/util src/util | ||
COPY cmd cmd |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why not copy the entire src and cmd ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it is my bad that when we run the test the built ./model will be inside the src folder. We need to fix that first then the src will be cleaned enough for entire copy.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I can check it out.
@@ -18,4 +18,5 @@ EXPOSE 8101 | |||
# port for Offline Trainer | |||
EXPOSE 8102 | |||
|
|||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
unrelated change?
Fix code errors requested by sunil. Signed-off-by: Kaiyi <[email protected]>
cmd/gunicorn.sh
Outdated
echo "Starting Model Server" | ||
export GUNICORN_PORT=$PORT_MODEL_SERVER | ||
gunicorn --config config/gunicorn_config.py src.server.model_server:app & | ||
echo "Starting Offline Trainer" | ||
export GUNICORN_PORT=$PORT_OFFLINE_TRAINER | ||
gunicorn --config config/gunicorn_config.py src.train.offline_trainer:app & | ||
wait |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we create separate containers for model-server and trainer?
This removes the issue of handling (unexpected) termination.
dockerfiles/Dockerfile.test-gunicorn
Outdated
# port for Offline Trainer | ||
EXPOSE ${PORT_OFFLINE_TRAINER} | ||
|
||
CMD ["sh", "cmd/gunicorn.sh"] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is the idea to replace current Dockerfile
with this?
Separated Gunicorn Offline Trainer and Model Server into separate containers. Signed-off-by: Kaiyi <[email protected]>
To Address: #259
A side effect of using Gunicorn is that it will replace the address with default address and default port (8000). This can be resolved by binding (ex. binding model_server to 0.0.0.0:8100). Binding with 0.0.0.0 is acceptable but according to Flask, it is better to also introduce nginx server to act as a reverse proxy. Will this be necessary to include?
Also, since model_server AND offline_trainer (and in future online_trainer) expects to use their own Flask Apps, supervisord is needed to run multiple CMD for model_server and offline_trainer in Dockerfile.
Currently, this changes here works for metal docker compose so long as python3.8 -u src/server/model_server.py is replaced by gunicorn -b -0.0.0.0.8100 -src.server.model_server:app. However, there is more functionality that can be introduced depending on what is acceptable or not.