Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Breeze build image or docs behind proxy #17158

Open
ciancolo opened this issue Jul 22, 2021 · 6 comments
Open

Breeze build image or docs behind proxy #17158

ciancolo opened this issue Jul 22, 2021 · 6 comments
Labels
area:dev-env CI, pre-commit, pylint and other changes that do not change the behavior of the final code area:dev-tools area:production-image Production image improvements and fixes good first issue kind:feature Feature Requests

Comments

@ciancolo
Copy link
Contributor

Hello all,

Description
Add to Breeze commands build-image and build-docs (needs also in other commands?) a parameter (or more) to specify proxy configurations for the image and docs build.

Use case / motivation
Recently I started to contribute to Airflow project, but I had some problems creating and configure the development environment with Breeze. My workstation is behind my company proxy and this creates a problem in the download of the necessary Python packages and the GPG public key during the image build. My workstation is already configured to work with proxy (I configured apt, docker...) but in this case, it didn't work.

I have a similar problem with ./breeze build-docs command.

Are you willing to submit a PR?

I'm not an expert in Breeze or bash script. Applying these 2 changes was enough to be able to build the image with Breeze:

  1. scripts/ci/libraries/_build_images.sh
    image

  2. scripts/docker/install_mysql.sh
    image

But of course, it will be better to use some global parameters, and I don't know if there are other cases where it is important to set proxy configuration.

@ciancolo ciancolo added the kind:feature Feature Requests label Jul 22, 2021
@boring-cyborg
Copy link

boring-cyborg bot commented Jul 22, 2021

Thanks for opening your first issue here! Be sure to follow the issue template!

@ciancolo ciancolo changed the title Breeze build image behind proxy Breeze build image or docs behind proxy Jul 22, 2021
@potiuk
Copy link
Member

potiuk commented Jul 22, 2021

Thanks! Very interesting and useful I think.

I think the way you described id is exactly how it should be done. Would you like to contribute a PR with those changes? In order to make it more "complete" the parameters should follow the convention AIRFLOW_HTTP_PROXY, AIRFLOW_HTTPS_PROXY and update the documentation in IMAGES.rst and maybe a few other places where build-args are mentioned (also in the instructions in `docs/docker-stack'.

I'd love if you contribute that change @ciancolo ? Would you :D ?

@potiuk potiuk added area:dev-env CI, pre-commit, pylint and other changes that do not change the behavior of the final code area:dev-tools area:production-image Production image improvements and fixes labels Jul 22, 2021
@ciancolo
Copy link
Contributor Author

I will try to propose a PR.
Do you know if there are tests for development env configurations?

@potiuk
Copy link
Member

potiuk commented Jul 26, 2021

I will try to propose a PR.
Do you know if there are tests for development env configurations

Most of the scripts in dev env are used in CI to run tests so pretty much all of them are tested in every single PR, same with Docker - it is build in a number of combinations and even automatically tested during the CI builds, so whenever you PR gets "green" you should not worry about regressions. About 90% of the dev scripts are also used for CI I'd estimate. And those that are not tested are mostly the ones that are very much "user-facing"

When it come to testing with proxy, we have no environment prepared to test it with a proxy, unless you want to set it up in our CI, but I think it's a bit too much hassle and it's a bit too "niche" to setup a CI for it, So I'd say review + manual testing should suffice.

@potiuk
Copy link
Member

potiuk commented Aug 1, 2021

Hey @ciancolo - how about the change here :) ?

@ciancolo
Copy link
Contributor Author

ciancolo commented Aug 4, 2021

Hi @potiuk,
It is still a WIP change. I prepared the parameters to set proxy configuration for image build and that seems to work fine, but it is necessary to set the proxy also in case run Airflow with Breeze local env: it is necessary for the installation of all the needed packages for webpage build.

Currently, I'm in vacation with no access to my dev env. I'm going to complete the work when I will back to work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:dev-env CI, pre-commit, pylint and other changes that do not change the behavior of the final code area:dev-tools area:production-image Production image improvements and fixes good first issue kind:feature Feature Requests
Projects
Status: No status
Development

No branches or pull requests

3 participants