uv pip compile does not add extra-index-url to the generated output file

[hofrob]

I think #1143 was only about parsing it from an existing input file. Running pip-compile will add this flag (and probably others) to the generated output.

While testing uv, I haven't found out how to do this or if it's just missing.

uv

uv pip compile \
    pyproject.toml \
    --output-file requirements.txt \
    --extra-index-url=https://private.example.com/pypi

Generates:

# This file was autogenerated by uv v0.1.2 via the following command:
#    uv pip compile pyproject.toml --output-file requirements.txt --extra-index-url=https://private.example.com/pypi

pip-compile

pip-compile \
    --output-file requirements.txt \
    --extra-index-url=https://private.example.com/pypi

Generates:

#
# This file is autogenerated by pip-compile with Python 3.12
# by the following command:
#
#    pip-compile --extra-index-url=https://private.example.com/pypi
#
--extra-index-url https://private.example.com/pypi

Adding --extra-index-url ... to the requirements file would allow one to run uv pip sync requirements.txt without having to add the cli argument.

[charliermarsh]

Can you try passing --emit-index-url on the command-line?

[hofrob]

Yes, that works. It will add the default index-url too which makes sense. Thanks!

I guess this could be the default behavior?

[charliermarsh]

I thought it wasn't the default in pip-compile, so I left it as not-the-default. I'd need to verify...

[charliermarsh]

(Also, for whatever reason, I hid this in the CLI.)

[thomasaarholt]

There are two more related issues here

unsupported environment variables

uv doesn't support environment variables when the extra index url is supplied in requirements.in (it does work when passed via the --extra-index-url cli argument).
With the following requirements.in file:

# requirements.in
--extra-index-url https://${GEMFURY_READ_TOKEN}@repo.fury.io/oda/

numpy

uv pip compile \
    --emit-index-url \
    requirements.in

generates

# This file was autogenerated by uv v0.1.3 via the following command:
#    uv pip compile --emit-index-url requirements.in
--index-url https://pypi.org/simple
--extra-index-url https://$%7BGEMFURY_READ_TOKEN%[email protected]/oda/

numpy==1.26.4

uv doesn't find private packages?

uv doesn't find our company package called testy-foobar, while pip-compile does.
With the following requirements.in file:

# requirements.in

testy-foobar

uv

uv pip compile \
    --extra-index-url=https://${GEMFURY_READ_TOKEN}@repo.fury.io/oda/ \
    --emit-index-url \
    requirements.in

generates

  × No solution found when resolving dependencies:
  ╰─▶ Because testy-foobar was not found in the package registry and you require testy-foobar, we can conclude that the requirements are unsatisfiable.

pip-compile

pip-compile \
    --extra-index-url=https://${GEMFURY_READ_TOKEN}@repo.fury.io/oda/ \
    requirements.in

generates

#
# This file is autogenerated by pip-compile with Python 3.11
# by the following command:
#
#    pip-compile --extra-index-url='https://****@repo.fury.io/oda/' requirements.in
#
--extra-index-url https://<STRIPPED>@repo.fury.io/oda/

asn1crypto==1.5.1
    # via snowflake-connector-python
cachetools==5.3.2
    # via google-auth
# many more packages

[nsphung]

Furthermore, it doesn't use/manage PIP_EXTRA_INDEX_URL environment variable configuration as pip does too.

[zanieb]

You can use UV_EXTRA_INDEX_URL instead. We are hesitant to read pip's configuration directly.

[nsphung]

Hello @zanieb

Thanks for your answer.

I've just tried adding UV_EXTRA_INDEX_URL. I'm using uv 0.1.5 and I have the following error still:

  × No solution found when resolving dependencies:
  ╰─▶ Because there is no version of mylib==2024.01.01 and you require mylib==2024.01.01, we can conclude that the requirements are
      unsatisfiable.

mylib==2024.01.01 is a lib in my private pypi repository. It seems it doesn't use UV_EXTRA_INDEX_URL as well.

[iflare3g]

Hello @zanieb

Thanks for your answer.

I've just tried adding UV_EXTRA_INDEX_URL. I'm using uv 0.1.5 and I have the following error still:

  × No solution found when resolving dependencies:
  ╰─▶ Because there is no version of mylib==2024.01.01 and you require mylib==2024.01.01, we can conclude that the requirements are
      unsatisfiable.

mylib==2024.01.01 is a lib in my private pypi repository. It seems it doesn't use UV_EXTRA_INDEX_URL as well.

The same here :(
Tried both with the environment variable and by passing the --extra-index-url

[charliermarsh]

@nsphung -- Is mylib shadowed on PyPI? As in, is there a package on PyPI with the same name?

[nsphung]

@charliermarsh -- mylib is not on PyPI. It's a private lib/dependency only available from our private pypi repo. The resolution is working well with pip-compile from pip-tools with PIP_EXTRA_INDEX_URL environment variable setup.

[hofrob]

@nsphung The question is, is there another library on PyPI with the same name? Because this issue is separately tracked in #1377

[nsphung]

@hofrob The answer is No. mylib is not using the same name as another library on PyPI.

[charliermarsh]

@thomasaarholt -- Unrelated to this issue, but heads up: if you're using Gemfury, you need to use pypi.fury.io instead of repo.fury.io in your index URL. Gemfury relies on the User Agent to do that mapping, and we (of course) don't send pip in our User Agent:

[charliermarsh]

@thomasaarholt - I filed your other issue here, thanks! #2035

[charliermarsh]

For clarity, this issue is now focused on whether we want to change our defaults around emitting index URLs.

[framillien]

For clarity, this issue is now focused on whether we want to change our defaults around emitting index URLs.

On this, maybe a source of confusion come from what is a pip index we should write.

By defaut pip-compile write all specified indexes, but not the default one.

When we activate the option, all pip indexes are written including the default one. And then if we activate this option by default (We should IMHO) we may want to create a new issue to only write specified indexes.

[iflare3g]

@charliermarsh hello! is there any update / plan for fixing this one ? #1502 (comment)

it's the main blocker for me ( and my team ) to replace our current slow workflow with a huge speed up from uv 😢

[thomasaarholt]

@iflare3g if you call uv pip compile --extra-index-url=<your-url> --emit-index-url requirements.in, then requirements.txt will contain it. This also works if requirements.in contains the extra-index-url in file itself.

[charliermarsh]

@iflare3g - are you referring to the index URL emission issue, or that a package isn’t being found in an alternate registry?

[iflare3g]

@iflare3g - are you referring to the index URL emission issue, or that a package isn’t being found in an alternate registry?

Second one, package isn't being found in private registry.
Tried via latest version of uv and still getting package not found 😔

[thomasaarholt]

i am successfully downloading and installing private packages using uv. Charlie's tip on the updated gemfury address fixed it for me.

What is your full command (hiding any sensitive token), @iflare3g ?

[iflare3g]

i am successfully downloading and installjng private packages using uv. Charlie's tip on the updated gemfury address fixed it for me.

What is your full command (hiding any sensitive token), @iflare3g ?

With UV_EXTRA_INDEX_URL=https://**:****@****.pkgs.visualstudio.com/_packaging/***/pypi/simple/ges/pypi/simple/ set
the command is just:

uv pip compile -o src/requirements/build.txt src/requirements/build.in --emit-index-url

without the env var, the command is:

uv pip compile -o src/requirements/build.txt src/requirements/build.in --emit-index-url --extra-index-url=https://**:****@****.pkgs.visualstudio.com/_packaging/***/pypi/simple/ges/pypi/simple/

and the result is for both:

 × No solution found when resolving dependencies:
  ╰─▶ Because xxx>=5.0 was not found in the package registry and you require xxx>=5.0, we can conclude that the requirements are unsatisfiable.

We use Microsoft's DevOps Artifacts for mirroring PyPI

[charliermarsh]

The only thing that knowingly doesn't work is if you have a package that exists both in your private registry and on PyPI (i.e., packages with the same name), and you expect to use a version of that package from PyPI instead of your private registry. Is that the case here? Does xxx exist on your private registry?

[iflare3g]

The only thing that knowingly doesn't work is if you have a package that exists both in your private registry and on PyPI (i.e., packages with the same name), and you expect to use a version of that package from PyPI instead of your private registry. Is that the case here? Does xxx exist on your private registry?

yup, only into private registry
it doesn't exist at all on pypi , so no risk of naming conflicts

[nsphung]

The only thing that knowingly doesn't work is if you have a package that exists both in your private registry and on PyPI (i.e., packages with the same name), and you expect to use a version of that package from PyPI instead of your private registry. Is that the case here? Does xxx exist on your private registry?

yup, only into private registry it doesn't exist at all on pypi , so no risk of naming conflicts

I'm waiting for this to be fixed as well. Else the resolution with index-url + extra-index-url is not working with uv compile vs pip-compile one.

[iflare3g]

The only thing that knowingly doesn't work is if you have a package that exists both in your private registry and on PyPI (i.e., packages with the same name), and you expect to use a version of that package from PyPI instead of your private registry. Is that the case here? Does xxx exist on your private registry?

yup, only into private registry it doesn't exist at all on pypi , so no risk of naming conflicts

I'm waiting for this to be fixed as well. Else the resolution with index-url + extra-index-url is not working with uv compile vs pip-compile one.

me 2, it's the only thing that blocks me and my team to fully switch to uv and we'd like to do as soon as this will be fixed and released

[thomasaarholt]

@iflare3g, I am successfully using uv to install a private package called fabrica-polars, which only exists in a private registry. Using the following style commands:

uv pip install --extra-index-url https://${GEMFURY_READ_TOKEN}@pypi.fury.io/oda/ fabrica-polars

Or with a requirements.in, I would call uv pip compile --emit-index-url

# my requirements.in
--extra-index-url https://${GEMFURY_READ_TOKEN}@pypi.fury.io/oda/
fabrica-polars

Ensure that you have updated uv in the venv you are using.

[iflare3g]

@iflare3g, I am successfully using uv to install a private package called fabrica-polars, which only exists in a private registry. Using the following style commands:

uv pip install --extra-index-url https://${GEMFURY_READ_TOKEN}@pypi.fury.io/oda/ fabrica-polars

Or with a requirements.in, I would call uv pip compile --emit-index-url

# my requirements.in
--extra-index-url https://${GEMFURY_READ_TOKEN}@pypi.fury.io/oda/
fabrica-polars

Ensure that you have updated uv in the venv you are using.

Already tried with the latest one, otherwise I'd would have commented here the resolution for me.
It seems for gemfury is ok, but with Microsoft's Azure Artifacts it doesn't work while pip-tools or just pip with extra index url do

If more details are needed, I can share the verbose compile command result

[thomasaarholt]

I think it would help if you shared the exact command you are running except for any secrets, and if you stick the verbose compilation log in a gist and share that too, I'm sure that would be helpful.

Edit: Sorry, I see you shared it above!

[charliermarsh]

@iflare3g - Do you mind following up here? #2138 I think these issues are unrelated to the --extra-index-url emit behavior, and it's making the thread a little hard to follow.

[iflare3g]

@iflare3g - Do you mind following up here? #2138 I think these issues are unrelated to the --extra-index-url emit behavior, and it's making the thread a little hard to follow.

Sure, thanks!