Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document uv-with-Jupyter workflows #7625

Merged
merged 2 commits into from
Sep 26, 2024
Merged

Document uv-with-Jupyter workflows #7625

merged 2 commits into from
Sep 26, 2024

Conversation

charliermarsh
Copy link
Member

@charliermarsh charliermarsh commented Sep 22, 2024

Summary

This is a work-in-progress as I actually want to change some behaviors here.

Closes #6329.

@charliermarsh charliermarsh added the documentation Improvements or additions to documentation label Sep 22, 2024

Since the Jupyter server is running in an isolated virtual environment, `!uv pip install` will install package's
into _Jupyter's_ environment, not the project environment. Such dependencies may disappear on subsequent `jupyter`
invocations. To install packages into the project environment, use `!uv add`.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, it looks like this is solvable using kernel ipython kernel install (#6329 (comment)).


Since the Jupyter server is running in an isolated virtual environment, `!uv pip install` will install package's
into _Jupyter's_ environment, not the project environment. Such dependencies may disappear on subsequent `jupyter`
invocations. To install packages into the project environment, use `!uv add`.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I sort of think we should not be discovering the tool environment here, though. Modifying the project environment isn't great, but modifying the tool environment is even worse! It means subsequent users of the tool have these same modifications. I'd propose that we skip tool and ephemeral environments when considering active virtual environments. \cc @zanieb

- Select the virtual environment that uv created. It will be named .venv/bin/python in this dropdown
(or maybe .venv\Scripts\python on windows)

If you don't `uv add ipykernel`, the notebook will fail to execute with an error.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be nice if we could find a way to make this work without requiring that you install ipykernel into the project environment itself. (We don't want users to have to add ipykernel as a project dependency just to interact with their project in a notebook.)

Copy link
Contributor

@bluss bluss Sep 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is that not wanted?

But the 'Uv Run' kernelspec I posted in the other issue seems to be the most normal way to achieve exactly that - it runs the kernel using uv run --with ipykernel. Maybe a vscode extension is a better user interface, most of my questions are around the interface and if it's worth it (Because vscode already picks up .venv natively, is it worth the fuss with kernelspecs or other ways to do it - or can those external configuration ways of working be put on the long term plan, that is working on better uv + vscode integration to make these cases easy or automatic in the long run.)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, if I'm working on a project, and I want to play with that project in a notebook, I don't want to add notebook dependencies as dependencies to the project. I don't want them recorded in the lock, etc.

with access to the project's virtual environment via the following:

```console
$ uv run --with jupyter jupyter notebook
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm sure there are various opinions are about this, but jupyter notebook is the legacy interface and jupyterlab is the current interface. I don't know what relation other people have to this, would it give a better impression to use jupyterlab here? Command jupyter lab.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I went back and forth. I'm fine to use jupyter lab, no big deal.

Copy link
Contributor

@bluss bluss Sep 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that's the best, it's what jupyter promotes as #1

@radames
Copy link

radames commented Sep 23, 2024

Just one note, following a fresh new setup with vscode

uv init my-notebook
cd my-notebook
uv add --dev ipykernel
uv run ipython kernel install --user --name=uv
code .
!uv pip install diffusers
/usr/bin/sh: 1: uv: not found

The notebook kernel couldn't find the correct env path for uv nor pip , not sure if it's specific to my env however uv is set correct via my .zshrc and .profile . "$HOME/.cargo/env". Doing the same steps with pip doesn't lead to an issue

@charliermarsh
Copy link
Member Author

Good call, I experienced the same (though I don't see that when running Jupyter directly).

@radames
Copy link

radames commented Sep 23, 2024

indeed uv run --with jupyter jupyter lab works, however it looks like there are still some PATH issues. pyproject.toml is not an error, just started it from a fresh dir

image

with `uv pip install ipykernel`.

If you need to manipulate the project's environment from within the notebook, you may need to add
`uv` as an explicit device dependency:

This comment was marked as resolved.

@charliermarsh
Copy link
Member Author

Do you mean that !pip list and !nvidia-smi are missing?

@charliermarsh
Copy link
Member Author

Does !nvidia-smi work in other Jupyter notebooks for you?

@radames
Copy link

radames commented Sep 23, 2024

Does !nvidia-smi work in other Jupyter notebooks for you?

yes, starting a jupyter lab from a new conda env, the env path are working correctly

conda create -n temp python=3.10 
pip install jupyterlab
jupyter lab  

@charliermarsh
Copy link
Member Author

I haven't been able to reproduce that unfortunately.

with access to the project's virtual environment via the following:

```console
$ uv run --with jupyter jupyter lab
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Future: I wonder if we want uvx --with-project jupyter lab?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I kinda like that. The repeated jupyter would be nice to avoid.

Then, you can create the kernel for `project` with:

```console
$ uv run ipython kernel install --user --name=project
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could be cool to have a command that emits the project name, so you could do --name=$(uv ...) as a one-liner that actually works in any project.

Copy link
Member

@zanieb zanieb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small comments

@charliermarsh charliermarsh enabled auto-merge (squash) September 26, 2024 00:54
@charliermarsh charliermarsh merged commit 4ba0e56 into main Sep 26, 2024
50 checks passed
@charliermarsh charliermarsh deleted the charlie/jupyter branch September 26, 2024 00:55
tmeijn pushed a commit to tmeijn/dotfiles that referenced this pull request Oct 7, 2024
This MR contains the following updates:

| Package | Update | Change |
|---|---|---|
| [astral-sh/uv](https:/astral-sh/uv) | patch | `0.4.15` -> `0.4.18` |

MR created with the help of [el-capitano/tools/renovate-bot](https://gitlab.com/el-capitano/tools/renovate-bot).

**Proposed changes to behavior should be submitted there as MRs.**

---

### Release Notes

<details>
<summary>astral-sh/uv (astral-sh/uv)</summary>

### [`v0.4.18`](https:/astral-sh/uv/blob/HEAD/CHANGELOG.md#0418)

[Compare Source](astral-sh/uv@0.4.17...0.4.18)

##### Enhancements

-   Allow multiple source entries for each package in `tool.uv.sources` ([#&#8203;7745](astral-sh/uv#7745))
-   Add `.gitignore` file to `uv build` output directory ([#&#8203;7835](astral-sh/uv#7835))
-   Disable jemalloc on FreeBSD ([#&#8203;7780](astral-sh/uv#7780))
-   Respect `PAGER` env var when paging in `uv help` command ([#&#8203;5511](astral-sh/uv#5511))
-   Support `uv run -m foo` to run a module ([#&#8203;7754](astral-sh/uv#7754))
-   Use a top-level output directory for `uv build` in workspaces ([#&#8203;7813](astral-sh/uv#7813))
-   Update `uv init --package` command to match project name ([#&#8203;7670](astral-sh/uv#7670))
-   Add a custom suggestion for `uv add dotenv` ([#&#8203;7799](astral-sh/uv#7799))
-   Add detailed errors for `tool.uv.sources` deserialization failures ([#&#8203;7823](astral-sh/uv#7823))
-   Improve error message copy for failed builds ([#&#8203;7849](astral-sh/uv#7849))
-   Use `serde-untagged` to improve some untagged enum error messages ([#&#8203;7822](astral-sh/uv#7822))
-   Use build failure hints for `dotenv` errors, rather than in `uv add` ([#&#8203;7825](astral-sh/uv#7825))

##### Configuration

-   Add `UV_NO_SYNC` environment variable ([#&#8203;7752](astral-sh/uv#7752))

##### Bug fixes

-   Accept `git+` prefix in `tool.uv.sources` ([#&#8203;7847](astral-sh/uv#7847))
-   Allow spaces in path requirements ([#&#8203;7767](astral-sh/uv#7767))
-   Avoid reusing cached downloaded binaries with `--no-binary` ([#&#8203;7772](astral-sh/uv#7772))
-   Correctly trims values during wheel WHEEL file parsing ([#&#8203;7770](astral-sh/uv#7770))
-   Fix `uv tree --invert` for platform dependencies ([#&#8203;7808](astral-sh/uv#7808))
-   Fix encoding mismatch between python child process and uv ([#&#8203;7757](astral-sh/uv#7757))
-   Reject self-dependencies in `uv add` ([#&#8203;7766](astral-sh/uv#7766))
-   Respect `tool.uv.environments` for legacy virtual workspace roots ([#&#8203;7824](astral-sh/uv#7824))
-   Retain empty extras on workspace members ([#&#8203;7762](astral-sh/uv#7762))
-   Use file stem when parsing cached wheel names ([#&#8203;7773](astral-sh/uv#7773))

##### Rust API

-   Make `FlatDistributions` public ([#&#8203;7833](astral-sh/uv#7833))

##### Documentation

-   Fix table of contents sizing ([#&#8203;7751](astral-sh/uv#7751))
-   GitLab Integration documentation ([#&#8203;6857](astral-sh/uv#6857))
-   Update documentation to setup-uv@v3 ([#&#8203;7807](astral-sh/uv#7807))
-   Use `uv publish` instead of twine in docs ([#&#8203;7837](astral-sh/uv#7837))
-   Fix typo in `projects.md` ([#&#8203;7784](astral-sh/uv#7784))

### [`v0.4.17`](https:/astral-sh/uv/blob/HEAD/CHANGELOG.md#0417)

[Compare Source](astral-sh/uv@0.4.16...0.4.17)

##### Enhancements

-   Add `uv build --all` to build all packages in a workspace ([#&#8203;7724](astral-sh/uv#7724))
-   Add support for `uv init --script` ([#&#8203;7565](astral-sh/uv#7565))
-   Add support for upgrading build environment for installed tools (`uv tool upgrade --python`) ([#&#8203;7605](astral-sh/uv#7605))
-   Initialize a Git repository in `uv init` ([#&#8203;5476](astral-sh/uv#5476))
-   Respect `--quiet` flag in `uv build` ([#&#8203;7674](astral-sh/uv#7674))
-   Add context message before listing available tools in `uvx` ([#&#8203;7641](astral-sh/uv#7641))

##### Bug fixes

-   Don't create Python bytecode files during interpreter discovery ([#&#8203;7707](astral-sh/uv#7707))
-   Escape glob patterns in workspace member discovery ([#&#8203;7709](astral-sh/uv#7709))
-   Avoid prefetching source distributions with unbounded lower-bound ranges ([#&#8203;7683](astral-sh/uv#7683))

##### Documentation

-   Add `uv build` and `uv publish` to features overview ([#&#8203;7716](astral-sh/uv#7716))
-   Add documentation on cache versioning ([#&#8203;7693](astral-sh/uv#7693))
-   Spell out the names of the Docker images for easier copy-paste ([#&#8203;7706](astral-sh/uv#7706))
-   Document uv-with-Jupyter workflows ([#&#8203;7625](astral-sh/uv#7625))
-   Note that `uv lock --upgrade-package` retains locked versions ([#&#8203;7694](astral-sh/uv#7694))

### [`v0.4.16`](https:/astral-sh/uv/blob/HEAD/CHANGELOG.md#0416)

[Compare Source](astral-sh/uv@0.4.15...0.4.16)

##### Enhancements

-   Add `uv publish` ([#&#8203;7475](astral-sh/uv#7475))
-   Add a `--project` argument to run a command from a project directory ([#&#8203;7603](astral-sh/uv#7603))
-   Display Python implementation when creating environments ([#&#8203;7652](astral-sh/uv#7652))
-   Implement trusted publishing for `uv publish` ([#&#8203;7548](astral-sh/uv#7548))
-   Respect lockfile preferences for `--with` requirements ([#&#8203;7627](astral-sh/uv#7627))
-   Unhide the `--directory` option ([#&#8203;7653](astral-sh/uv#7653))
-   Allow requesting free-threaded Python interpreters ([#&#8203;7431](astral-sh/uv#7431))
-   Show a dedicated PubGrub hint for `--unsafe-best-match` ([#&#8203;7645](astral-sh/uv#7645))
-   Add resolver error checking for conflicting distributions ([#&#8203;7595](astral-sh/uv#7595))

##### Bug fixes

-   Avoid adding double-newlines for CRLF ([#&#8203;7640](astral-sh/uv#7640))
-   Avoid retaining forks when `requires-python` range changes ([#&#8203;7624](astral-sh/uv#7624))
-   Determine if pre-release Python downloads should be allowed using the version specifiers ([#&#8203;7638](astral-sh/uv#7638))
-   Fix `link-mode=clone` for directories on Linux ([#&#8203;7620](astral-sh/uv#7620))
-   Improve Python executable name discovery when using alternative implementations ([#&#8203;7649](astral-sh/uv#7649))
-   Require opt-in to use alternative Python implementations ([#&#8203;7650](astral-sh/uv#7650))
-   Use the first pre-release discovered when only pre-release Python versions are available ([#&#8203;7666](astral-sh/uv#7666))

##### Documentation

-   Document environment variable that disables printing of virtual environment name in prompt ([#&#8203;7648](astral-sh/uv#7648))
-   Remove double whitespaces from the code ([#&#8203;7623](astral-sh/uv#7623))
-   Use anchorlinks rather than permalinks ([#&#8203;7626](astral-sh/uv#7626))

##### Preview features

-   Add build backend scaffolding ([#&#8203;7662](astral-sh/uv#7662))

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever MR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this MR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this MR, check this box

---

This MR has been generated by [Renovate Bot](https:/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy40NDAuNyIsInVwZGF0ZWRJblZlciI6IjM3LjQ0MC43IiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJSZW5vdmF0ZSBCb3QiXX0=-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Consider adding documentation on using uv ... to use a .venv in Jupyter notebooks
4 participants