Future of pyctdev? Considering alternatives? How about our workflow?

[maximlt]

pyctdev is an interesting approach to project management, providing a unified interface to usual tasks (test, build packages, build docs, ...). Another interesting feature it has is providing a unified interface to managing a project dependencies with conda or pip. I follow discuss.python.org and see that standardizing common project tasks is something that is in the air (a project like VSCode would greatly benefit from this being standardized for instance), however, that's pretty difficult to standardize globally and will certainly not happen any time soon, if ever.

pyctdev isn't perfect yet and it feels sometimes it's getting in our way. I'll make some comments on why I think that:

• it is yet another tool to maintain and that can break sometimes
• it is yet another tool, that happens not to be popular at all and as such is something potential contributors have to learn and it's not particularly well documented
• its tasks are extensible, which is nice, but that means it doesn't necessarily enforce all of them. For instance, across the HoloViz projects running linters isn't done using the same command (doit test_lint, doit test_flakes, ...).
• the commands users should mostly execute are the ones that run the tests (e.g. doit test_unit). However as a developer working on a bug fix or a new feature you would usually not run the whole test suite but instead run some specific files or even specific tests. So you need to find out which command is executed by pyctdev (that would be in tox.ini) and adapt it to your use case.
• the tests a project may want to run are defined in a tox.ini file. Tox is used internally by pyctdev but as a user you would usually not interact with it directly (as least I haven't done it), which I find pretty confusing.
• maybe the biggest difficulty we have around pyctdev and our worklow, is that we use conda to manage the project dependencies (at least that's the case on the CI). The developer workflow with conda isn't quite yet optimal: channel management is tricky, conflicts happen regularly in a changing ecosystem, environment solving can be very slow, and managing an environment (e.g. installing a new dependency) is pretty tricky as conda has to deal with a package installed in editable mode (maybe overriding it with a dev version it'll install from conda) and as a user you have to remember the channels that were used to create the environment
• an additional thing that makes it difficult to adopt even in the HoloViz group is that some of us (including me) don't have a single environment per project but have shared environments

Given all that, I think it's worth considering changing our workflow/pyctdev to improve the HoloViz developer experience, including for us and our contributors.

Some general questions we could ask ourselves would be:

• Even it's likely we'll always need our own custom things, should we still have pyctdev being our front-facing develop CLI?
• Should we still recommend/use conda as the package manager?

They are more and more projects in the Python environment that are dedicated to project management and running tasks. Here's a list of projects that might be interest to look at, as potential replacement or to draw some inspiration from:

• tox and tox-conda
• nox (has some conda support)
• hatch and hatch-conda
• pdm (has some conda support)
• poetry (doesn't seem to have any conda support)
• devpy/pydevtool: two projects related to the efforts of SciPy to develop a nice developer CLI (based on doit)
• conda-project
• ...

It may also be wise having a look at how projects which manage multiple sub-projects handle that:

• Jupyter
• ExecutableBooks
• ...

Finally I'd like to ask a few questions to the HoloViz developers, to see what's their experience with pyctdev and how they use it.

1. Do you use pyctdev? If not, what do you use instead?

2. Do you use conda to create environments? If not, what do you use?

3. Do you use conda or pip or both to install your Python project dependencies?

4. If you do use pyctdev, which commands do you use? How often do you use them or in which case?

Please @jbednar @philippjfr @jlstevens @hoxbro @ianthomas23 @droumis reply to these questions below.

[maximlt]

I'll start answering my own questions :D

Do you use pyctdev? If not, what do you use instead?

I do.

Do you use conda to create environments? If not, what do you use?

Yes mostly conda, venv occasionally

Do you use conda or pip or both to install your Python project dependencies?

conda as much as possible. When I'm afraid it's gonna take too long to install, or don't want to bother to fiddle with channels, I use pip.

If you do use pyctdev, which commands do you use? How often do you use them or in which case?

I currently have two dev environments, one dedicated to Param/Panel/Lumen and one to hvPlot/HoloViews/GeoViews. I created them using pyctdev. I don't run the tests with pyctdev, rather directly with pytest. If I had to create a new environment to build a site or a package locally, I would use pyctdev to create the environment, checking first the Github workflows files to see if they do specific things. I would build a package using pyctdev, I would build a site running nbsite directly. I wouldn't use pyctdev for anything else.

[droumis]

Thanks for bringing this up @maximlt !!

1. Do you use pyctdev? If not, what do you use instead?

Yes, for HoloViz projects I try to use pyctdev unless things get borked, in which case I just use whatever gets the job done

2. Do you use conda to create environments? If not, what do you use?

Yes, conda

3. Do you use conda or pip or both to install your Python project dependencies?

Mostly conda, sometimes pip if conda is being difficult, sometimes mamba

4. If you do use pyctdev, which commands do you use? How often do you use them or in which case?

Mostly just doit develop_install for installing an editable HoloViz package

[ianthomas23]

Do you use pyctdev? If not, what do you use instead?

Only occasionally, if I am trying to precisely reproduce failures that occur in github CI.

Do you use conda to create environments? If not, what do you use?

conda and virtualenv.

Do you use conda or pip or both to install your Python project dependencies?

Both. My default workflow is to use conda to create an environment with a particular python version, then use pip from that point onwards.

If you do use pyctdev, which commands do you use? How often do you use them or in which case?

When I very occasionally use it, I precisely follow the steps used in github CI.

No doubt my experience is colored by my tendency to work at the bottom of the stack (e.g. datashader) rather than the top (hvplot) where there are fewer dependencies and hence less need for pyctdev.

As someone who works on other open source projects outside of holoviz, I am required for those projects to be familiar with both pip and conda. The existence of pyctdev makes my life more difficult rather than easier as I need to both understand it and be able to drop back to the underlying pip/conda implementation when things go wrong. So for me it is an unnecessary complication. The dependency on tox means yet something else to learn. On fsspec we have just removed the historic use of tox entirely to make the developer and CI experiences simpler.

[jlstevens]

Do you use pyctdev? If not, what do you use instead?

Not directly as I tend to rely on CI for testing, doc and package building etc.

Do you use conda to create environments? If not, what do you use?

I use conda to create, manage and install environments. I use pip in precisely three instances:

1. No conda package available but it is on PyPI
2. I need to use dev version of a package that has no release (but is on github) in which case I use the pip+git protocol.
3. pip install -e . --no-deps for development.

Options 1. and 2. can also apply when using anaconda project (using a pip: section).

Do you use conda or pip or both to install your Python project dependencies?

Both as described above, but I would love to have a conda only workflow (only resorting to pip if the package is only on PyPI).

If you do use pyctdev, which commands do you use? How often do you use them or in which case?

See above!

[jbednar]

Do you use pyctdev? If not, what do you use instead?

Not directly, only via CI. If we had a way to run GH Actions locally without involving GH, I'd be as happy to use that instead.

Do you use conda to create environments? If not, what do you use?

All my environments are conda environments.

Do you use conda or pip or both to install your Python project dependencies?

I only use conda to install indirect dependencies, but I use pip install --no-deps -e to create an editable install of HoloViz or other projects that I edit. I normally have one master holoviz environment where dependencies are installed using conda, updated every so often, and all HoloViz projects besides Panel and GeoViews permanently accessed through an editable install. Whenever I want to test anything, I git pull on the projects of interest (and others if needed), then open a session in that main environment. The main environment is quite fat, with any random thing installed in it whenever it's convenient, except that I don't ever try to build GeoViews or Panel because of their requirement for custom Bokeh models that makes them much trickier to set up, and GeoViews also has complex dependencies that often cause installation to have issues. Those issues are a barrier to me contributing to those specific projects, but otherwise this setup works fine for me.

Separately, I use anaconda-project to maintain minimal environments for particular projects. Those environments are always created from scratch based on the .yml spec, and I make sure they have no unneeded dependencies and don't normally include any editable installs. I only expect those separate environments to be reproducible.

I don't currently have a good way to reproduce issues found by CI, which was a goal of pyctdev but largely has not worked out for me.

If you do use pyctdev, which commands do you use? How often do you use them or in which case?

I only use ones that I've cut and pasted from the CI setup, usually of a failing job, often without success because they depend on other setup I didn't copy (e.g. special environments).

[hoxbro]

Do you use pyctdev? If not, what do you use instead?

I don't use it except on the CI. It gives an extra level of abstraction, which is fine if what you want is precisely what it provides, but if not, it is hard to make minor tweaks to the command. Most of the commands I need are pretty simple.

I also think it is hard to find the "real" set of commands, like conda or pip. E.g. first, you have to look at the github.yaml to see the a combination of "real" commands and pyctdev commands, and then you have to look at pyctdev to find out what its command does behind the scene.

I don't know the best solution, but I'm leaning more toward a simpler approach, like scripts located in a folder that can both be called from github actions or locally (if the sets of commands have to follow a specific order).

Do you use conda to create environments? If not, what do you use?

I use mamba + conda-forge (no defaults).

Do you use conda or pip or both to install your Python project dependencies?

Same as @jlstevens.

If you do use pyctdev, which commands do you use? How often do you use them or in which case?

Same as @jbednar.

[sandhujasmine]

Thanks for the great summary @maximlt

I just setup holoviews dev env and encountered pyctdev in that process so noting my experience below.

it is yet another tool to maintain and that can break sometimes

It was also opaque to me since coming from conda I expected to see a more transparent environment file or list of packages. However, I can appreciate the value of maintaining dependencies in one place (ie setup.py).

use pyproject.toml for dev dependency management and when building the package

Keeps all dependencies in one place for dev env or building packages. However, I believe the tools that support this spec only work with pip (poetry is popular?)

the commands users should mostly execute are the ones that run the tests (e.g. doit test_unit). However as a developer working on a bug fix or a new feature you would usually not run the whole test suite but instead run some specific files or even specific tests. So you need to find out which command is executed by pyctdev (that would be in tox.ini) and adapt it to your use case.

Agreed but I would expect a learning curve on this for an alternative solution as well. You would still need to look up the pytest commands and adapt them to your case,

is that we use conda to manage the project dependencies (at least that's the case on the CI). The developer workflow with conda isn't quite yet optimal: channel management is tricky, conflicts happen regularly in a changing ecosystem, environment solving can be very slow, and managing an environment (e.g. installing a new dependency) is pretty tricky as conda has to deal with a package installed in editable mode (maybe overriding it with a dev version it'll install from conda) and as a user you have to remember the channels that were used to create the environment

Newbie here but sharing the questions / thoughts that came to my mind.
These are all conda issues so unsure how the future of pyctdev would solve these. pyctdev or pyct seems to be a tool for helping with CI tasks and avoiding repeatable code; but it also adds this extra layer of abstraction / complexity for developers. Though perhaps there are alternatives for helping with CI tasks (poetry with pyproject.toml perhaps?); however, those likely do not support conda so I understand why we have pyctdev - which seems like a conda centric solution - maybe there are other solutions for conda that work better (conda-lock?)
It seems we need to figure out how to make conda work better for holoviz projects. With the binary dependencies (gdal) and lots of conda users, I expect we would want a conda solution. Thank you for raising/pointing out these issues. It would also be good to list the requirements for CI work and for dev work to see what the intersection of the two is as we look at solutions.

an additional thing that makes it difficult to adopt even in the HoloViz group is that some of us (including me) don't have a single environment per project but have shared environments

I just started this process and only created a holoviews env. That seemed to have worked well but libmamba didn't work for some reason.

In the end, the following worked well for me. Since I was only making 1 change I skipped -o tests in favor of manually installing pytest.

conda create -n pyctdev -c pyviz "pyctdev>0.5.0"

# some duplication in pyproject.toml and setup.py but all dependencies
# are in setup.py in one place
doit env_create -c pyviz/label/dev -c conda-forge --name=hv_dev --python=3.9

vim ~/.condarc	# comment out libmamba solver

# this installs a wheel for holoviews package in editable mode
# install only pytest since we don't necessarily need all the testing tools
doit develop_install -c conda-forge -c bokeh -o recommended
conda install pytest

### pointing to local version
>>> import holoviews
>>> holoviews.__file__
'/Users/jasmine/projects/holoviews/holoviews/__init__.py'


# Don't forget to install the pre-commit package
conda install -c conda-forge pre-commit

Some easy things to perhaps help improve our docs around this (seems holoviews has open issue # 4135 for it):

1. all holoviz projects should point to the holoviz Developer Instructions. I didn't see this linked in the holoviews docs.
2. I only installed a dev env for holoviews and only installed pytest.
3. An overview of how / why we use pyctdev as opposed to conda env file would be helpful. Some motivation and documentation on how the env creation works and how it creates a standard conda env would be useful since this is a custom tool which abstracts the process of creating a standard conda env with which users are familiar.
4. Some guidance around what tests to add or which test suites to run would be useful. For instance, our CI tools will run the full test suite so it maybe sufficient for a user to run a subset of unit tests and let the CI flag errors. This would be more efficient for small changes.

[sandhujasmine]

Oops - I forgot to answer the questions:

Do you use pyctdev? If not, what do you use instead?

Yes - it took a little bit to understand how things work but worked well for setting up holoviews dev env.

Do you use conda to create environments? If not, what do you use?

Yes.

Do you use conda or pip or both to install your Python project dependencies?

Mostly conda if I can find a package for it.

If you do use pyctdev, which commands do you use? How often do you use them or in which case?

Only for installing holoviews dev env.