Infrastructure to change many repositories at once #1
Comments
|
I have a script to verify that the lists of supported versions are consistent between setup.py classifiers, python_requires (if specified), tox.ini, appveyor.yml, and manylinux-install.sh. |
|
@mgedmin Cool, this tool is nice to make the supported versions even. Did you already run it against the zopefoundation repositories? What was the outcome? |
|
Here's a full run: https://gist.github.com/mgedmin/52f6a14da53ce66f58f6a1c08803eb0b
|
|
(Fix for zope.mimetype: zopefoundation/zope.mimetype#16) |
|
@mgedmin Thank you! This is an interesting list. Maybe we have to set some repositories into archive mode because no-one cares about them any more. |
|
I should mention that I've published my script to PyPI as check-python-versions. |
|
Maybe https:/asottile/all-repos could also help. |
|
That looks very promising. |
|
https:/jelmer/silver-platter could also be interesting |
|
You can now automate updating supported Python versions with It'll show you diffs and ask for confirmation before writing any changes to disk. You'll have to update the changelog and do a git commit/pull request dance yourself. There are more possibilities, e.g. would allow you to normalize very dusty packages that still support 2.6 and 3.3; |
|
As Python adopts a 12 month release cycle (see https://lwn.net/Articles/803679/), I see more pressure to automate adding new versions and removing old ones. |
|
Travis CI has a new feature that might allow us to reduce duplication in our .travis.yml files: https://blog.travis-ci.com/2019-11-11-build-config-imports |
That is very cool. It could work right away for projects without C extensions and manylinux builds. For manylinux builds, we have shell scripts in each repo that also need updating. Perhaps we could embed those in the YAML and echo them out to disk. |
|
See zopefoundation/zope.app.appsetup#4 where I tried the TravisCI import feature. |
|
It looks fine. Seems we need different configs for some packages e. g. the ones having tested documentation. So looking into each package is necessary to decide which config should be used. |
|
This sounds really great! While this eliminates the duplication between repositories, we still have the duplication between What I recently started doing for my own repositories... I just install and call Any opinions on this? |
|
That has been the normal procedure for most packages that see regular updates... |
|
I was originally very impressed with tox-travis, but received pushback on ZF repos due to custom jobs (docs, pure-python etc). I'm more familiar with Travis CI today and I don't think that's a problem (worst case, you can always specify a custom script in a matrix job, if a custom TOXENV is not sufficient). (Also, on my personal projects I prefer not to have coverage tracking in tox, for local development iteration speed, but I do want it on Travis. A quick grep shows this is not a concern for the majority of ZF repos.) |
Maybe, but this blueprint does not call tox. meta/travis-ci-config/minimal.yml Lines 1 to 20 in f4d90f4
When I develop, I run tox with one env, and only all incl. cov at the end. Additionally, I use pre-commit hooks. Well, it is all a matter of personal taste. |
|
I don't even know that "blueprint", I look at actual maintained projects. |
|
Maybe I get this wrong, but for me it looks like @icemac is about applying this blueprint for all ZopeFoundation projects? Starting with some old ones... |
|
I didn't watch closely enough, way too much on my plate right now. @icemac Will there be a sharable configuration using tox that "normal" Zope product packages re-use? |
|
@dataflake Yes there will be a first shareable version (hopefully by today). |
|
Some suggestions of @glyph in https://glyph.twistedmatrix.com/2020/02/modules-for-maintenance.html we could discuss to use in the ZopeFoundation projects, too. |
|
While @glyph's blog post is about tools, I also recommend @hynek's talk "Maintaining a Python Project When It’s Not Your Job" which is also about best practices to maintain floss projects. Video: Article: When I have a look at @glpyh's list, we already use a lot of those tools. As I both use bandit and pre-commit for my personal / work projects, and this combination does not seem to work well (when run via pre-commit, bandit does not pick up its configuration), I wanted to have a look how @glyph configures this in e.g. https:/twisted/twisted - turns out neither One major downside of Update There is another Python security checker, which claims to have all the checks bandit has plus some more: https:/tonybaloney/pycharm-security As the name suggests, it is a plugin for pycharm, but also can be run headless in ci / docker. |
|
There were some great links in those articles. I especially liked https://writing.jan.io/2017/03/06/sustainable-open-source-the-maintainers-perspective-or-how-i-learned-to-stop-caring-and-love-open-source.html. |
|
@jugmac00 FWIW, that list of tools was the full list of everything that I've put together one all the projects I've done build automation on, which includes dozens of projects, mostly open source, but definitely some commercial ones as well. Part of the point of the article was that there's no single repo where I've managed to set up absolutely everything correctly. If anything, Twisted lags a bit behind everything else, since it's dragging a longer tail of legacy & workflow around behind it. |
|
Today, I used all-repos twice on the Zope repositories... As suggested by @mgedmin in #37, here is again the link to my write up P.S.: As an addition, and as @mgedmin asked in one comment for the many PRs... Before there is the need to apply some mass updates again, it would make sense to discuss whether it is ok to push trivial changes directly to master, and if so, who got the rights to do this (maybe the core maintainers?). |
|
Branch protection rules exist on a number of repositories. Most of those only protect We have a ton of repositories and I personally am a bit scared of letting some script make all those changes and push it all to I'd be against removing those branch protection rules just to make such sweeping changes easier. Doing those changes across many repositories is or should be a rare exception. I don't mind the extra review/merge work. |
interim result of mostly @icemac's relentless effortsrepos with .meta.toml == updated with the config toolrepos without .meta.toml |
|
So not even half of the repositories are migrated to GHA … |
repos without
|
As the Python release manager plans to shorten the Python release cycle to one year (source) we should get some tools to add remove support for python versions from/to packages.
ghcloneall could be a starting point to get all (>300) repos of the zopefoundation.
The text was updated successfully, but these errors were encountered: