All protobuf dependent bottles are broken (gz-msgs*, gz-fuel-tools*, etc)

[scpeters]

Please supply your brew configuration:

Brew configuration

from https://build.osrfoundation.org/view/ign-garden/job/ignition_fuel-tools8-install_bottle-homebrew-amd64/222/

$ brew config
+ brew config
HOMEBREW_VERSION: 4.0.21
ORIGIN: https:/Homebrew/brew
HEAD: 9d0b7361d1a197949dbb47b761405075f83983b7
Last commit: 34 hours ago
Core tap origin: https:/Homebrew/homebrew-core
Core tap HEAD: 1f9b329aacf9df9df60f7bd87f8d169f7d30e2c1
Core tap last commit: 12 minutes ago
Core tap branch: master
Core tap JSON: 01 Jun 11:12 UTC
HOMEBREW_PREFIX: /usr/local
HOMEBREW_CASK_OPTS: []
HOMEBREW_GIT_EMAIL: [email protected]
HOMEBREW_GIT_NAME: OSRF Build Bot
HOMEBREW_MAKE_JOBS: 12
HOMEBREW_NO_AUTO_UPDATE: set
HOMEBREW_NO_INSTALL_FROM_API: set
HOMEBREW_UPDATE_TO_TAG: set
Homebrew Ruby: 2.6.10 => /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/bin/ruby
CPU: dodeca-core 64-bit kabylake
Clang: 14.0.0 build 1400
Git: 2.41.0 => /usr/local/bin/git
Curl: 7.87.0 => /usr/bin/curl
macOS: 12.6.6-x86_64
CLT: 14.2.0.0.1.1668646533
Xcode: N/A
+ brew doctor
Your system is ready to brew.

Brew linkage test result

$ + brew linkage --test ignition-fuel-tools8
Broken dependencies:
  /usr/local/opt/gz-msgs9/lib/libgz-msgs9.9.dylib (gz-msgs9)
  /usr/local/opt/protobuf/lib/libprotobuf.32.dylib (protobuf)

Checklist

• Identify change in homebrew-core that broke the bottle: Homebrew/homebrew-core@37b747c, part of grpc 1.55.0 protobuf 22.5 Homebrew/homebrew-core#131405
• Identify list of affected formulae
  • gz-msgs* and dependents
  • gz-transport* and dependents
  • gazebo11
• Remove broken bottles: for each affected formula, run brew bump-revision --remove-bottle-block
  • gz-msgs* and dependents Remove broken bottles, depend on protobuf@21 #2275
  • gz-transport* and dependents Remove broken bottles, depend on protobuf@21 #2275
  • gazebo11 Remove broken bottles, depend on protobuf@21 #2275
• Patches for broken formulae
  • Some protobuf patches for citadel/fortress #2279
  • gz-msgs9/transport12: patches for new protobuf #2281
  • gz-gui3, gz-gui6: add protobuf patches #2288
  • gz-sim3, gz-sim6: add patch to support new protobuf #2291
  • gazebo11: patch for new version of protobuf #2294
  • gz-gui7, gz-sensors7, gz-sim: patch for protobuf #2298
  • gz-msgs GzProtobuf: Do not require version 3 do support Protobuf 4.23.2 (23.2) gazebosim/gz-msgs#346
  • gz-transport Fix compatibility with protobuf 22 gazebosim/gz-transport#405
  • gz-gui Protobuf: Do not require version 3 do support Protobuf 4.23.2 (23.2) gazebosim/gz-gui#544
  • gz-sensors Fix finding GzProtobuf to support version 4.23.2 gazebosim/gz-sensors#351, Backport #351: Support protobuf >= 22 gazebosim/gz-sensors#353
  • gz-sim Protobuf: Do not require version 3 do support Protobuf 4.23.2 (23.2) gazebosim/gz-sim#2006
  • Support for protobuf >= 22 by using find_package(Protobuf CONFIG) gazebosim/gazebo-classic#3331
• Rebuild btoken bottles
  • Garden: gz-msgs9 and dependents
    • gz-msgs9, gz-transport12: rebuild bottles #2284
    • gz-fuel-tools8 apache-arrow and friends: use unversioned protobuf, grpc Homebrew/homebrew-core#133746
    • gz-gui7 gz-gui7, gz-sensors7: rebuild for protobuf #2299
    • gz-sensors7 gz-gui7, gz-sensors7: rebuild for protobuf #2299
    • gz-sim7
    • gz-launch6
    • gz-garden
  • Fortress: gz-msgs8 and dependents
    • Rebuild some fortress bottles #2283
    • gz-gui6 Rebuild rest of Fortress for protobuf #2292
    • gz-sim6 Rebuild rest of Fortress for protobuf #2292
    • gz-launch5 Rebuild rest of Fortress for protobuf #2292
    • gz-fortress Rebuild rest of Fortress for protobuf #2292
  • Citadel: gz-msgs5 and dependents
    • gz-msgs5: rebuild, depend on protobuf #2277
    • gz-transport8: depend on protobuf, rebuild bottle #2278
    • Rebuild some citadel bottles #2282
    • gz-gui3 Rebuild rest of Citadel for protobuf #2293
    • gz-sim3 Rebuild rest of Citadel for protobuf #2293
    • gz-launch2 Rebuild rest of Citadel for protobuf #2293
    • gz-citadel Rebuild rest of Citadel for protobuf #2293
  • gazebo11 gazebo11: revision bump for rebuild #2295

[scpeters]

This was precipitated by the merge of protobuf 22.5 in Homebrew/homebrew-core#131405, but a complicating factor is that protobuf-c doesn't yet support this new version (protobuf-c/protobuf-c#544), and gz-msgs* uses protobuf-c, so for consistency we should switch all of our formulae to protobuf@21 to match the change in protobuf-c from Homebrew/homebrew-core@37b747c. This will likely cause other issues because as a versioned formula, protobuf@21 is keg-only, which means that it does not symlink into the standard brew --prefix (/usr/local for intel machines) and will likely require extra environment variables to be set by our CI and users building from source (similar to this code).

[scpeters]

Disabling broken bottles in #2275

[scpeters]

ideally, we would only have protobuf@21 and not protobuf installed on a system, but protobuf is an indirect dependency of gz-common5 (via grpc and gdal), so this may get messy for packages like gz-gui, gz-sensors, and gz-sim that depend on both gz-msgs* and gz-common5

[traversaro]

gz-msgs* uses protobuf-c,

I probably be missing something really obvious, but by inspecting the CMake of gz-msgs I could not find any protobuf-c dependency, only the regular protobuf dependency. Where does the dependency on protobuf-c come from?

[scpeters]

gz-msgs* uses protobuf-c,

I probably be missing something really obvious, but by inspecting the CMake of gz-msgs I could not find any protobuf-c dependency, only the regular protobuf dependency. Where does the dependency on protobuf-c come from?

I think you're right; this dependency has been in use for a while but I'm not sure why

[scpeters]

gz-msgs* uses protobuf-c,

I probably be missing something really obvious, but by inspecting the CMake of gz-msgs I could not find any protobuf-c dependency, only the regular protobuf dependency. Where does the dependency on protobuf-c come from?

I think you're right; this dependency has been in use for a while but I'm not sure why

I thought the protoc compiler was only provided by protobuf-c, but that is not the case now at least; maybe it was always wrong. I'm glad you noticed this though, because this will let us stop using protobuf-c and switch back from protobuf@21 to protobuf!

[scpeters]

@traversaro thanks for being on top of this with gazebosim/gz-msgs#346 and other patches in that series!

[traversaro]

@traversaro thanks for being on top of this with gazebosim/gz-msgs#346 and other patches in that series!

FYI I reached some more problems in gz-sim conda-forge/gz-sim-feedstock#16 (comment), but I am not investigating. Note that those could be conda-forge-specific.

[scpeters]

I'm putting the garden rebuild on hold and am rebuilding the citadel bottles one by one starting with msgs5 in #2277

[scpeters]

I've rebuilt most of citadel and fortress bottles (except gz-gui6 and gz-sim+), but I'm having trouble with some of the garden bottles because gz-common5 has an indirect dependency on protobuf@21 via apache-arrow / gdal, and this is mixing up the bottle builds of gz-fuel-tools8 and others.

[scpeters]

I've rebuilt most of citadel and fortress bottles (except gz-gui6 and gz-sim+), but I'm having trouble with some of the garden bottles because gz-common5 has an indirect dependency on protobuf@21 via apache-arrow / gdal, and this is mixing up the bottle builds of gz-fuel-tools8 and others.

related apache-arrow issues: apache/arrow#35986, apache/arrow#35987

[carlocab]

I might be able to unblock you with Homebrew/homebrew-core#133623, since apache/arrow#35987 suggests that that's the fix for newer grpc/protobuf for apache-arrow.

[scpeters]

I might be able to unblock you with Homebrew/homebrew-core#133623, since apache/arrow#35987 suggests that that's the fix for newer grpc/protobuf for apache-arrow.

thanks @carlocab!

[scpeters]

I think our dependency issues with apache-arrow will resolve with Homebrew/homebrew-core#133496

[carlocab]

I think our dependency issues with apache-arrow will resolve with Homebrew/homebrew-core#133496

Unfortunately I think we may be blocked by protobuf-c, since some dependents depend on both apache-arrow and protobuf-c, so we can't mix their protobuf/grpc dependencies. Homebrew/homebrew-core#133746

[scpeters]

a new protobuf was merged, with what I assume is an ABI change because our bottles broke, so I removed them again in Homebrew/homebrew-core#133818

[traversaro]

a new protobuf was merged, with what I assume is an ABI change because our bottles broke, so I removed them again in Homebrew/homebrew-core#133818

Unfortunatly since protobuf 4.22 (or 22, depending to which version scheme you use) now patch (or minor, again depending if you refer to 4.23.3 or 23.3) version explicitly break the SOVERSION/ABI, see:

• Use release version instead of libtool version in Makefile protocolbuffers/protobuf#10355
• Release 21.3 broke ABI, ironically because of #10271 protocolbuffers/protobuf#10323
• Next round of abseil / grpc / protobuf upgrades conda-forge/conda-forge-pinning-feedstock#4075 (comment)

[scpeters]

I've patched and rebuilt citadel, fortress and gazebo11 bottles at this point, but garden bottles are still broken, due to gz-common5's indirect dependency on protobuf@21 (via apache-arrow and gdal). I've explored removing gdal as a dependency of gz-common5 (#2309), which would disable its geospatial component and found that this requires changes in gz-physics6, which I believe I have made in gazebosim/gz-physics#516, but I found that it would require changes in gz-rendering7, which may be more extensive.

I then refocused my efforts on patching protobuf-c to support the new protobuf and may have come up with something in protobuf-c/protobuf-c#556

[scpeters]

garden bottles have been rebuilt in #2313