BUG: segfault on macOS (amr64)

[neutrinoceros]

Bug report

For visibility, I think this segfault seen in CI (example logs) is recurrent but not deterministic. It may be an upstream bug and I'm not even sure that it's platform-dependent or Python-version-dependent, so I'm reporting this occurrence and will start paying attention next time around so we can gather some more clues. If it turns out it's really just about Python 3.9 it may not be worth trying to fix it since it might soon be time for us to drop support for it completely.

[nastasha-w]

Yeah, I'm seeing a segfault in that log. I guess it could be some cython code issue, but if it's only showing up on this one mac/python combo, that does kinda smell like some other issue. I'm honestly getting pretty annoyed with code dev on my mac

[neutrinoceros]

I'm honestly getting pretty annoyed with code dev on my mac

Wait do you see this segfault locally too ?

[nastasha-w]

Not personally, but I haven't been able to run anything with openMP (at least through python) on my mac. That's the annoyance I was thinking of. I'm using python 3.11 on my laptop, so at least a 2021 14-inch M1 Mac + python3.11 with no attempt to use multi-threading doesn't seem to have this issue.

[nastasha-w]

I might be getting a similar issue in this test for PR #4939, which fails specifically on python3.12 with macOS.

full copy of the error report in case new tests invalidate the link:

Fatal Python error: Bus error

Thread 0x00000001705df000 (most recent call first):
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/threading.py", line 359 in wait
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/threading.py", line 655 in wait
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/tqdm/_monitor.py", line 60 in run
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/threading.py", line 1030 in _bootstrap

Current thread 0x00000002000e4c00 (most recent call first):
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/numpy/lib/_arraysetops_impl.py", line 356 in _unique1d
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/numpy/lib/_arraysetops_impl.py", line 289 in unique
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/unyt/array.py", line 2040 in __array_function__
  File "/Users/runner/work/yt/yt/yt/frontends/stream/tests/test_stream_stretched.py", line 35 in test_variable_dx
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/_pytest/python.py", line 159 in pytest_pyfunc_call
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/_pytest/python.py", line 1627 in runtest
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/_pytest/runner.py", line 174 in pytest_runtest_call
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/_pytest/runner.py", line 242 in <lambda>
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/_pytest/runner.py", line 341 in from_call
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/_pytest/runner.py", line 241 in call_and_report
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/_pytest/runner.py", line 132 in runtestprotocol
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/_pytest/runner.py", line 113 in pytest_runtest_protocol
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/_pytest/main.py", line 362 in pytest_runtestloop
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/_pytest/main.py", line 337 in _main
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/_pytest/main.py", line 283 in wrap_session
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/_pytest/main.py", line [330](https:/yt-project/yt/actions/runs/10066586909/job/27828578887?pr=4939#step:7:331) in pytest_cmdline_main
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/_pytest/config/__init__.py", line 175 in main
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/_pytest/config/__init__.py", line 201 in console_main
  File "/Users/runner/hostedtoolcache/Python/3.12.4/arm64/bin/pytest", line 8 in <module>

Extension modules: markupsafe._speedups, yaml._yaml, numpy._core._multiarray_umath, numpy._core._multiarray_tests, numpy.linalg._umath_linalg, cftime._cftime, netCDF4._netCDF4, PIL._imaging, kiwisolver._cext, yt.utilities.lib.allocation_container, yt.utilities.lib.bitarray, yt.geometry.grid_visitors, yt.geometry.oct_container, yt.geometry.oct_visitors, yt.utilities.lib.partitioned_grid, yt.utilities.lib._octree_raytracing, yt.utilities.lib.lenses, yt.utilities.lib.grid_traversal, yt.utilities.lib.image_samplers, yt.utilities.lib.fnv_hash, yt.geometry.selection_routines, yt.utilities.lib.misc_utilities, yt.utilities.lib.particle_mesh_operations, yt.utilities.lib.image_utilities, yt.utilities.lib.quad_tree, yt.geometry.grid_container, yt.utilities.lib.amr_kdtools, yt.utilities.lib.marching_cubes, yt.geometry.particle_deposit, yt.utilities.lib.interpolators, yt.utilities.lib.mesh_utilities, yt.utilities.lib.distance_queue, yt.geometry.particle_smooth, ewah_bool_utils.morton_utils, ewah_bool_utils.ewah_bool_wrap, yt.utilities.lib.geometry_utils, yt.geometry.particle_oct_container, yt.utilities.lib.autogenerated_element_samplers, yt.utilities.lib.element_mappings, yt.utilities.lib.bounded_priority_queue, yt.utilities.lib.cykdtree.kdtree, yt.utilities.lib.particle_kdtree_tools, yt.utilities.lib.pixelization_routines, yt.utilities.lib.cyoctree, yt.utilities.lib.primitives, yt.utilities.lib.bounding_volume_hierarchy, yt.utilities.lib.basic_octree, yt.utilities.lib.contour_finding, yt.utilities.lib.depth_first_octree, yt.utilities.lib.fortran_reader, yt.utilities.lib.line_integral_convolution, yt.utilities.lib.points_in_volume, yt.utilities.lib.write_array, yt.utilities.lib.mesh_triangulation, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, yt.utilities.cython_fortran_utils, yt.utilities.lib.cosmology_time, yt.frontends.ramses.io_utils, yt.utilities.lib.alt_ray_tracers, yt.utilities.lib.ragged_arrays, scipy._lib._ccallback_c, charset_normalizer.md, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg.cython_blas, scipy.linalg._matfuncs_expm, scipy.linalg._decomp_update, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.linalg._propack._spropack, scipy.sparse.linalg._propack._dpropack, scipy.sparse.linalg._propack._cpropack, scipy.sparse.linalg._propack._zpropack, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, scipy.spatial._ckdtree, scipy._lib.messagestream, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial._distance_wrap, scipy.spatial._hausdorff, scipy.special._ufuncs_cxx, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.special._ellip_harm_2, scipy.spatial.transform._rotation, scipy.ndimage._nd_image, _ni_label, scipy.ndimage._ni_label, pandas._libs.tslibs.ccalendar, pandas._libs.tslibs.np_datetime, pandas._libs.tslibs.dtypes, pandas._libs.tslibs.base, pandas._libs.tslibs.nattype, pandas._libs.tslibs.timezones, pandas._libs.tslibs.fields, pandas._libs.tslibs.timedeltas, pandas._libs.tslibs.tzconversion, pandas._libs.tslibs.timestamps, pandas._libs.properties, pandas._libs.tslibs.offsets, pandas._libs.tslibs.strptime, pandas._libs.tslibs.parsing, pandas._libs.tslibs.conversion, pandas._libs.tslibs.period, pandas._libs.tslibs.vectorized, pandas._libs.ops_dispatch, pandas._libs.missing, pandas._libs.hashtable, pandas._libs.algos, pandas._libs.interval, pandas._libs.lib, pandas._libs.ops, pandas._libs.hashing, pandas._libs.arrays, pandas._libs.tslib, pandas._libs.sparse, pandas._libs.internals, pandas._libs.indexing, pandas._libs.index, pandas._libs.writers, pandas._libs.join, pandas._libs.window.aggregations, pandas._libs.window.indexers, pandas._libs.reshape, pandas._libs.groupby, pandas._libs.json, pandas._libs.parsers, pandas._libs.testing, fontTools.misc.bezierTools, fontTools.varLib.iup, PIL._imagingmath, h5py._errors, h5py.defs, h5py._objects, h5py.h5, h5py.utils, h5py.h5t, h5py.h5s, h5py.h5ac, h5py.h5p, h5py.h5r, h5py._proxy, h5py._conv, h5py.h5z, h5py.h5a, h5py.h5d, h5py.h5ds, h5py.h5g, h5py.h5i, h5py.h5f, h5py.h5fd, h5py.h5pl, h5py.h5o, h5py.h5l, h5py._selector, requests.packages.charset_normalizer.md, requests.packages.chardet.md, scipy.interpolate._fitpack, scipy.interpolate._dfitpack, scipy.optimize._group_columns, scipy.optimize._trlib._trlib, scipy.optimize._lbfgsb, _moduleTNC, scipy.optimize._moduleTNC, scipy.optimize._cobyla, scipy.optimize._slsqp, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, scipy.optimize._highs.cython.src._highs_wrapper, scipy.optimize._highs._highs_wrapper, scipy.optimize._highs.cython.src._highs_constants, scipy.optimize._highs._highs_constants, scipy.linalg._interpolative, scipy.optimize._bglu_dense, scipy.optimize._lsap, scipy.optimize._direct, scipy.interpolate._bspl, scipy.interpolate._ppoly, scipy.interpolate.interpnd, scipy.interpolate._rbfinterp_pythran, scipy.interpolate._rgi_cython, astropy.stats._stats, erfa.ufunc, astropy.stats._fast_sigma_clip, astropy.time._parse_times, astropy.table._column_mixins, astropy.table._np_utils, astropy.io.ascii.cparser, astropy.utils.xml._iterparser, astropy.io.fits._utils, astropy.io.fits.hdu.compressed._compression, astropy.io.votable.tablewriter, astropy.wcs._wcs, shapely.lib, shapely._geos, shapely._geometry_helpers, psutil._psutil_osx, psutil._psutil_posix, scipy._lib._uarray._uarray, scipy.fftpack.convolve, fast_histogram._histogram_core, yt.frontends.artio._artio_caller, yt.frontends.gamer.cfields, miniball.bindings (total: 228)
/Users/runner/work/_temp/01d6d7c8-0c85-4930-8b0d-c69202ddbd97.sh: line 1:  5411 Bus error: 10           pytest --color=yes
/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
yt/frontends/stream/tests/test_stream_stretched.py::test_variable_dx 
Error: Process completed with exit code 138.

[nastasha-w]

Huh, I just got another error on a macOS-lastest, python 3.12 test, which occurred during the exact same test (yt/frontends/stream/tests/test_stream_stretched.py::test_variable_dx)! This time it's a segmentation fault instead of a bus error though.

[neutrinoceros]

It also looks like test_variable_dx is the culprit in
https:/yt-project/yt/actions/runs/10089899596/job/27898229486
(but I'm still suspecting it could really be triggered in a previous test)

Furthermore, this time it happened on Python 3.10 so it's not 3.9-specific.

[neutrinoceros]

Good news, I actually tried it and I can reproduce this locally with

pytest yt/frontends/stream/tests/test_stream_stretched.py

More notes:

• the non-deterministic nature of this crash is also seen locally: it doesn't fail everytime.
• I think what changed between the last successful run of cibuildwheel and the first failed one is that numpy 2.0.1 was released. I'll dig further this afternoon but this probably needs to be reported and addressed upstream.

[neutrinoceros]

marking this as a blocker since we basically cannot release until this is adressed

[neutrinoceros]

progress: I bisected it down to numpy/numpy#26821 (which is a backport of numpy/numpy#26797)

[neutrinoceros]

reported upstream as numpy/numpy#27037

[neutrinoceros]

And patched upstream numpy/numpy#27070
The patch was also backported so the next version of numpy (2.0.2 or 2.1.0) will be stable with regard to this test. '