Fix ox-ruby build, fix detection of ruby targets, make ruby more consistent with other languages. #12199
Conversation
|
/gcbrun trial_build.py ruby |
|
@AdvenamTacet @mschwager by the way is there a way we can bundle the ruby interpreter, with the fuzzers/scripts like we do with Python? |
|
This week I have no access to my PC, but if you could point me to what you exactly mean, I should have enough time for open source next week to try to implement it. I believe that Ruby and Python implementations are similar enough to assume that it's possible. |
Sorry for the delayed responses! I'm chiming in @jonathanmetzman 's stead here. What we mean is, for Python for example we bundle the interpreter via pyinstaller. see https://google.github.io/oss-fuzz/getting-started/new-project-guide/python-lang/#buildsh: Is there some equivalent for Ruby we can use here? |
|
Hey, thanks for the PR, overall this LGTM 👍
Hmm, searching around a bit, it looks like |
|
tail of https://oss-fuzz-build-logs.storage.googleapis.com/index.html#ox-ruby Outfile should rename to pass check. And maybe edit ruzzy_build or ox/build.sh can solve the error. such as : |
|
And in Step #3 - "compile-libfuzzer-address-x86_64" how to borrow ox env(gem install ox) to next step(runner)? should edit ox/Dockerfile to install ox again in runner env？ |
No. Installing ox in runner should not be necessary. The idea is to install ox (or any other project) in oss-fuzz/infra/base-images/base-builder-ruby/Dockerfile Lines 52 to 53 in e0d3b99 |
Unfortunately, this (making the builds self contained and portable in some way) is a pretty big requirement for us. Our actual execution environment (ClusterFuzz), uses a different image from OSS-Fuzz, and changes over time. We need the ability to execute older builds even if the runtime image is updated (for bisecting older builds). |
|
Hey all, I've started to work on this, but I'm having some issues running things locally. It appears to be issues compiling centipede locally: This issue (tensorflow/models#3647 (comment)) has lead me to believe I need to lower the job count and/or increase memory. I've given the Docker containers 72GB of memory, so I'm surprised it's that memory hungry. I've also tried passing Have you seen anything like this before, or are there any steps I'm missing for running locally? |
|
@mschwager you can see this error while building base-image locally. Instead, you can pull the Also, |
Hmm, I'm sure I can hack up files locally to make things build. But the problem I'm currently having is I guess I can add a line to copy it in in |
|
@mschwager you pull only Edit: and yes, you have to rebuild
I added it there, because all other languages were implemented that way already. Thankfully, |
|
I've made some progress, but I haven't gotten anything working end-to-end. The different architectures in play here are giving my system issues. I'm running this on a Mac with Apple silicon, so ARM64, and using Docker Desktop for Mac. I was initially using the TebakoI started with tebako. I got quite far with this tool, but I haven't been able to successfully build a packaged binary with all the necessary contents. Here was the final error: The ruby-packerThis ultimately resulted in an Additionally, traveling-rubyThis project also appears to be unmaintained, and hasn't had an update in 3 years. It states that arbitrary Ruby C extensions aren't supported. Since this is a requirement for Ruzzy, I don't think this project will work either. mruby-cliI didn't try this one, but it hasn't had an update in 8 years, so I doubt it will work. Conclusion
|
|
This was as far as I got with I think if we can ignore the following CMake compiler warnings then I think it's failing on the zstd compilation here: https:/tamatebako/tebako/blob/685a6ea6e3869fa2db4f121aaf7d4549eb084920/CMakeLists.txt#L309-L328 |
|
Okay, some updates here. I've gotten FROM --platform=linux/arm64 ubuntu:20.04
ENV DEBIAN_FRONTEND=noninteractive
RUN apt update
RUN apt install -y sudo git curl build-essential pkg-config bison flex autoconf \
binutils-dev libevent-dev acl-dev libfmt-dev libjemalloc-dev libiberty-dev \
libdouble-conversion-dev liblz4-dev liblzma-dev libssl-dev libunwind-dev \
libboost-filesystem-dev libboost-program-options-dev libboost-system-dev \
libboost-iostreams-dev libboost-date-time-dev libboost-context-dev \
libboost-regex-dev libboost-thread-dev libbrotli-dev libdwarf-dev libelf-dev \
libgoogle-glog-dev libffi-dev libgdbm-dev libyaml-dev libncurses-dev \
libreadline-dev libutfcpp-dev libncurses-dev libreadline-dev gcc-10 g++-10 \
clang-12 clang++-12 ruby-dev ruby-bundler libstdc++-10-dev
ARG CMAKE_VERSION=3.26.5
ENV CC=clang-12
ENV CXX=clang++-12
RUN mkdir -p cmake-install && \
curl -L -o cmake-install/cmake.sh \
https:/Kitware/CMake/releases/download/v"$CMAKE_VERSION"/cmake-"$CMAKE_VERSION"-linux-aarch64.sh && \
sh cmake-install/cmake.sh --skip-license --prefix="/usr/local"
RUN git clone --recurse-submodules https:/tamatebako/tebako.git
WORKDIR /tebako
RUN git checkout v0.8.7
RUN bundle install
RUN exe/tebako clean_ruby
RUN exe/tebako setup -R 3.2.5
RUN git clone https:/ohler55/ox
RUN mkdir -p ox/bin/
COPY fuzz_parse.rb ox/bin/
CMD ["/bin/bash"]
# Add the following to ox/ox.gemspec to allow the tebako entrypoint to work:
# s.files = Dir['{lib,ext}/**/*.{rb,h,c}'] + ['LICENSE', 'README.md', 'CHANGELOG.md'] + Dir["bin/**/*.rb"]
# s.bindir = 'bin'
# s.executables << "fuzz_parse.rb”
# s.add_runtime_dependency 'ruzzy'
# RUN MAKE="make --environment-overrides V=1" \
# LDSHARED="$CC -shared" \
# LDSHAREDXX="$CXX -shared" \
# CFLAGS="-fsanitize=fuzzer-no-link -fno-omit-frame-pointer -fno-common -fPIC -g" \
# CXXFLAGS="-fsanitize=fuzzer-no-link -fno-omit-frame-pointer -fno-common -fPIC -g" \
# exe/tebako press --root=ox --entry-point=fuzz_parse.rb --output=fuzz_parse -R 3.2.5NOTE: The extensive compilations happening here have quite a bit of non-determinism. I've broken necessary tools and re-fixed them by simply running the same command. Take this into consideration if you're going down this path. For example, the If you run the above Docker image you'll be dropped into a shell to play around with the resultant artifacts. After I'm not sure why it's seg faulting - there could be a million reasons. I suspect something isn't linking properly. Here's the backtrace: So something with So next steps would be debugging this segmentation fault and determining why fuzzing isn't fully executing as expected. |
No description provided.