Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate SDK from BPF to SBF #28125

Merged
merged 1 commit into from
Oct 7, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

12 changes: 6 additions & 6 deletions ci/buildkite-pipeline-in-disk.sh
Original file line number Diff line number Diff line change
Expand Up @@ -163,13 +163,13 @@ all_test_steps() {
command_step stable ". ci/rust-version.sh; ci/docker-run.sh \$\$rust_stable_docker_image ci/test-stable.sh" 70
wait_step

# BPF test suite
# SBF test suite
if affects \
.rs$ \
Cargo.lock$ \
Cargo.toml$ \
^ci/rust-version.sh \
^ci/test-stable-bpf.sh \
^ci/test-stable-sbf.sh \
^ci/test-stable.sh \
^ci/test-local-cluster.sh \
^core/build.rs \
Expand All @@ -178,16 +178,16 @@ all_test_steps() {
^sdk/ \
; then
cat >> "$output_file" <<"EOF"
- command: "ci/test-stable-bpf.sh"
name: "stable-bpf"
- command: "ci/test-stable-sbf.sh"
name: "stable-sbf"
timeout_in_minutes: 35
artifact_paths: "bpf-dumps.tar.bz2"
artifact_paths: "sbf-dumps.tar.bz2"
agents:
queue: "gcp"
EOF
else
annotate --style info \
"Stable-BPF skipped as no relevant files were modified"
"Stable-SBF skipped as no relevant files were modified"
fi

# Perf test suite
Expand Down
12 changes: 6 additions & 6 deletions ci/buildkite-pipeline.sh
Original file line number Diff line number Diff line change
Expand Up @@ -163,13 +163,13 @@ all_test_steps() {
fi
wait_step

# BPF test suite
# SBF test suite
if affects \
.rs$ \
Cargo.lock$ \
Cargo.toml$ \
^ci/rust-version.sh \
^ci/test-stable-bpf.sh \
^ci/test-stable-sbf.sh \
^ci/test-stable.sh \
^ci/test-local-cluster.sh \
^core/build.rs \
Expand All @@ -178,16 +178,16 @@ all_test_steps() {
^sdk/ \
; then
cat >> "$output_file" <<"EOF"
- command: "ci/test-stable-bpf.sh"
name: "stable-bpf"
- command: "ci/test-stable-sbf.sh"
name: "stable-sbf"
timeout_in_minutes: 35
artifact_paths: "bpf-dumps.tar.bz2"
artifact_paths: "sbf-dumps.tar.bz2"
agents:
queue: "gcp"
EOF
else
annotate --style info \
"Stable-BPF skipped as no relevant files were modified"
"Stable-SBF skipped as no relevant files were modified"
fi

# Perf test suite
Expand Down
12 changes: 6 additions & 6 deletions ci/buildkite-solana-private.sh
Original file line number Diff line number Diff line change
Expand Up @@ -163,13 +163,13 @@ all_test_steps() {
fi
wait_step

# BPF test suite
# SBF test suite
if affects \
.rs$ \
Cargo.lock$ \
Cargo.toml$ \
^ci/rust-version.sh \
^ci/test-stable-bpf.sh \
^ci/test-stable-sbf.sh \
^ci/test-stable.sh \
^ci/test-local-cluster.sh \
^core/build.rs \
Expand All @@ -178,16 +178,16 @@ all_test_steps() {
^sdk/ \
; then
cat >> "$output_file" <<"EOF"
- command: "ci/test-stable-bpf.sh"
name: "stable-bpf"
- command: "ci/test-stable-sbf.sh"
name: "stable-sbf"
timeout_in_minutes: 35
artifact_paths: "bpf-dumps.tar.bz2"
artifact_paths: "sbf-dumps.tar.bz2"
agents:
queue: "sol-private"
EOF
else
annotate --style info \
"Stable-BPF skipped as no relevant files were modified"
"Stable-SBF skipped as no relevant files were modified"
fi

# Perf test suite
Expand Down
4 changes: 2 additions & 2 deletions ci/nits.sh
Original file line number Diff line number Diff line change
Expand Up @@ -25,9 +25,9 @@ declare print_free_tree=(
':metrics/src/**.rs'
':net-utils/src/**.rs'
':runtime/src/**.rs'
':sdk/bpf/rust/rust-utils/**.rs'
':sdk/sbf/rust/rust-utils/**.rs'
':sdk/**.rs'
':^sdk/cargo-build-bpf/**.rs'
':^sdk/cargo-build-sbf/**.rs'
':^sdk/program/src/program_option.rs'
':^sdk/program/src/program_stubs.rs'
':programs/**.rs'
Expand Down
6 changes: 3 additions & 3 deletions ci/publish-tarball.sh
Original file line number Diff line number Diff line change
Expand Up @@ -102,10 +102,10 @@ MAYBE_TARBALLS=
if [[ "$CI_OS_NAME" = linux ]]; then
(
set -x
sdk/bpf/scripts/package.sh
[[ -f bpf-sdk.tar.bz2 ]]
sdk/sbf/scripts/package.sh
[[ -f sbf-sdk.tar.bz2 ]]
)
MAYBE_TARBALLS="bpf-sdk.tar.bz2"
MAYBE_TARBALLS="sbf-sdk.tar.bz2"
fi

source ci/upload-ci-artifact.sh
Expand Down
2 changes: 1 addition & 1 deletion ci/run-local.sh
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ steps+=(shellcheck)
steps+=(test-checks)
steps+=(test-coverage)
steps+=(test-stable)
steps+=(test-stable-bpf)
steps+=(test-stable-sbf)
steps+=(test-stable-perf)
steps+=(test-downstream-builds)
steps+=(test-bench)
Expand Down
6 changes: 3 additions & 3 deletions ci/test-bench.sh
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,8 @@ _ "$cargo" build --manifest-path=keygen/Cargo.toml
export PATH="$PWD/target/debug":$PATH

# Clear the C dependency files, if dependency moves these files are not regenerated
test -d target/debug/bpf && find target/debug/bpf -name '*.d' -delete
test -d target/release/bpf && find target/release/bpf -name '*.d' -delete
test -d target/debug/sbf && find target/debug/sbf -name '*.d' -delete
test -d target/release/sbf && find target/release/sbf -name '*.d' -delete

# Ensure all dependencies are built
_ "$cargo" nightly build --release
Expand Down Expand Up @@ -57,7 +57,7 @@ _ "$cargo" nightly bench --manifest-path poh/Cargo.toml ${V:+--verbose} \
_ "$cargo" nightly bench --manifest-path core/Cargo.toml ${V:+--verbose} \
-- -Z unstable-options --format=json | tee -a "$BENCH_FILE"

# Run bpf benches
# Run sbf benches
_ "$cargo" nightly bench --manifest-path programs/bpf/Cargo.toml ${V:+--verbose} --features=bpf_c \
-- -Z unstable-options --format=json --nocapture | tee -a "$BENCH_FILE"

Expand Down
File renamed without changes.
48 changes: 24 additions & 24 deletions ci/test-stable.sh
Original file line number Diff line number Diff line change
Expand Up @@ -68,25 +68,25 @@ test-stable)
_ "$cargo" stable test --jobs "$JOBS" --all --tests --exclude solana-local-cluster ${V:+--verbose} -- --nocapture
fi
;;
test-stable-bpf)
test-stable-sbf)
# Clear the C dependency files, if dependency moves these files are not regenerated
test -d target/debug/bpf && find target/debug/bpf -name '*.d' -delete
test -d target/release/bpf && find target/release/bpf -name '*.d' -delete
test -d target/debug/sbf && find target/debug/sbf -name '*.d' -delete
test -d target/release/sbf && find target/release/sbf -name '*.d' -delete

# rustfilt required for dumping BPF assembly listings
# rustfilt required for dumping SBF assembly listings
"$cargo" install rustfilt

# solana-keygen required when building C programs
_ "$cargo" build --manifest-path=keygen/Cargo.toml

export PATH="$PWD/target/debug":$PATH
cargo_build_bpf="$(realpath ./cargo-build-bpf)"
cargo_test_bpf="$(realpath ./cargo-test-bpf)"
cargo_build_sbf="$(realpath ./cargo-build-sbf)"
cargo_test_sbf="$(realpath ./cargo-test-sbf)"

# BPF solana-sdk legacy compile test
"$cargo_build_bpf" --manifest-path sdk/Cargo.toml
# SBF solana-sdk legacy compile test
"$cargo_build_sbf" --manifest-path sdk/Cargo.toml

# BPF C program system tests
# SBF C program system tests
_ make -C programs/bpf/c tests
if need_to_generate_test_result; then
_ "$cargo" stable test \
Expand All @@ -99,16 +99,16 @@ test-stable-bpf)
--no-default-features --features=bpf_c,bpf_rust -- --nocapture
fi

# BPF Rust program unit tests
for bpf_test in programs/bpf/rust/*; do
if pushd "$bpf_test"; then
# SBF Rust program unit tests
for sbf_test in programs/bpf/rust/*; do
if pushd "$sbf_test"; then
"$cargo" test
"$cargo_build_bpf" --bpf-sdk ../../../../sdk/bpf --dump
"$cargo_test_bpf" --bpf-sdk ../../../../sdk/bpf
"$cargo_build_sbf" --sbf-sdk ../../../../sdk/sbf --dump
"$cargo_test_sbf" --sbf-sdk ../../../../sdk/sbf
popd
fi
done |& tee cargo.log
# Save the output of cargo building the bpf tests so we can analyze
# Save the output of cargo building the sbf tests so we can analyze
# the number of redundant rebuilds of dependency crates. The
# expected number of solana-program crate compilations is 4. There
# should be 3 builds of solana-program while 128bit crate is
Expand All @@ -125,27 +125,27 @@ test-stable-bpf)
exit 1
fi

# bpf-tools version
"$cargo_build_bpf" -V
# sbf-tools version
"$cargo_build_sbf" -V

# BPF program instruction count assertion
bpf_target_path=programs/bpf/target
# SBF program instruction count assertion
sbf_target_path=programs/bpf/target
if need_to_generate_test_result; then
_ "$cargo" stable test \
--manifest-path programs/bpf/Cargo.toml \
--no-default-features --features=bpf_c,bpf_rust assert_instruction_count \
-- -Z unstable-options --format json --report-time |& tee results.json
awk '!/{ "type": .* }/' results.json >"${bpf_target_path}"/deploy/instuction_counts.txt
awk '!/{ "type": .* }/' results.json >"${sbf_target_path}"/deploy/instuction_counts.txt
else
_ "$cargo" stable test \
--manifest-path programs/bpf/Cargo.toml \
--no-default-features --features=bpf_c,bpf_rust assert_instruction_count \
-- --nocapture &> "${bpf_target_path}"/deploy/instuction_counts.txt
-- --nocapture &> "${sbf_target_path}"/deploy/instuction_counts.txt
fi

bpf_dump_archive="bpf-dumps.tar.bz2"
rm -f "$bpf_dump_archive"
tar cjvf "$bpf_dump_archive" "${bpf_target_path}"/{deploy/*.txt,bpfel-unknown-unknown/release/*.so}
sbf_dump_archive="sbf-dumps.tar.bz2"
rm -f "$sbf_dump_archive"
tar cjvf "$sbf_dump_archive" "${sbf_target_path}"/{deploy/*.txt,sbf-solana-solana/release/*.so}
exit 0
;;
test-stable-perf)
Expand Down
34 changes: 17 additions & 17 deletions docs/src/developing/on-chain-programs/developing-c.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,10 +18,10 @@ The `makefile` should contain the following:

```bash
OUT_DIR := <path to place to resulting shared object>
include ~/.local/share/solana/install/active_release/bin/sdk/bpf/c/bpf.mk
include ~/.local/share/solana/install/active_release/bin/sdk/sbf/c/sbf.mk
```

The bpf-sdk may not be in the exact place specified above but if you setup your
The sbf-sdk may not be in the exact place specified above but if you setup your
environment per [How to Build](#how-to-build) then it should be.

Take a look at
Expand Down Expand Up @@ -56,14 +56,14 @@ information on how to write a test case.
## Program Entrypoint

Programs export a known entrypoint symbol which the Solana runtime looks up and
calls when invoking a program. Solana supports multiple [versions of the BPF
calls when invoking a program. Solana supports multiple [versions of the SBF
loader](overview.md#versions) and the entrypoints may vary between them.
Programs must be written for and deployed to the same loader. For more details
see the [overview](overview#loaders).

Currently there are two supported loaders [BPF
Currently there are two supported loaders [SBF
Loader](https:/solana-labs/solana/blob/7ddf10e602d2ed87a9e3737aa8c32f1db9f909d8/sdk/program/src/bpf_loader.rs#L17)
and [BPF loader
and [SBF loader
deprecated](https:/solana-labs/solana/blob/7ddf10e602d2ed87a9e3737aa8c32f1db9f909d8/sdk/program/src/bpf_loader_deprecated.rs#L14).

They both have the same raw entrypoint definition, the following is the raw
Expand All @@ -89,10 +89,10 @@ function](https:/solana-labs/example-helloworld/blob/bc0b25c0ccebeff
Each loader provides a helper function that deserializes the program's input
parameters into C types:

- [BPF Loader
deserialization](https:/solana-labs/solana/blob/d2ee9db2143859fa5dc26b15ee6da9c25cc0429c/sdk/bpf/c/inc/solana_sdk.h#L304)
- [BPF Loader deprecated
deserialization](https:/solana-labs/solana/blob/8415c22b593f164020adc7afe782e8041d756ddf/sdk/bpf/c/inc/deserialize_deprecated.h#L25)
- [SBF Loader
deserialization](https:/solana-labs/solana/blob/d2ee9db2143859fa5dc26b15ee6da9c25cc0429c/sdk/sbf/c/inc/solana_sdk.h#L304)
- [SBF Loader deprecated
deserialization](https:/solana-labs/solana/blob/8415c22b593f164020adc7afe782e8041d756ddf/sdk/sbf/c/inc/deserialize_deprecated.h#L25)

Some programs may want to perform deserialization themselves, and they can by
providing their own implementation of the [raw entrypoint](#program-entrypoint).
Expand All @@ -109,7 +109,7 @@ Details on how the loader serializes the program inputs can be found in the
## Data Types

The loader's deserialization helper function populates the
[SolParameters](https:/solana-labs/solana/blob/8415c22b593f164020adc7afe782e8041d756ddf/sdk/bpf/c/inc/solana_sdk.h#L276)
[SolParameters](https:/solana-labs/solana/blob/8415c22b593f164020adc7afe782e8041d756ddf/sdk/sbf/c/inc/solana_sdk.h#L276)
structure:

```c
Expand All @@ -128,7 +128,7 @@ typedef struct {

'ka' is an ordered array of the accounts referenced by the instruction and
represented as a
[SolAccountInfo](https:/solana-labs/solana/blob/8415c22b593f164020adc7afe782e8041d756ddf/sdk/bpf/c/inc/solana_sdk.h#L173)
[SolAccountInfo](https:/solana-labs/solana/blob/8415c22b593f164020adc7afe782e8041d756ddf/sdk/sbf/c/inc/solana_sdk.h#L173)
structures. An account's place in the array signifies its meaning, for example,
when transferring lamports an instruction may define the first account as the
source and the second as the destination.
Expand All @@ -153,7 +153,7 @@ processed.
## Heap

C programs can allocate memory via the system call
[`calloc`](https:/solana-labs/solana/blob/c3d2d2134c93001566e1e56f691582f379b5ae55/sdk/bpf/c/inc/solana_sdk.h#L245)
[`calloc`](https:/solana-labs/solana/blob/c3d2d2134c93001566e1e56f691582f379b5ae55/sdk/sbf/c/inc/solana_sdk.h#L245)
or implement their own heap on top of the 32KB heap region starting at virtual
address x300000000. The heap region is also used by `calloc` so if a program
implements their own heap it should not also call `calloc`.
Expand All @@ -163,16 +163,16 @@ implements their own heap it should not also call `calloc`.
The runtime provides two system calls that take data and log it to the program
logs.

- [`sol_log(const char*)`](https:/solana-labs/solana/blob/d2ee9db2143859fa5dc26b15ee6da9c25cc0429c/sdk/bpf/c/inc/solana_sdk.h#L128)
- [`sol_log_64(uint64_t, uint64_t, uint64_t, uint64_t, uint64_t)`](https:/solana-labs/solana/blob/d2ee9db2143859fa5dc26b15ee6da9c25cc0429c/sdk/bpf/c/inc/solana_sdk.h#L134)
- [`sol_log(const char*)`](https:/solana-labs/solana/blob/d2ee9db2143859fa5dc26b15ee6da9c25cc0429c/sdk/sbf/c/inc/solana_sdk.h#L128)
- [`sol_log_64(uint64_t, uint64_t, uint64_t, uint64_t, uint64_t)`](https:/solana-labs/solana/blob/d2ee9db2143859fa5dc26b15ee6da9c25cc0429c/sdk/sbf/c/inc/solana_sdk.h#L134)

The [debugging](debugging.md#logging) section has more information about working
with program logs.

## Compute Budget

Use the system call
[`sol_log_compute_units()`](https:/solana-labs/solana/blob/d3a3a7548c857f26ec2cb10e270da72d373020ec/sdk/bpf/c/inc/solana_sdk.h#L140)
[`sol_log_compute_units()`](https:/solana-labs/solana/blob/d3a3a7548c857f26ec2cb10e270da72d373020ec/sdk/sbf/c/inc/solana_sdk.h#L140)
to log a message containing the remaining number of compute units the program
may consume before execution is halted

Expand All @@ -181,10 +181,10 @@ for more information.

## ELF Dump

The BPF shared object internals can be dumped to a text file to gain more
The SBF shared object internals can be dumped to a text file to gain more
insight into a program's composition and what it may be doing at runtime. The
dump will contain both the ELF information as well as a list of all the symbols
and the instructions that implement them. Some of the BPF loader's error log
and the instructions that implement them. Some of the SBF loader's error log
messages will reference specific instruction numbers where the error occurred.
These references can be looked up in the ELF dump to identify the offending
instruction and its context.
Expand Down
4 changes: 2 additions & 2 deletions programs/bpf/c/makefile
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
BPF_SDK := ../../../sdk/bpf/c
include $(BPF_SDK)/bpf.mk
SBF_SDK := ../../../sdk/sbf/c
include $(SBF_SDK)/sbf.mk
4 changes: 2 additions & 2 deletions programs/bpf_loader/test_elfs/makefile
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
BPF_SDK := ../../../sdk/bpf/c
include $(BPF_SDK)/bpf.mk
SBF_SDK := ../../../sdk/sbf/c
include $(SBF_SDK)/sbf.mk
Loading