starting using gsc with python example fails

[polestar1988]

I have installed gramine and SGX driver and DCAP and PSW on my server which is sgx enables, Now I want to build a docker image using gsc, how could I start? I have created the docker image also cloned gsc. Is there something else that I have to do? How should I create manifest file? could some one explain it step by step?
Besides, I just started a test :

meson --version
1.3.1

docker pull python:bullseye
cd gsc
cp config.yaml.template config.yaml
openssl genrsa -3 -out enclave-key.pem 3072
./gsc build --insecure-args python:bullseye --build-arg http_proxy="${http_proxy}" --build-arg https_proxy="${https_proxy}" --build-arg no_proxy="${no_proxy}" test/generic.manifest
and I get :

Building unsigned graminized Docker image `gsc-python:bullseye-unsigned` from original application image `python:bullseye`...
Step 1/29 : FROM ubuntu:20.04 AS gramine

 ---> f78909c2b360
Step 2/29 : RUN env DEBIAN_FRONTEND=noninteractive apt-get update     && env DEBIAN_FRONTEND=noninteractive apt-get install -y         autoconf         bison         build-essential         coreutils         curl         gawk         git         libprotobuf-c-dev         linux-headers-generic         nasm         ninja-build         pkg-config         protobuf-c-compiler         protobuf-compiler         python3         python3-cryptography         python3-pip         python3-protobuf         wget     && /usr/bin/python3 -B -m pip install 'tomli>=1.1.0' 'tomli-w>=0.4.0' 'meson>=0.56,!=1.2.*'

 ---> Using cache
 ---> ab6706949da2
Step 3/29 : COPY intel-sgx-deb.key /

 ---> Using cache
 ---> e7b9e13f84fc
Step 4/29 : RUN echo 'deb [arch=amd64] https://download.01.org/intel-sgx/sgx_repo/ubuntu bionic main'     > /etc/apt/sources.list.d/intel-sgx.list     && apt-key add intel-sgx-deb.key

 ---> Using cache
 ---> e9b14528ca0f
Step 5/29 : RUN env DEBIAN_FRONTEND=noninteractive apt-get update     && env DEBIAN_FRONTEND=noninteractive apt-get install -y libsgx-dcap-quote-verify-dev

 ---> Using cache
 ---> 55b9f6c5c0e9
Step 6/29 : RUN git clone https:/gramineproject/gramine.git /gramine

 ---> Using cache
 ---> 0fa361838a35
Step 7/29 : RUN cd /gramine     && git fetch origin v1.5     && git checkout v1.5

 ---> Using cache
 ---> b81a98f61a80
Step 8/29 : RUN mkdir -p /gramine/driver/asm     && cd /gramine/driver/asm     && wget --timeout=10 -O sgx.h         https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/plain/arch/x86/include/uapi/asm/sgx.h?h=v5.11     && sha256sum sgx.h | grep -q a34a997ade42b61376b1c5d3d50f839fd28f2253fa047cb9c0e68a1b00477956

 ---> Using cache
 ---> 15d1fc0bad91
Step 9/29 : RUN cd /gramine     && meson setup build/ --prefix="/gramine/meson_build_output"        --buildtype=release        -Ddirect=enabled -Dsgx=enabled        -Ddcap=enabled                -Dsgx_driver=upstream -Dsgx_driver_include_path=/gramine/driver             && ninja -C build     && ninja -C build install

 ---> Running in 6ae361ea73ad
The Meson build system
Version: 1.3.1
Source dir: /gramine
Build dir: /gramine/build
Build type: native build
Project name: gramine
Project version: 1.5
C compiler for the host machine: cc (gcc 9.4.0 "cc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0")
C linker for the host machine: cc ld.bfd 2.34
C++ compiler for the host machine: c++ (gcc 9.4.0 "c++ (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0")
C++ linker for the host machine: c++ ld.bfd 2.34
Host machine cpu family: x86_64
Host machine cpu: x86_64
Program check-no-reloc.sh found: YES (/gramine/scripts/check-no-reloc.sh)
Program gen-pal-map.py found: YES (/gramine/scripts/gen-pal-map.py)
Program get-python-platlib.py found: YES (/gramine/scripts/get-python-platlib.py)
Program meson-clang-format.sh found: YES (/gramine/scripts/meson-clang-format.sh)
Fetching value of define "__GLIBC__" : 2 
Program objcopy found: YES (/usr/bin/objcopy)
Program python3 found: YES (/usr/bin/python3)
WARNING: You should add the boolean check kwarg to the run_command call.
         It currently defaults to false,
         but it will default to true in future releases of meson.
         See also: https:/mesonbuild/meson/issues/9300
Program nasm found: YES (/usr/bin/nasm)
Compiler for C supports arguments -Wtrampolines: YES 
Compiler for C supports arguments -Wnull-dereference: YES 
meson.build:74: WARNING: Consider using the built-in warning_level option instead of using "-Wall".
meson.build:74: WARNING: Consider using the built-in warning_level option instead of using "-Wextra".
Has header "/gramine/driver/asm/sgx.h" : YES 
Program nm found: YES (/usr/bin/nm)
Compiler for C supports arguments -fstack-protector-strong -mstack-protector-guard=tls -mstack-protector-guard-reg=%gs -mstack-protector-guard-offset=8: YES 
Downloading tomlc99-208203af46bdbdb29ba199660ed78d09c220b6c5 source from https:/cktan/tomlc99/archive/208203af46bdbdb29ba199660ed78d09c220b6c5.tar.gz
Downloading file of unknown size.

Executing subproject tomlc99-208203af46bdbdb29ba199660ed78d09c220b6c5 

tomlc99-208203af46bdbdb29ba199660ed78d09c220b6c5| Project name: tomlc99
tomlc99-208203af46bdbdb29ba199660ed78d09c220b6c5| Project version: undefined
tomlc99-208203af46bdbdb29ba199660ed78d09c220b6c5| C compiler for the host machine: cc (gcc 9.4.0 "cc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0")
tomlc99-208203af46bdbdb29ba199660ed78d09c220b6c5| C linker for the host machine: cc ld.bfd 2.34
tomlc99-208203af46bdbdb29ba199660ed78d09c220b6c5| Program patch found: YES (/usr/bin/patch)
tomlc99-208203af46bdbdb29ba199660ed78d09c220b6c5| Build targets in project: 2
tomlc99-208203af46bdbdb29ba199660ed78d09c220b6c5| Subproject tomlc99-208203af46bdbdb29ba199660ed78d09c220b6c5 finished.

Downloading uthash-2.1.0 source from https:/troydhanson/uthash/archive/v2.1.0.tar.gz
Downloading file of unknown size.

Executing subproject uthash-2.1.0 

uthash-2.1.0| Project name: uthash
uthash-2.1.0| Project version: undefined
uthash-2.1.0| C compiler for the host machine: cc (gcc 9.4.0 "cc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0")
uthash-2.1.0| C linker for the host machine: cc ld.bfd 2.34
uthash-2.1.0| Program patch found: YES (/usr/bin/patch)
uthash-2.1.0| Build targets in project: 3
uthash-2.1.0| Subproject uthash-2.1.0 finished.

Downloading mbedtls-mbedtls-3.4.0 source from https:/ARMmbed/mbedtls/archive/mbedtls-3.4.0.tar.gz
Downloading file of unknown size.

Executing subproject mbedtls-mbedtls-3.4.0 

mbedtls-mbedtls-3.4.0| Project name: mbedtls
mbedtls-mbedtls-3.4.0| Project version: 3.4.0
mbedtls-mbedtls-3.4.0| C compiler for the host machine: cc (gcc 9.4.0 "cc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0")
mbedtls-mbedtls-3.4.0| C linker for the host machine: cc ld.bfd 2.34
mbedtls-mbedtls-3.4.0| Fetching value of define "__GLIBC__" : 2 (cached)
mbedtls-mbedtls-3.4.0| Program compile-gramine.sh found: YES (/bin/sh /gramine/subprojects/mbedtls-mbedtls-3.4.0/compile-gramine.sh)
mbedtls-mbedtls-3.4.0| Found pkg-config: YES (/usr/bin/pkg-config) 0.29.1
mbedtls-mbedtls-3.4.0| Program compile-pal.sh found: YES (/gramine/subprojects/mbedtls-mbedtls-3.4.0/compile-pal.sh)
mbedtls-mbedtls-3.4.0| Program compile-curl.sh found: YES (/bin/sh /gramine/subprojects/mbedtls-mbedtls-3.4.0/compile-curl.sh)
mbedtls-mbedtls-3.4.0| Build targets in project: 6
mbedtls-mbedtls-3.4.0| Subproject mbedtls-mbedtls-3.4.0 finished.

Downloading curl-7.84.0 source from https:/curl/curl/releases/download/curl-7_84_0/curl-7.84.0.tar.gz

Executing subproject curl-7.84.0 

curl-7.84.0| Project name: curl
curl-7.84.0| Project version: 7.84.0
curl-7.84.0| C compiler for the host machine: cc (gcc 9.4.0 "cc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0")
curl-7.84.0| C linker for the host machine: cc ld.bfd 2.34
curl-7.84.0| Program compile.sh found: YES (/bin/sh /gramine/subprojects/curl-7.84.0/compile.sh)
curl-7.84.0| Build targets in project: 7
curl-7.84.0| Subproject curl-7.84.0 finished.

Downloading cJSON-1.7.12 source from https:/DaveGamble/cJSON/archive/v1.7.12.tar.gz

Executing subproject cJSON-1.7.12 

cJSON-1.7.12| Project name: cJSON
cJSON-1.7.12| Project version: undefined
cJSON-1.7.12| C compiler for the host machine: cc (gcc 9.4.0 "cc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0")
cJSON-1.7.12| C linker for the host machine: cc ld.bfd 2.34
cJSON-1.7.12| Build targets in project: 7
cJSON-1.7.12| Subproject cJSON-1.7.12 finished.

Program protoc-c found: YES (/usr/bin/protoc-c)
Run-time dependency threads found: YES
Run-time dependency libprotobuf-c found: YES 1.3.3
Library sgx_dcap_quoteverify found: YES
Configuring host_sgx_driver.h using configuration
Compiler for C supports arguments -Wno-gnu-alignof-expression: NO 
Configuring __init__.py using configuration
Configuring gramine-direct using configuration

tools/sgx/common/meson.build:1:11: ERROR: Cannot link_whole a custom or Rust target 'curl' into a static library 'sgx_util'. Instead, pass individual object files with the "objects:" keyword argument if possible. Meson had to promote link to link_whole because 'sgx_util' is installed but not 'curl', and thus has to include objects from 'curl' to be usable.

A full log can be found at /gramine/build/meson-logs/meson-log.txt
Failed to build unsigned graminized Docker image `gsc-python:bullseye-unsigned`.

[dimakuv]

You seem to hit this error: #1501

This means that you're not using the latest Gramine version (inside GSC). Can you show the contents of config.yaml in the GSC directory?

You should use Gramine's v1.6 version or master.

[polestar1988]

config.yml:

Specify the OS distro that is used to build Gramine, i.e., the distro from where the Gramine build
# gets all tools and dependencies from. This distro should match the distro underlying the
# application's Docker image; otherwise the results may be unpredictable.
#
# Currently supported distros are:
# - ubuntu:20.04, ubuntu:21.04
# - debian:10, debian:11
# - centos:8
Distro: "ubuntu:20.04"

# If the image has a specific registry, define it here.
# Empty by default; example value: "registry.access.redhat.com/ubi8".
Registry: ""

# If you're using your own fork and branch of Gramine, specify the GitHub link and the branch name
# below; typically, you want to keep the default values though.
#
# It is also possible to specify the prebuilt Gramine Docker image (that was built previously via
# the `gsc build-gramine` command). For this, remove Repository and Branch and instead write:
#   Image:      "<prebuilt Gramine Docker image>"
#
# GSC releases are guaranteed to work with corresponding Gramine releases (and GSC `master`
# branch is guaranteed to work with current Gramine `master` branch).
Gramine:
    Repository: "https:/gramineproject/gramine.git"
    Branch:     "v1.5"

# Specify the Intel SGX driver installed on your machine (more specifically, on the machine where
# the graminized Docker container will run); there are several variants of the SGX driver:
#
#   - upstream (in-kernel) driver: use empty values like below
#         Repository: ""
#         Branch:     ""
#
#   - DCAP out-of-tree driver: same as above, use empty values
#         Repository: ""
#         Branch:     ""
#
#   - legacy out-of-tree driver: use something like the below values, but adjust the branch name
#         Repository: "https:/01org/linux-sgx-driver.git"
#         Branch:     "sgx_driver_1.9"
#
SGXDriver:
    Repository: ""
    Branch:     ""

I used Gramine's master

[polestar1988]

@dimakuv I already have used --insecure-args when building gsc image , but I have to pass some arguments as you see in docker compose (password and token) to the application , when I use normal image in docker compose it is working but when I use gsc image I get error from my application regards to empty password. Do you know how could I solve it?

[dimakuv]

@polestar1988 I still don't understand what is your original Docker image (that you supply to gsc build).

It looks like your original Docker image contains the ENTRYPOINT which is the Docker Compose invocation? Are you trying to use Docker Compose inside the Gramine SGX enclave? Is this your purpose?

[polestar1988]

@dimakuv okay ,The entry Point in the Dockerfile is my application (connectorApp) ,
I moved on and want to use docker run instead of docker compose ,
this is the manifest:

sgx.enclave_size = "4G"
sgx.max_threads = 30
sgx.remote_attestation = "dcap"

sgx.trusted_files = [
  "file:/gramine/app_files/entrypoint.manifest",  # unused entry, only to test merging of manifests
  "file:ConnectorConfig.properties",
]
sgx.allowed_files = [
 "file:data/certs/connector-req.pem", 
 "file:data/certs/connector-key.pem",
 "file:data/certs/connector-cert.pem",
]

inst it that allowed_files are for both creating and reading from files , but when I execute command docker run when the application wants to create this files I get error:

docker run --name sgxconnector3 -it --device=/dev/sgx_enclave -v /var/run/aesmd/aesm.socket:/var/run/aesmd/aesm.socket  -p 50052:50052 -v /home/sgx_user/connector/ConnectorConfig.properties:/app/ConnectorConfig.properties gsc-project/connector -p test -t <token>
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
Gramine is starting. Parsing TOML manifest file, this may take some time...
-----------------------------------------------------------------------------------------------------------------------
Gramine detected the following insecure configurations:

  - loader.insecure__use_cmdline_argv = true   (forwarding command-line args from untrusted host to the app)
  - sgx.allowed_files = [ ... ]                (some files are passed through from untrusted host without verification)

Gramine will continue application execution, but this configuration must not be used in production!
-----------------------------------------------------------------------------------------------------------------------

Emulating a raw syscall instruction. This degrades performance, consider patching your application to use Gramine syscall API.
2024/01/24 13:26:19 ERROR Failed to create file for encrypted key. Error="open data/certs/connector-key.pem: permission denied"
2024/01/24 13:26:19 ERROR Failed to initialize certificate Manager. Config=ConnectorConfig.properties Error="Error encrypting private key: open data/certs/connector-key.pem: permission denied"

the data/certs/connector-key.pem should create by the application ,
in docker file path data/certs/ has already created :

...
WORKDIR /app

RUN mkdir -p data/certs
....

[dimakuv]

@polestar1988 I think the problem is that you have relative paths in sgx.allowed_files.

Given that you have WORKDIR /app, why not specify absolute paths? This way you won't need to care about the mismatch of the starting directory. Just try this:

sgx.allowed_files = [
 "file:/app/data/certs/connector-req.pem", 
 "file:/app/data/certs/connector-key.pem",
 "file:/app/data/certs/connector-cert.pem",
]

[polestar1988]

@dimakuv oh yes that's right.