Skip to content

Latest commit

 

History

History
263 lines (250 loc) · 15.4 KB

Readme.md

File metadata and controls

263 lines (250 loc) · 15.4 KB
Raw

PosePair++

This repo contains all the code pertaining the PosePair++ demo at IPSN'19.

Settings on the watch

Different factors seem to affect watch performance, here's a list of settings that produce good results:

  • Connectivity > Bluetooth: Off
  • Developer options > Mobile battery saver: Off
  • Display > Always-on screen: Off
  • Gestures > Tilt-to-wake: Off
  • Gestures > Touch-to-wake: Off
  • Gestures > Wrist gestures: Off

Visualization

The visualization code is divided into two main components:

  • A Node.js+Express backend server (in folder visualization/server). This server is responsible for receiving and parsing gRPC packets containing raw sensor data (launches its own gRPC server). Note: for debugging purposes, we also include a mock gRPC client which sends random sensor data using the appropriate proto-contract data structures.
  • A React-based frontend client (in folder visualization/client). This client displays a web interface that receives chunks of sensor data from the backend server (using Server-Side Events) and renders them in real-time using a plotting library (plotly.js).

How to run the visualization

First of all, make sure you have all the dependencies installed, by running:

sh visualization/install_npm_deps.sh

(which internally calls npm install for you)

There are currently two options based on the source of the sensor data (commands assume you first cd visualization):

  • If you want to use our mock gRPC data, run npm run start-mock
  • Otherwise, assuming smartwatch(es) are already feeding in sensor data, simply run npm start (or npm run start)

Running OpenPose on GPU (Mac OS X instructions)

It is actually quite challenging to setup the environment for OpenPose to run on external GPUs on Mac, but we finally figured out how!

IMPORTANT NOTE: so far this only works on Mac OS X High Sierra or below (10.13.6-), since Nvidia hasn't released web drivers for Mojave yet.

Instructions are divided into four main steps:

Installing eGPU drivers and CUDA

  1. Install eGPU drivers by following these instructions:
    1. Disable SIP, or at least kext signing: reboot while holding down Cmd+R, terminal csrutil enable --without kext (or csrutil disable) and reboot normally
    2. Disconnect peripherals (especially eGPUs) and open the terminal (not iTerm!)
    3. Ensure you are running with sudo privileges: 
      sudo su
    4. Download the script: 
      curl -s https://raw.githubusercontent.com/learex/macOS-eGPU/master/macOS-eGPU.sh > macOS-eGPU.sh && chmod +x macOS-eGPU.sh
    5. Install the script and patch for Mac OS X 10.13.6: 
      ./macOS-eGPU.sh --install --iopcieTunneledPatch --nvidiaDriver --cudaDriver
    6. Reboot to complete installation
  2. Install CUDA 10.1 (instructions here):
    1. CUDA 10.1 requires Xcode 10.1 command line tools. If you have a different version of Xcode, follow these steps:
      1. Log in to https://developer.apple.com/downloads/
      2. Download Xcode CLT (Command Line Tools) 10.1 and install
      3. Select the Command Line Tools you just downloaded 
        sudo xcode-select --switch /Library/Developer/CommandLineTools
      4. Verify that clang has been downgraded via clang --version (should say Apple LLVM version 10.0.0 [...])
    2. Download CUDA 10.1 installer and install CUDA
    3. Update your PATH and DYLD_LIBRARY_PATH by adding these lines at the end of your ~/.bash_profile: 
      export PATH="/Developer/NVIDIA/CUDA-10.1/bin:$PATH"
export DYLD_LIBRARY_PATH="/Developer/NVIDIA/CUDA-10.1/lib:$DYLD_LIBRARY_PATH"
      and either opening a new tab/window or running source ~/.bash_profile.
    4. Verify the installation by:
      1. Running kextstat | grep -i cuda (and seeing one line output like [...] com.nvidia.CUDA (1.1.0) [...])
      2. Compiling and running one of the samples: 
        cp -R /usr/local/cuda/samples ~/Documents/Nvidia\ Cuda\ Examples
cd ~/Documents/Nvidia\ Cuda\ Examples
make -C 1_Utilities/deviceQuery
cd bin/x86_64/darwin/release
./deviceQuery
        should print some info about your GPU. (Note: at least in my case it will only show the eGPU if it was plugged when logged out [ > Log out or right after rebooting but before logging in]; if you plug it in and run the test without logging out first, it'll fail)
  3. Install cuDNN 7.5 [needs (free) registration on the Nvidia developer program]:
    1. (As of March 25th 2019 the latest version available for Mac for CUDA 10.1 is v7.5.0.56). Download and extract the tgz file
    2. Copy the files to /usr/local/cuda, maintaining the folder structure (that is, files inside the downloaded include folder go inside /usr/local/cuda/include and lib inside /usr/local/cuda/lib). [Original instructions]
    3. Verify cuDNN was successfully installed: 
      echo -e '#include"cudnn.h"\n int main(){return 0;}' | nvcc -x c - -o /dev/null -I/usr/local/cuda/include -L/usr/local/cuda/lib -lcudnn
      should not produce any error/output

Building Caffe (with GPU support) from source - Python 2

Follow these instructions to build Caffe for Python 2. If you want to use Python 3, skip to the next section.

  1. Clone Caffe's repo: 
    git clone https:/CMU-Perceptual-Computing-Lab/caffe && cd caffe
  2. Modify the file cmake/Dependencies.cmake:
    1. Comment out line 116 (the one that said if(NOT APPLE))
    2. Delete the letters else from line 134 (so now it should say if(APPLE))
  3. Create build dir: 
    mkdir build && cd build
  4. Create a virtual environment and activate it (we suggest using Miniconda): 
    conda create -y -n posepair python=2
conda activate posepair
  5. Install dependencies: 
    conda install opencv numpy mkl mkl-include boost protobuf glog gflags hdf5 lmdb leveldb snappy scikit-image
    • Currently, Anaconda's main channel doesn't provide the latest version of protobuf and opencv, which leads to problems down the road. Install/update them from conda-forge: 
      conda install -c conda-forge protobuf opencv=4
    • (if error: use of undeclared identifier 'CV_LOAD_IMAGE_COLOR' is thrown when making Caffe on step 7 below) -> Apply BVLC/caffe#6638 patch (offers OpenCV 4 compatibility)
  6. Configure project: 
    cmake -Dpython_version=2 -DBLAS=MKL -DCMAKE_PREFIX_PATH="/Library/Developer/CommandLineTools/usr/bin;${CONDA_PREFIX}" -DCMAKE_INSTALL_PREFIX=${CONDA_PREFIX} ..
  7. Make Caffe: 
    make -j8 install
  8. Avoid having to set PYTHONPATH every time by symlinking: 
    python -c "import site; import os; caffe_path='${CONDA_PREFIX}/python/caffe'; os.chdir(caffe_path); print(os.symlink('_caffe.dylib', '_caffe.so') is None if not os.path.exists('_caffe.so') else '_caffe.so already exists, nothing to do'); site_pkgs=site.getsitepackages()[0]; os.chdir(site_pkgs); print(os.symlink(os.path.relpath(caffe_path, site_pkgs), 'caffe') is None if not os.path.exists('caffe') else 'caffe already symlinked to site-packages, nothing to do'); print('Symlinks done :)');"
  9. Test Caffe: 
    python -c "import caffe; print('SUCCESS! Caffe version {} installed'.format(caffe.__version__))"

Building Caffe (with GPU support) from source - Python 3

Follow these instructions to build Caffe for Python 3. If you want to use Python 2, check out the previous section and skip this one.

For some reason I struggle to run python -c "import caffe" without getting a "beautiful" Segmentation fault on Python 3 + Miniconda. Not sure whether this is an issue with conda's version of some dependency, but here's the instructions I followed to compile the latest Caffe with GPU support on Mac (thanks to Homebrew).

  1. Clone Caffe's repo: 
    git clone https:/CMU-Perceptual-Computing-Lab/caffe && cd caffe
  2. Modify line 116 of file cmake/Dependencies.cmake from if(NOT APPLE) to if(TRUE)
  3. Install Homebrew: 
    /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
  4. Install dependencies: 
    brew install cmake numpy opencv openblas protobuf boost boost-python3 glog gflags hdf5 lmdb leveldb
    • Make sure python3 points to Homebrew's Python 3: 
      ls -l `which python3`
      should print something like /usr/local/bin/python3 -> ../Cellar/python/3.7.3/bin/python3.
    • If that's not the case: 
      export PATH="/usr/local/opt/python/libexec/bin:$PATH"
  5. Create a virtual environment (replace </path/to/new/virtualenv> by the actual path where you'd like to create the virtualenv, e.g. ~/virtualenvs/posepair) and activate it: 
    python3 -m venv </path/to/new/virtualenv>
source </path/to/new/virtualenv>/bin/activate
  6. Install python dependencies: 
    pip install numpy protobuf scikit-image opencv-python
  7. Create build dir: 
    mkdir build && cd build
  8. Configure project: 
    cmake -Dpython_version=3 -DBLAS=Open -DCMAKE_PREFIX_PATH="/usr/local/Cellar/openblas/0.3.5" -DCMAKE_INSTALL_PREFIX=${VIRTUAL_ENV} ..
    • If you get an error saying that The dependency target "pycaffe" of target "pytest" does not exist caused by -- Could NOT find Boost, modify the file cmake/Dependencies.cmake (again):
      • In lines 157 and 164 replace "python-py${boost_py_version}" by "python${boost_py_version}"
      • In lines 158 and 165 replace ${Boost_PYTHON-PY${boost_py_version}_FOUND} by ${Boost_PYTHON${boost_py_version}_FOUND}
  9. Make Caffe: 
    make -j8 install
    • (if error: use of undeclared identifier 'CV_LOAD_IMAGE_COLOR' is thrown) -> Apply BVLC/caffe#6638 patch (offers OpenCV 4 compatibility)
  10. Avoid having to set PYTHONPATH every time by symlinking: 
    python -c "import site; import os; caffe_path='${VIRTUAL_ENV}/python/caffe'; os.chdir(caffe_path); print(os.symlink('_caffe.dylib', '_caffe.so') is None if not os.path.exists('_caffe.so') else '_caffe.so already exists, nothing to do'); site_pkgs=site.getsitepackages()[0]; os.chdir(site_pkgs); print(os.symlink(os.path.relpath(caffe_path, site_pkgs), 'caffe') is None if not os.path.exists('caffe') else 'caffe already symlinked to site-packages, nothing to do'); print('Symlinks done :)');"
  11. Test Caffe: 
    python -c "import caffe; print('SUCCESS! Caffe version {} installed'.format(caffe.__version__))"

Building OpenPose with GPU support

  1. Clone OpenPose's repo: 
    git clone https:/CMU-Perceptual-Computing-Lab/openpose.git && cd openpose
  2. Comment out line 327 (the one that said op_detect_darwin_version(OSX_VERSION)) of file cmake/Cuda.cmake
  3. Create build dir: 
    mkdir build && cd build
  4. Activate your Python environment if you haven't done so, and define a custom env variable PYTHON_ROOT so the rest of the steps are the same for both Python versions:
    • Python 2 with conda: 
      conda activate posepair
export PYTHON_ROOT=$CONDA_PREFIX
    • Python 3 with venv: 
      source </path/to/new/virtualenv>/bin/activate
export PYTHON_ROOT=$VIRTUAL_ENV
  5. Configure project (if you don't have it installed, you might need to install Doxygen first): 
    cmake -DCaffe_INCLUDE_DIRS=${PYTHON_ROOT}/include -DCaffe_LIBS=${PYTHON_ROOT}/lib/libcaffe.dylib -DBUILD_CAFFE=OFF -DBUILD_PYTHON=ON -DBUILD_DOCS=ON -DCPU_ONLY=OFF -DGPU_MODE=CUDA -DCUDA_USE_STATIC_CUDA_RUNTIME=OFF -DDOWNLOAD_BODY_25_MODEL=ON -DDOWNLOAD_FACE_MODEL=ON -DDOWNLOAD_HAND_MODEL=ON -DWITH_OPENCV_WITH_OPENGL=ON -DCMAKE_PREFIX_PATH="/Library/Developer/CommandLineTools/usr/bin;${PYTHON_ROOT}" -DCMAKE_INSTALL_PREFIX=${PYTHON_ROOT} ..
  6. Make OpenPose: 
    make -j8 install
  7. Avoid having to set PYTHONPATH every time by symlinking (assumes you are still inside the build folder of openpose): 
    python -c "import site; import os; openpose_path='$PYTHON_ROOT/python/openpose'; models_path=os.path.join(openpose_path, 'models'); print(os.symlink(os.path.abspath('../models'), models_path) is None if not os.path.exists(models_path) else 'Models already symlinked, nothing to do'); site_pkgs=site.getsitepackages()[0]; os.chdir(site_pkgs); print(os.symlink(os.path.relpath(openpose_path, site_pkgs), 'openpose') is None if not os.path.exists('openpose') else 'openpose already symlinked to site-packages, nothing to do'); print('Symlinks done :)');"
  8. Test it: 
    pushd examples/tutorial_api_python && python openpose_python.py && popd

Building PyTorch from source

Finally, in order to use PyTorch with CUDA, it has to be compiled from source. Follow these instructions:

  1. Clone PyTorch: 
    git clone --recursive https:/pytorch/pytorch && cd pytorch
    • Just in case, today's latest commit, which works, is: git checkout 12abc8a99a5fc60603b3aecf5faa37600ad4fff6
  2. Install dependencies:
    • Python 2 with conda: 
      conda activate posepair
conda install -y numpy pyyaml setuptools cmake cffi mkl-include typing
conda install torchvision --no-deps -c pytorch
pip install future
    • Python 3 with venv: 
      source </path/to/new/virtualenv>/bin/activate
pip install pyyaml torchvision
  3. Compile and install PyTorch (go grab a coffe, it might take almost an hour...): 
    export CUDNN_INCLUDE_DIR=/usr/local/cuda/include
export CUDNN_LIB_DIR=/usr/local/cuda/lib
MACOSX_DEPLOYMENT_TARGET=10.13.6 CC=clang CXX=clang++ python setup.py install
  4. Test PyTorch: 
    pushd / && python -c "import torch; print('SUCCESS! PyTorch version {} installed'.format(torch.__version__)); t=torch.rand(3); r=t.cuda(); print('Here\'s a random 1x3 Tensor: {}'.format(r))" && popd