NPM package is too large for AWS Lambda

[claytondus]

Adding all three architectures' binary assets to a single package exceeds the 256 MB maximum for AWS Lambda. I realize I could create a Docker image but I would prefer to keep using Serverless. Is there a way to distribute a split package with optional dependencies restricted to the OS and architecture, like sharp?

[artemp]

@claytondus - I like prebuildify+node-gyp-build but agree the size of a package can be an issue.

possible approach (?)

"optionalDependencies" with "postinstall" fallback script to install arch/os specific mapnik binaries dependencies.

https://sentry.engineering/blog/publishing-binaries-on-npm
evanw/esbuild#1621

Actual node bindings shouldn't be an issue I reckon

2.3M  node_modules/@mapnik/mapnik/prebuilds//darwin-arm64/@mapnik+mapnik.node
2.4M  node_modules/@mapnik/mapnik/prebuilds//darwin-x64/@mapnik+mapnik.node
3.1M  node_modules/@mapnik/mapnik/prebuilds//linux-x64/@mapnik+mapnik.node

I'm trying to figure out if this is a viable solution long term any feedback appreciated.

@mapnik/node-mapnik

[claytondus]

@artemp Thanks, yes this approach is acceptable to me. The size of the mapnik libraries themselves is the critical factor.

[artemp]

@claytondus This is all very much [WIP] and needs more work.

Could you try and let me know if this solves package size issue on Lambda.

npm install @mapnik/[email protected]

On macOS arm64 I get -

npm ls --depth=1
node-mapnik-testing@ /private/tmp/node-mapnik-testing
└─┬ @mapnik/[email protected]
  ├── @mapnik/[email protected]
  ├── UNMET OPTIONAL DEPENDENCY @mapnik/[email protected]
  ├── UNMET OPTIONAL DEPENDENCY @mapnik/[email protected]
  ├── [email protected]
  ├── [email protected]
  └── [email protected]

On Linux x86_64

npm ls --depth=1
testing-node-mapnik@ /tmp/testing-node-mapnik
└─┬ @mapnik/[email protected]
  ├── UNMET OPTIONAL DEPENDENCY @mapnik/[email protected]
  ├── UNMET OPTIONAL DEPENDENCY @mapnik/[email protected]
  ├── @mapnik/[email protected]
  ├── [email protected]
  ├── [email protected]
  └── [email protected]

[claytondus]

@artemp I was able to duplicate your results, and yes the install size is below the 250MB threshold (if I delete the prebuilds in @mapnik/mapnik). However, when adding this package as an override, node-gyp-build is being called:

package.json

{
  "name": "mapnik-split-test",
  "version": "1.0.0",
  "main": "index.js",
  "author": "",
  "license": "ISC",
  "description": "",
  "dependencies": {
    "@mapbox/spritezero": "^8.0.3"
  },
  "overrides": {
    "@mapbox/spritezero": {
      "mapnik": "npm:@mapnik/[email protected]"
    }
  }
}

npm install (npm 10.8.2, node 20.17.0, darwin arm64)

11:08 $ npm i 
npm warn deprecated [email protected]: Modern JS already guarantees Array#sort() is a stable sort, so this library is deprecated. See the compatibility table on MDN: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/sort#browser_compatibility
npm error code 1
npm error path /Users/CZD05/src/rms-api/mapnik-split-test/node_modules/mapnik
npm error command failed
npm error command sh -c node-gyp-build
npm error gyp info it worked if it ends with ok
npm error gyp info using [email protected]
npm error gyp info using [email protected] | darwin | arm64
npm error gyp info find Python using Python version 3.12.6 found at "/opt/homebrew/opt/[email protected]/bin/python3.12"
npm error gyp info spawn /opt/homebrew/opt/[email protected]/bin/python3.12
npm error gyp info spawn args [
npm error gyp info spawn args '/Users/CZD05/.nvm/versions/node/v20.17.0/lib/node_modules/npm/node_modules/node-gyp/gyp/gyp_main.py',
npm error gyp info spawn args 'binding.gyp',
npm error gyp info spawn args '-f',
npm error gyp info spawn args 'make',
npm error gyp info spawn args '-I',
npm error gyp info spawn args '/Users/CZD05/src/rms-api/mapnik-split-test/node_modules/mapnik/build/config.gypi',
npm error gyp info spawn args '-I',
npm error gyp info spawn args '/Users/CZD05/src/rms-api/mapnik-split-test/node_modules/mapnik/common.gypi',
npm error gyp info spawn args '-I',
npm error gyp info spawn args '/Users/CZD05/.nvm/versions/node/v20.17.0/lib/node_modules/npm/node_modules/node-gyp/addon.gypi',
npm error gyp info spawn args '-I',
npm error gyp info spawn args '/Users/CZD05/Library/Caches/node-gyp/20.17.0/include/node/common.gypi',
npm error gyp info spawn args '-Dlibrary=shared_library',
npm error gyp info spawn args '-Dvisibility=default',
npm error gyp info spawn args '-Dnode_root_dir=/Users/CZD05/Library/Caches/node-gyp/20.17.0',
npm error gyp info spawn args '-Dnode_gyp_dir=/Users/CZD05/.nvm/versions/node/v20.17.0/lib/node_modules/npm/node_modules/node-gyp',
npm error gyp info spawn args '-Dnode_lib_file=/Users/CZD05/Library/Caches/node-gyp/20.17.0/<(target_arch)/node.lib',
npm error gyp info spawn args '-Dmodule_root_dir=/Users/CZD05/src/rms-api/mapnik-split-test/node_modules/mapnik',
npm error gyp info spawn args '-Dnode_engine=v8',
npm error gyp info spawn args '--depth=.',
npm error gyp info spawn args '--no-parallel',
npm error gyp info spawn args '--generator-output',
npm error gyp info spawn args 'build',
npm error gyp info spawn args '-Goutput_dir=.'
npm error gyp info spawn args ]
npm error /bin/sh: mapnik-config: command not found
npm error gyp: Call to 'mapnik-config --cflags' returned exit status 127 while in binding.gyp. while trying to load binding.gyp
npm error gyp ERR! configure error 
npm error gyp ERR! stack Error: `gyp` failed with exit code: 1
npm error gyp ERR! stack at ChildProcess.<anonymous> (/Users/CZD05/.nvm/versions/node/v20.17.0/lib/node_modules/npm/node_modules/node-gyp/lib/configure.js:297:18)
npm error gyp ERR! stack at ChildProcess.emit (node:events:519:28)
npm error gyp ERR! stack at ChildProcess._handle.onexit (node:internal/child_process:294:12)
npm error gyp ERR! System Darwin 23.6.0
npm error gyp ERR! command "/Users/CZD05/.nvm/versions/node/v20.17.0/bin/node" "/Users/CZD05/.nvm/versions/node/v20.17.0/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
npm error gyp ERR! cwd /Users/CZD05/src/rms-api/mapnik-split-test/node_modules/mapnik
npm error gyp ERR! node -v v20.17.0
npm error gyp ERR! node-gyp -v v10.1.0
npm error gyp ERR! not ok
npm error A complete log of this run can be found in: /Users/CZD05/.npm/_logs/2024-09-27T15_09_31_768Z-debug-0.log

[artemp]

@claytondus thanks for testing!
Could you run

npm install @mapnik/[email protected] --foreground-scripts --loglevel=verbose

and post output either here or as a gist if too large.

Also, could you try installing core package only with

npm install @mapnik/core-darwin-arm64

? I'm wondering if preinstall script is not running or failing for some reason, thanks.

[claytondus]

@artemp The commands you gave installed successfully. I was only able to replicate my issue when using overrides in the package.json I posted above.

When running npm install --foreground-scripts --loglevel=verbose with my package.json, it looks like the preinstall script is still looking for the prebuild:

13:34 $ npm i --foreground-scripts --loglevel=verbose 
npm verbose cli /Users/CZD05/.nvm/versions/node/v20.17.0/bin/node /Users/CZD05/.nvm/versions/node/v20.17.0/bin/npm
npm info using [email protected]
npm info using [email protected]
npm verbose title npm i
npm verbose argv "i" "--foreground-scripts" "--loglevel" "verbose"
npm verbose logfile logs-max:10 dir:/Users/CZD05/.npm/_logs/2024-09-27T17_35_33_099Z-
npm verbose logfile /Users/CZD05/.npm/_logs/2024-09-27T17_35_33_099Z-debug-0.log
npm http fetch GET 200 https://registry.npmjs.org/@mapbox%2fspritezero 718ms (cache revalidated)
npm http fetch GET 200 https://registry.npmjs.org/json-stable-stringify 93ms (cache revalidated)
npm http fetch GET 200 https://registry.npmjs.org/svgo 98ms (cache revalidated)
npm http fetch GET 200 https://npm.pkg.github.com/@mapnik%2fmapnik 176ms (cache updated)
npm http fetch GET 200 https://registry.npmjs.org/svg-boundings 205ms (cache revalidated)
npm http fetch GET 200 https://registry.npmjs.org/queue-async 359ms (cache revalidated)
npm http fetch GET 200 https://registry.npmjs.org/@mapbox%2fshelf-pack 640ms (cache revalidated)
npm http fetch GET 200 https://registry.npmjs.org/node-gyp-build 15ms (cache hit)
npm http fetch GET 200 https://registry.npmjs.org/node-addon-api 16ms (cache hit)
npm http fetch GET 200 https://registry.npmjs.org/prebuildify 17ms (cache hit)
npm http fetch GET 200 https://registry.npmjs.org/object-keys 45ms (cache revalidated)
npm http fetch GET 200 https://registry.npmjs.org/jsonify 47ms (cache revalidated)
npm http fetch GET 200 https://registry.npmjs.org/isarray 50ms (cache revalidated)
npm http fetch GET 200 https://registry.npmjs.org/csso 54ms (cache revalidated)
npm http fetch GET 200 https://registry.npmjs.org/call-bind 57ms (cache revalidated)
npm http fetch GET 200 https://registry.npmjs.org/css-tree 60ms (cache revalidated)
npm http fetch GET 200 https://registry.npmjs.org/css-select 42ms (cache revalidated)
npm http fetch GET 200 https://registry.npmjs.org/picocolors 44ms (cache revalidated)
npm http fetch GET 200 https://registry.npmjs.org/stable 81ms (cache revalidated)
npm http fetch GET 200 https://npm.pkg.github.com/@mapnik%2fcore-linux-x64 99ms (cache updated)
npm http fetch GET 200 https://registry.npmjs.org/commander 93ms (cache revalidated)
npm http fetch GET 200 https://npm.pkg.github.com/@mapnik%2fcore-darwin-x64 119ms (cache updated)
npm http fetch GET 200 https://npm.pkg.github.com/@mapnik%2fcore-darwin-arm64 120ms (cache updated)
npm http fetch GET 200 https://registry.npmjs.org/@trysound%2fsax 71ms (cache revalidated)
npm http fetch GET 200 https://registry.npmjs.org/set-function-length 29ms (cache revalidated)
npm http fetch GET 200 https://registry.npmjs.org/es-define-property 34ms (cache revalidated)
npm http fetch GET 200 https://registry.npmjs.org/get-intrinsic 37ms (cache revalidated)
npm http fetch GET 200 https://registry.npmjs.org/es-errors 42ms (cache revalidated)
npm http fetch GET 200 https://registry.npmjs.org/function-bind 70ms (cache revalidated)
npm http fetch GET 200 https://registry.npmjs.org/define-data-property 30ms (cache revalidated)
npm http fetch GET 200 https://registry.npmjs.org/has-proto 42ms (cache revalidated)
npm http fetch GET 200 https://registry.npmjs.org/gopd 43ms (cache revalidated)
npm http fetch GET 200 https://registry.npmjs.org/hasown 47ms (cache revalidated)
npm http fetch GET 200 https://registry.npmjs.org/has-property-descriptors 46ms (cache revalidated)
npm http fetch GET 200 https://registry.npmjs.org/has-symbols 55ms (cache revalidated)
npm http fetch GET 200 https://registry.npmjs.org/mkdirp-classic 6ms (cache hit)
npm http fetch GET 200 https://registry.npmjs.org/minimist 6ms (cache hit)
npm http fetch GET 200 https://registry.npmjs.org/node-abi 6ms (cache hit)
npm http fetch GET 200 https://registry.npmjs.org/npm-run-path 7ms (cache hit)
npm http fetch GET 200 https://registry.npmjs.org/pump 8ms (cache hit)
npm http fetch GET 200 https://registry.npmjs.org/tar-fs 8ms (cache hit)
npm http fetch GET 200 https://registry.npmjs.org/semver 6ms (cache hit)
npm http fetch GET 200 https://registry.npmjs.org/path-key 7ms (cache hit)
npm http fetch GET 200 https://registry.npmjs.org/end-of-stream 7ms (cache hit)
npm http fetch GET 200 https://registry.npmjs.org/once 8ms (cache hit)
npm http fetch GET 200 https://registry.npmjs.org/chownr 8ms (cache hit)
npm http fetch GET 200 https://registry.npmjs.org/tar-stream 8ms (cache hit)
npm http fetch GET 200 https://registry.npmjs.org/wrappy 2ms (cache hit)
npm http fetch GET 200 https://registry.npmjs.org/domhandler 37ms (cache revalidated)
npm http fetch GET 200 https://registry.npmjs.org/domutils 42ms (cache revalidated)
npm http fetch GET 200 https://registry.npmjs.org/mdn-data 43ms (cache revalidated)
npm http fetch GET 200 https://registry.npmjs.org/boolbase 48ms (cache revalidated)
npm http fetch GET 200 https://registry.npmjs.org/css-what 51ms (cache revalidated)
npm http fetch GET 200 https://registry.npmjs.org/source-map 52ms (cache revalidated)
npm http fetch GET 200 https://registry.npmjs.org/nth-check 72ms (cache revalidated)
npm http fetch GET 200 https://registry.npmjs.org/dom-serializer 29ms (cache revalidated)
npm http fetch GET 200 https://registry.npmjs.org/domelementtype 39ms (cache revalidated)
npm http fetch GET 200 https://registry.npmjs.org/domelementtype 71ms (cache revalidated)
npm http fetch GET 200 https://registry.npmjs.org/entities 35ms (cache revalidated)
npm http fetch GET 200 https://registry.npmjs.org/fs-constants 3ms (cache hit)
npm http fetch GET 200 https://registry.npmjs.org/bl 3ms (cache hit)
npm http fetch GET 200 https://registry.npmjs.org/readable-stream 5ms (cache hit)
npm http fetch GET 200 https://registry.npmjs.org/inherits 8ms (cache hit)
npm http fetch GET 200 https://registry.npmjs.org/string_decoder 4ms (cache hit)
npm http fetch GET 200 https://registry.npmjs.org/buffer 4ms (cache hit)
npm http fetch GET 200 https://registry.npmjs.org/util-deprecate 6ms (cache hit)
npm http fetch GET 200 https://registry.npmjs.org/base64-js 2ms (cache hit)
npm http fetch GET 200 https://registry.npmjs.org/ieee754 3ms (cache hit)
npm http fetch GET 200 https://registry.npmjs.org/safe-buffer 1ms (cache hit)
npm verbose reify failed optional dependency /Users/CZD05/src/rms-api/mapnik-split-test/node_modules/@mapnik/core-linux-x64
npm verbose reify failed optional dependency /Users/CZD05/src/rms-api/mapnik-split-test/node_modules/@mapnik/core-darwin-x64
npm http fetch POST 200 https://registry.npmjs.org/-/npm/v1/security/advisories/bulk 400ms
npm warn deprecated [email protected]: Modern JS already guarantees Array#sort() is a stable sort, so this library is deprecated. See the compatibility table on MDN: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/sort#browser_compatibility
npm info run @mapnik/[email protected] preinstall node_modules/mapnik ./scripts/preinstall.sh

> @mapnik/[email protected] preinstall
> ./scripts/preinstall.sh

trying -> /Users/CZD05/src/rms-api/mapnik-split-test/node_modules/mapnik/../core-darwin-arm64
MODULE_PATH:/Users/CZD05/src/rms-api/mapnik-split-test/node_modules/mapnik/prebuilds/darwin-arm64
MAPNIK_CORE_PATH:/Users/CZD05/src/rms-api/mapnik-split-test/node_modules/mapnik/../core-darwin-arm64
CUR_DIR:/Users/CZD05/src/rms-api/mapnik-split-test/node_modules/mapnik
npm info run @mapnik/[email protected] preinstall { code: 0, signal: null }
npm info run @mapnik/[email protected] install node_modules/mapnik node-gyp-build

> @mapnik/[email protected] install
> node-gyp-build

node:internal/modules/cjs/loader:1586
  return process.dlopen(module, path.toNamespacedPath(filename));
                 ^

Error: dlopen(/Users/CZD05/src/rms-api/mapnik-split-test/node_modules/mapnik/prebuilds/darwin-arm64/@mapnik+mapnik.node, 0x0001): Library not loaded: @loader_path/lib/libmapnik.dylib
  Referenced from: <89B654B0-6E4C-31F1-B64F-4B373A6147D5> /Users/CZD05/src/rms-api/mapnik-split-test/node_modules/mapnik/prebuilds/darwin-arm64/@mapnik+mapnik.node
  Reason: tried: '/Users/CZD05/src/rms-api/mapnik-split-test/node_modules/mapnik/prebuilds/darwin-arm64/lib/libmapnik.dylib' (no such file)
    at Module._extensions..node (node:internal/modules/cjs/loader:1586:18)
    at Module.load (node:internal/modules/cjs/loader:1288:32)
    at Module._load (node:internal/modules/cjs/loader:1104:12)
    at Module.require (node:internal/modules/cjs/loader:1311:19)
    at require (node:internal/modules/helpers:179:18)
    at load (/Users/CZD05/src/rms-api/mapnik-split-test/node_modules/node-gyp-build/node-gyp-build.js:22:10)
    at Object.<anonymous> (/Users/CZD05/src/rms-api/mapnik-split-test/node_modules/node-gyp-build/build-test.js:19:19)
    at Module._compile (node:internal/modules/cjs/loader:1469:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1548:10)
    at Module.load (node:internal/modules/cjs/loader:1288:32) {
  code: 'ERR_DLOPEN_FAILED'
}

Node.js v20.17.0

gyp info it worked if it ends with ok
gyp verb cli [
gyp verb cli '/Users/CZD05/.nvm/versions/node/v20.17.0/bin/node',
gyp verb cli '/Users/CZD05/.nvm/versions/node/v20.17.0/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js',
gyp verb cli 'rebuild'
gyp verb cli ]
... more node-gyp output....

[artemp]

@claytondus ok, thanks for feedback. I'll try to replicate this issue locally.

[artemp]

@claytondus - I see the issue -> "overrides" strips @scope (e.g @mapnik) which in turn breaks node_modules directory logic in preinstall script.

...
MODULE_PATH:/Users/CZD05/src/rms-api/mapnik-split-test/node_modules/mapnik/prebuilds/darwin-arm64
...

preinstall is expecting it to be

MODULE_PATH:/Users/CZD05/src/rms-api/mapnik-split-test/node_modules/@mapnik/mapnik/prebuilds/darwin-arm64

[artemp]

@mapnik/[email protected]

@claytondus should work, tested with -

{                                                                                                                                                             
  "name": "mapnik-split-test",                                                                                                                                
  "version": "1.0.0",                                                                                                                                         
  "main": "index.js",                                                                                                                                         
  "author": "",                                                                                                                                               
  "license": "ISC",                                                                                                                                           
  "description": "",                                                                                                                                          
  "dependencies": {                                                                                                                                           
    "@mapbox/spritezero": "^8.0.3"                                                                                                                            
  },                                                                                                                                                          
  "overrides": {                                                                                                                                              
    "@mapbox/spritezero": {                                                                                                                                   
      "mapnik": "npm:@mapnik/[email protected]"                                                                                                       
    }                                                                                                                                                         
  }                                                                                                                                                           
}         

[claytondus]

@artemp Thanks, 4.6.4-experimental installed cleanly. I'm having some trouble with esbuild and the new package, but that is probably related to @mapbox/spritezero. It is working in my app without a bundler, so I likely just need to tune esbuild to bundle mapnik correctly with the override.