Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: re-install lmdb when detecting the absence of @LMDB prebuilt packages #38691

Merged
merged 14 commits into from
Nov 21, 2023
Merged

fix: re-install lmdb when detecting the absence of @LMDB prebuilt packages #38691

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
67c71f3
fix: re-install lmdb when detecting the absence of @lmdb prebuilt pac…
JGAntunes Nov 3, 2023
9e50274
chore: set the correct root lmdb and add further comments
JGAntunes Nov 6, 2023
619abf1
chore: linter
JGAntunes Nov 6, 2023
c3cd62b
chore: linting
JGAntunes Nov 6, 2023
22be691
chore: make the install check for reliable and install @lmdb instead …
JGAntunes Nov 9, 2023
2555ba9
fix: no need to touch the patch, we can leverage the force binary flag
JGAntunes Nov 10, 2023
ae45348
chore: add integration test for lmdb fix
JGAntunes Nov 17, 2023
965db45
Merge branch 'master' into fix/install-missing-lmdb-prebuilt-packages
JGAntunes Nov 17, 2023
7c0e053
chore: add CCI config for new integration test
JGAntunes Nov 20, 2023
01bf938
add integration_tests_esm_in_gatsby_files to jobs
pieh Nov 20, 2023
915dbe0
fix: integration test for lmdb-regeneration
JGAntunes Nov 21, 2023
f2deaac
fix: cli path in test
JGAntunes Nov 21, 2023
9c23192
chore: actually add lmdb's dist directory
JGAntunes Nov 21, 2023
8d793e4
fix: back to the npmrc and just rm -rf @lmdb modules
JGAntunes Nov 21, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
71 changes: 71 additions & 0 deletions packages/gatsby/src/schema/graphql-engine/bundle-webpack.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,10 @@

import * as path from "path"
import * as fs from "fs-extra"
import execa, { Options as ExecaOptions } from "execa"
import webpack, { Module, NormalModule, Compilation } from "webpack"
import ConcatenatedModule from "webpack/lib/optimize/ConcatenatedModule"
import { dependencies } from "gatsby/package.json"
import { printQueryEnginePlugins } from "./print-plugins"
import mod from "module"
import { WebpackLoggingPlugin } from "../../utils/webpack/plugins/webpack-logging"
Expand Down Expand Up @@ -35,6 +37,71 @@ function getApisToRemoveForQueryEngine(): Array<GatsbyNodeAPI> {
return apisToRemove
}

const getInternalPackagesCacheDir = (): string =>
path.join(process.cwd(), `.cache/internal-packages`)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do wonder if there is something we could do to "sandbox" this to not have potential repo/site level .npmrc (or even user level .npmrc locally) potentially cause problems with installing optional packages. If there is .npmrc with omit=optional (or however the syntax it is) - that would use in this directory as well?

Alternatively, maybe instead of installing whole lmdb (and let that install optional packages with prebuilds) - maybe we try to install those prebuild package directly? Maybe then it would be less prone to inherit same problem that cause those not to be installed normally in the first place?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alternatively, maybe instead of installing whole lmdb (and let that install optional packages with prebuilds) - maybe we try to install those prebuild package directly?

I guess what I wonder is, in the presence of @lmdb/x package will that always be favoured even if there's a build from source in the lmdb directory? (I can dig through the code to try to understand that but tbh I worry that we might miss something by doing so)

Copy link
Contributor

@pieh pieh Nov 9, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess what I wonder is, in the presence of @lmdb/x package will that always be favoured even if there's a build from source in the lmdb directory? (I can dig through the code to try to understand that but tbh I worry that we might miss something by doing so)

It looks like general order of loading is to first try to load from build/Release first before trying optional packages. However this relies on fact that there is install script that before trying to compile from source check if there is binary that can be loaded - and when prebuilts are installed - it finds those and that check prevent from even compiling from source ( https://unpkg.com/browse/node-gyp-build-optional-packages/ is doing all those things ).

However - our lmdb-bundling-patch replace runtime usage of node-gyp-build(-optional-packages) from lmdb and replace that with direct path to .node file, so we are not subject to regular behavior when bundling for lambdas as we get rid of it here:

gatsby/packages/gatsby/src/schema/graphql-engine/lmdb-bundling-patch.ts

Lines 74 to 85 in 22c2412

.replace(
`require$1('node-gyp-build-optional-packages')(dirName)`,
`require(${JSON.stringify(lmdbBinaryLocation)})`
)
.replace(
`require$1('node-gyp-build')(dirName)`,
`require(${JSON.stringify(lmdbBinaryLocation)})`
)
.replace(
`loadNAPI__default["default"](dirName);`,
`require(${JSON.stringify(lmdbBinaryLocation)})`
)

So I think if we can install just one of prebuild packages like @lmdb/lmdb-linux-x64 and get path to .node file and make the lmdb bundling patch use that - I think this should be fine (?)

Also when we install whole lmdb in .cache/internal-packages, we don't actually use lmdb instance from there currently - we could add webpack alias to use that instance of lmdb package for lambda, but my biggest hang up with it is that installing main lmdb package it in nested directory could inherit any configuration that caused those prebuilt packages to not be installed in first place.

PS. after going through node-gyp-build-optional-packages I also found that:

build-from-source = true

in .npmrc would also likely cause the issue (similarly to omit = optional)


// Create a directory and JS module where we install internally used packages
const createInternalPackagesCacheDir = async (): Promise<void> => {
const cacheDir = getInternalPackagesCacheDir()
await fs.ensureDir(cacheDir)
await fs.emptyDir(cacheDir)

const packageJsonPath = path.join(cacheDir, `package.json`)

await fs.outputJson(packageJsonPath, {
name: `gatsby-internal-packages`,
description: `This directory contains internal packages installed by Gatsby used to comply with the current platform requirements`,
version: `1.0.0`,
private: true,
author: `Gatsby`,
license: `MIT`,
})
}

// Detect if the prebuilt binaries for lmdb have been installed. These are installed under @lmdb and are tied to each platform/arch. We've seen instances where regular installations lack these modules because of a broken lockfile or skipping optional dependencies installs
function lmdbPrebuiltPackagePresent(): boolean {
try {
require.resolve(`@lmdb/lmdb-${process.platform}-${process.arch}`)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In some cases this would not resolve even if it's actually installed when we try to require(.resolve) from gatsby package context.

Yarn PnP wouldn't allow that because those prebuilt packages are not dependencies of gatsby and are transitive dependency and you can't require/import transitive deps that you package doesn't declare as deps.

In node_modules cases it also is possible this won't work if package manager would decide to structure things like this:

node_modules
├── gatsby
└── lmdb
    └── @lmdb/lmdb-${process.platform}-${process.arch}

this would also mean it doesn't resolve.

For this check to work without relying on node_modules hierarchy we need to resolve prebuilt package from lmdb package context:

Suggested change
require.resolve(`@lmdb/lmdb-${process.platform}-${process.arch}`)
const lmdbRequire = mod.createRequire(
require.resolve(`lmdb`)
)
lmdbRequire.resolve(`@lmdb/lmdb-${process.platform}-${process.arch}`)

The other part is that I'm not sure about is if there are prebuilts for all the platform+arch combinations that exist. If prebuilt for the combo doesn't exist we will always try to install lmdb but that wouldn't help, so maybe before we even attempt to install - we should try to check if @lmdb/lmdb-${process.platform}-${process.arch} is included in list of available prebuilts (gathered from lmdb's optional dependencies list):

[
  '@lmdb/lmdb-darwin-arm64', 
  '@lmdb/lmdb-darwin-x64', 
  '@lmdb/lmdb-linux-arm', 
  '@lmdb/lmdb-linux-arm64', 
  '@lmdb/lmdb-linux-x64', 
  '@lmdb/lmdb-win32-x64'
]

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Greate suggestions, I'll add those in 👍

return true
} catch (e) {
return false
}
}

// Install lmdb under our internal cache if we detect the current installation
// isn't using the pre-build binaries
async function installIfMissingLmdb(): Promise<string | undefined> {
if (lmdbPrebuiltPackagePresent()) return undefined

await createInternalPackagesCacheDir()

const cacheDir = getInternalPackagesCacheDir()
const options: ExecaOptions = {
stderr: `inherit`,
cwd: cacheDir,
}

const npmAdditionalCliArgs = [
`--no-progress`,
`--no-audit`,
`--no-fund`,
`--loglevel`,
`error`,
`--color`,
`always`,
`--legacy-peer-deps`,
`--save-exact`,
]

await execa(
`npm`,
[`install`, ...npmAdditionalCliArgs, `lmdb@${dependencies.lmdb}`],
options
)

return path.join(cacheDir, `node_modules`, `lmdb`)
}

export async function createGraphqlEngineBundle(
rootDir: string,
reporter: Reporter,
Expand All @@ -57,6 +124,9 @@ export async function createGraphqlEngineBundle(
require.resolve(`gatsby-plugin-typescript`)
)

// Alternative lmdb path we've created to self heal from a "broken" lmdb installation
const alternativeLmdbPath = await installIfMissingLmdb()

const compiler = webpack({
name: `Query Engine`,
// mode: `production`,
Expand Down Expand Up @@ -121,6 +191,7 @@ export async function createGraphqlEngineBundle(
{
loader: require.resolve(`./lmdb-bundling-patch`),
options: {
alternativeLmdbPath,
forcedBinaryModule: store.getState().adapter.instance
? `@lmdb/lmdb-${process.platform}-${process.arch}/node.abi83.glibc.node`
: undefined,
Expand Down
20 changes: 16 additions & 4 deletions packages/gatsby/src/schema/graphql-engine/lmdb-bundling-patch.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,12 +26,24 @@ const { createRequire } = require(`module`)
export default function (this: any, source: string): string {
let lmdbBinaryLocation: string | undefined

const alternativeLmdbPath = this.getOptions()?.alternativeLmdbPath
try {
const lmdbRoot =
this?._module.resourceResolveData?.descriptionFileRoot ||
path.dirname(this.resourcePath).replace(`/dist`, ``)
let lmdbRoot: string
let requirePath: string

// If we've been provided an alternative lmdb module path we should use it as require path and the our lmdb root
if (alternativeLmdbPath) {
lmdbRoot = alternativeLmdbPath
requirePath = alternativeLmdbPath
} else {
lmdbRoot =
this?._module.resourceresolvedata?.descriptionfileroot ||
path.dirname(this.resourcepath).replace(`/dist`, ``)
requirePath = this.resourcePath
}

const lmdbRequire = createRequire(requirePath)

const lmdbRequire = createRequire(this.resourcePath)
const forcedBinaryModule = this.getOptions()?.forcedBinaryModule
let absoluteModulePath
if (forcedBinaryModule) {
Expand Down