MSVS and Fortran

[kgryte]

Problem

Trying to compile native add-ons which hook into Fortran libraries. Linux and MacOS systems compile without issue due to the availability of gcc/g++/gfortran. Issues arise, however, when attempting to compile on Windows.

In particular, Microsoft Visual Studio (MSVS) does not have readily available Fortran support. While possible to use ifort and other tooling, these integrations are predominantly proprietary and not freely available. MinGW and MSYS2 provide gfortran support. However, neither MinGW nor MSYS2 (via Cygwin) are supported by Node.js.

The stance of various maintainers (e.g., @bnoordhuis here and here and @TooTallNate here) has been that GYP and MSVS are the tools which should be used to build add-ons on Windows. GYP because of initial precedent and continued use and MSVS because this is what is used to compile Node.js itself (i.e., don't mix compilers).

The constraints, however, imposed by GYP and MSVS 1) prohibit usage of other toolchains and 2) cause issues when attempting to integrate compiled libraries which have extended requirements (e.g., access to a Fortran compiler). In my use case, I want the ability to compile high-performance numeric and scientific computing libraries (including custom Fortran libraries shipped with the add-on) written in Fortran. Yes, tools like CMake.js, nbind, and node-ffi exist; however, including these imposes an additional burden (both in terms of development cost and maintenance), especially as these require entirely separate configuration and procedures.

And yes, an alternative is to ship precompiled binaries for target systems, including Windows. However, I would either a) need access to a local Windows box (I don't) or b) need to use a CI service such as AppVeyor. In either case, as I discovered while trying to get add-ons to work on Windows, on Windows, not obvious where to access Node.js headers. Based on my assessment, node-gyp downloads the headers, stores in a .node-gyp directory, and then points to them accordingly. Thus, to pre-compile, I would need to do something similar, thus replicating node-gyp functionality on Windows. From my POV, to do this independently of node-gyp when node-gyp already does this (i.e., gets the necessary headers, etc) seems wasteful.

In a more perfect world, to extend the range of possible libraries and applications, would be beneficial to have more flexibility in terms of Windows toolchains. For my use cases (e.g., large scale and high performance data analysis), this means being able to leverage GNU compilers, as is possible in R, Python, and Julia.

I recognize that this issue is likely dead-on-arrival (similar to others), but I thought I would raise the issue again in hopes that people might recognize that using compilers other than those provided via MSVS is not just by choice, but also sometimes by necessity.

Related Issues

• ImportError: No module named gyp under MSYS2 #740
• would it be possible (or impossible) to use mingw-64 instead of visual studio on Windows ? #660
• Submitted request for MinGW GCC compiler support for Windows to Google GYP team #494
• gnu compiler #467
• npm-gyp on Windows should allow build tools to be configured to use gcc #402
• Win7 MingW #280
• Support for MinGW/Cygwin as an alternative to Visual Studio #194
• Windows users are not happy. #629

[bnoordhuis]

My second comment explains why we currently don't support anything besides Visual Studio (and now MS Build Tools): other compilers emit ABI-incompatible code.

clang++ might be an acceptable substitute someday (perhaps it already is) but mingw-g++ is never going to cut it - but it seems support for the latter is what you are asking for.

What exactly do you propose to do differently? Note that it should be something actionable, not just wishful (or wistful?) "wouldn't it be nice if?".

[kgryte]

@bnoordhuis I am aware of your second comment and was alluding to it when I mentioned "don't mix compilers". Just to clarify, the issue is not that non-VS (Visual Studio) code is ABI non-compliant, but that the compilers implement different ABIs (to be clear, the GNU C++ compiler adheres to an industry standard C++ ABI).

To summarize, because of this discrepancy, compiled output from one compiler is most likely not interoperable with output from another compiler (this includes the same compiler but different versions). Hence, if I compile a native add-on on Windows using the GNU C/C++/Fortran compiler toolchain, this will most likely be incompatible with Node.js, as Node is compiled via Microsoft Visual Studio (MSVS), whose compilers implement a different ABI.

The issue I raise is that, given the current constraints for compiling native add-ons on Windows, this limits the types of applications which are possible. Notably, because of MSVS' lack of a Fortran compiler, compiling Node.js native add-ons on Windows which have Fortran dependencies is, while possible (e.g., pay for a license for the Intel Fortran Compiler ifort or Lahey or perform dynamic library linking magic), not viable. In general, that MSVS is a consistent obstacle when compiling Fortran on Windows is well-known, as, for example, documented in the Python community (NumPy, NumPy, NumPy, SciPy, SciPy).

The predominant way numeric computing communities build libraries on Windows is to use either MinGW/MSYS or Cygwin (including MSYS2). For instance, NumPy encourages MinGW. SciPy encourages MinGW. Julia uses MinGW/MSYS2. OpenBLAS builds using MinGW, only providing MSVS support via C only libraries (which are significantly slower). ATLAS, due to its Linux focus, only works on Windows via Cygwin.

Early in its history, Node.js offered experimental support for compiling with MinGW and Cygwin. I am aware that Cygwin is problematic due to its terminal emulation (e.g., use of pipes). I am less aware as to why MinGW is problematic, other than that its toolchain is not an official Microsoft toolchain.

Regarding clang++, my understanding is that "maybe someday" captures the situation. Specifically, I am aware that Clang attempted to be MSVS ABI compatible, but ran into patent issues (?). To my knowledge, MSVS-Clang is still a work in progress.

In terms of actionable proposals, I am afraid I don't have enough knowledge of Node's motivations or needs or have enough knowledge/familiarity building C/C++ Windows applications to speak knowledgeably and/or authoritatively. If someone is willing to provide a detailed, step-by-step protocol and example showing how to compile Fortran and build a Node.js native add-on on Windows using MSVS without requiring proprietary Fortran compilers that would solve my immediate need.

However, this won't address the larger, more systemic issue of integrating with other libraries requiring other toolchains (e.g., MinGW or Clang). Possibly to clarify, may be useful to have a detailed explanation listing all the reasons why non-MSVS toolchains are explicitly not supported by Node.js on Windows systems. At the moment, I have found no such official record (Wiki or otherwise).

[bnoordhuis]

Possibly to clarify, may be useful to have a detailed explanation listing all the reasons why non-MSVS toolchains are explicitly not supported by Node.js on Windows systems.

There's not much to detail. MSVS and mingw-g++ are not ABI-compatible. Since node.exe is compiled with MSVS, you cannot load C++ add-ons compiled with mingw-g++; you get dynamic linker errors. (Name mangling isn't the only incompatibility, just the first one you'll run into.)

Code with C linkage might work but since node's and V8's APIs are C++, that ultimately doesn't help. Since there isn't anything actionable, I'll go ahead and close this out.

[kgryte]

@bnoordhuis Can you provide any insight into whether any consideration is being given to support non-MSVS toolchains in the future?

If Node.js is purely committed to only supporting MSVS, Windows users will likely never have low-level bindings to numeric and scientific computing libraries, thus hindering the growth of Node in this area and significantly impacting performance. I would not imagine the Node community would view this as a positive outcome.

[joshgav]

At the least, thanks for the feedback @kgryte, certainly something for us Microsofties to think about too.

/cc @nodejs/platform-windows @digitalinfinity @bowdenk7 @AndrewPardoe

[kgryte]

@joshgav Thanks for the note! Don't hesitate to reach out if you have questions or want additional feedback.

[AndrewPardoe]

Thank you for bringing this up, @kgryte. Fortran dependencies are difficult as we no longer produce Microsoft Fortran. The majority of customers who need Fortran are satisfied with the Intel Fortran compiler. There's just not the market for us to invest in.

If C/C++ numerical libraries are an option but not currently compiling we should fix that in MSVC.

[AndrewPardoe]

Also, please lean on your friendly neighborhood WG21 C++ Standards committee member to demand that they produce a standardized C++ ABI that works across all compilers.

[kgryte]

@AndrewPardoe Yeah, I can imagine that Intel Fortran is more viable in the enterprise space, where larger companies are willing to pay for licensing. Unfortunately, in the open source community, this is not the case.

Based on my assessment, R, Julia, and Python have moved to support initiatives like OpenBLAS and ATLAS, which explicitly build via the GNU compiler toolchain.

Regarding C/C++ numerical libraries, in a limited sense, they are an option; however, these typically do not yield the same performance benefits. OpenBLAS can compile via MSVC, but only using C kernels and with notable performance degradation. In general, while possible to hand port Fortran libraries (which are many) to C/C++, this is less than ideal due to development and maintenance costs.

And certainly, if I ever come across a WG21 C++ Standards committee member, I will be sure to lay down my demands! :)

Thanks for your input!