[build][docs] Remove Python dependency for documentation generation. #718
Comments
|
There is a fair bit going on in the documentation generation, so here are my ideas on a piecewise transition from Python to CMake. I think the juice is worth the squeeze. I'd love to hear your thoughts. It would be great to find out if some of these ideas resonate with you. Finding Doxygen and requiring dotCMake provides a Doxygen find package module, which dates back to the minimum CMake version requirements for this project. The Doxygen find_package() module cleverly enforces the default need for
Python subprocess for doxygen commandWe can add additional commands to the custom target 'documentation' in An added benefit of this approach is that warnings and errors encountered by doxygen will be displayed as part of the build process. Currently, running the Python file copy operationsThere are a lot of file copy operations which occur in order to conform the source file layout to the expectation set for the INPUT tag in Doxyfile.in.
The module flattening and file copy operations in |
|
A first pass on the above ideas can be seen here: https:/phniix/USD/commit/75be658bb0746dc949d0c7d8c446d4d625e77bd3 In this first pass, the module flattening code remains in Python. I stop short of transposing it to CMake, as there may be alternatives to the GLOB style approach currently being employed. Alternatives could be a myriad of concepts. One alternative could be to specify explicitly the modules that are to be documented. This could, for example be through the declarations These specific declarations are the mechanism for the container which would be used to populate the 'INPUT' tag in And while yes, having these warnings now exposed may trigger a quick change to the relevant files, it might be nice to not have to worry about it ever again! Furthermore, such a container will know the exact files needed to be copied over for the Having just said that, the specific page ordering required by the docs does indeed make it a little bit more challenging to articulate the requirement through declarations. All that aside, I think it's a fun first pass, and I hope you do too! P.S. I have addressed these two errors through issue #719. Apply the changes as described in that issue, and the errors should resolve. |
|
Why can't the INPUT tag point directly at files in the git repository? This
would have the advantage that Doxygen errors would actually contain the
name of the file that needs to be fixed, as well as avoiding a lot of
copying. Failing that I would recommend symbolic links (they work on
Windows nowadays, right?)
…On Fri, Dec 7, 2018 at 10:40 PM pheonix ***@***.***> wrote:
A first pass on the above ideas can be seen here: ***@***.***
<phniix/USD@75be658>
In this first pass, the module flattening code remains in Python.
I stop short of transposing it to CMake, as there may be alternatives to
the GLOB style approach currently being employed. Alternatives could be a
myriad of concepts. One alternative could be to specify explicitly the
modules that are to be documented. This could, for example be through the
declarations pxr_library(), pxr_shared_library().
These specific declarations are the mechanism for the container which
would be used to populate the 'INPUT' tag in Doxyfile.in, thus automating
this process so the values always remain current. For example, with the
changes in this commit, I have the following report from a build:
λ cmake --build . --target documentation --config Release
... snip ...
CUSTOMBUILD : warning : tag INPUT: input source `src/modules/gal' does not exist [D:\gitrepos\USD\build\documentation.vcxproj]
CUSTOMBUILD : warning : tag INPUT: input source `src/modules/glfq' does not exist [D:\gitrepos\USD\build\documentation.vcxproj]
CUSTOMBUILD : warning : tag INPUT: input source `src/modules/gusd/overview.dox' does not exist [D:\gitrepos\USD\build\documentation.vcxproj]
CUSTOMBUILD : warning : source src/modules/gal is not a readable file or directory... skipping. [D:\gitrepos\USD\build\documentation.vcxproj]
CUSTOMBUILD : warning : source src/modules/glfq is not a readable file or directory... skipping. [D:\gitrepos\USD\build\documentation.vcxproj]
CUSTOMBUILD : warning : source src/modules/gusd/overview.dox is not a readable file or directory... skipping. [D:\gitrepos\USD\build\documentation.vcxproj]
... snip ...
And while yes, having these warnings now exposed may trigger a quick
change to the relevant files, it might be nice to not have to worry about
it ever again! Furthermore, such a container will know the exact files
needed to be copied over for the src/modules directory, thus reducing the
copy overhead and improving the documentation build performance.
Having just said that, the specific page ordering required by the docs
does indeed make it a little bit more challenging to articulate the
requirement through declarations.
All that aside, I think its a fun first pass, and I hope you do too!
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#718 (comment)>,
or mute the thread
<https:/notifications/unsubscribe-auth/ABLBEdKdX_dpKcNF3kotDTvRjFAhmzJLks5u216_gaJpZM4ZJZKf>
.
|
|
That's a really good question @spitzak. It was one of my original notions for files being used for documentation generation to be added through CMake's Delving into what exists currently in the CMakeLists.txt files, not every header file is explicitly named. A cursory assessment such as the following shows what I mean. (I'm doing this while sitting on SHA1 79f095a) pxr directory Header file count. third_party directory Header file count. Due to not every header being named, there would need to be some changes in the way that the CMake scripts are collating required files. Taking a look at the implementation of those As a result, it unfortunately strips away an easy-peasy approach that may go something like: With all that said, we do need to take a look at the current Doxyfile configuration. Currently, USD's doxygen configuration template file, Doxyfile.in, declares a specific ordering of files mixed with directory names. It uses Doxygen's recursive pattern match to drill down and collate relevant files for document generation. This specific ordering is vital in order to align the documentation with the Project's intended communication style and approach. This adds a complexity to the named file or directory via CMake's It is plausible for an adjustment to be made to the current It is a balance of sorts, and I see why the current approach has been taken. I too am left wondering if we can avoid major file copy operations altogether, and reference original file locations ? On the point of using symbolic links as a fallback option, whilst remembering that the goal is to remove the Python build time dependency for document generation, CMake currently only supports this operation on Unix systems, ie: Having said that, symbolic links will still eat up inodes, and the less inodes consumed by this process, the better. |
|
Filed as internal issue #USD-4978. |
|
Kindly upload Pixar USD maya Xcode project source? |
|
@usmanredsoft This being a CMake project, any platform specific project sources like those that would be for Xcode, would be generated by CMake. Are you able to elaborate a little bit more, maybe I am not understanding your question ? |
|
Just wanted to mention that I had an opportunity to revisit this as part of other work I'm doing and was able to modify the build to no longer rely on the generateDocs.py script. This should get pushed out soon! |
|
This is tremendously great news! We are all looking forward to this! Thank you :) (@sunyab) |
Fixes #718 (Internal change: 2120133)
Description of Issue
The documentation generation process has a dependency on Python. As part of the build process, it would be nice if we could have a cross-platform solution that doesn't depend on Python being available on a host build system.
Recently, a convo in issue #605 discussed the build time Python dependency for copying header files. The general feeling that we flowed towards was removing all Python build time dependencies, unless specifically asking to build with Python support. As a result, the Python dependency on document generation was brought up, something I thought would be worthwhile pursuing here in this separately tracked issue.
We would like to discuss the transition of the current docgen code from Python to CMake.
Current Python docgen code: https:/PixarAnimationStudios/USD/blob/master/cmake/macros/generateDocs.py
Current CMake docgen code utilising the above Python docgen code: https:/PixarAnimationStudios/USD/blob/6c50b29bb76f0a7c176bae05668ab195520a019b/CMakeLists.txt#L70
From our convo on this topic in issue #605 , a question we did raise was: is the juice worth the squeeze ?
Steps to Reproduce
Make sure that the minimum requirements for document generation have been met by the build system you are on.
See Optional Components - Documentation. These tools need to be on your PATH (*nix,darwin|win), or in your REGISTRY (win).
Generate build files from a build directory.
System Information (OS, Hardware)
Windows 10 Home - 10.0.17134 Build 17134
Visual Studio 2017 Community v15.8.8 - CL.exe (x64) 19.15.26732.1
USD dev branch. Commit SHA1: 79f095a
Package Versions
Doxygen 1.8.4, 1.8.6, 1.8.14 * (I did try this using v1.8.4 because that's the version reported in Doxyfile.in)
Graphviz 2.38.0
Python 3.7.0
CMake 3.11.4
Boost 1.68
TBB 2018 U5
Build Flags
Command Log Dump
Build file generation.
Document Generation
The text was updated successfully, but these errors were encountered: