Flowgraph constructions (#773)

* basic flowgraph implementation

* add visualization test

* refactor

* pre-branch

* new backend

* rebase master with pypi packaging

* preliminary run graph algo

* implementing graph running

* passing basic run test

* more cycle reduction tests

* better capture of errors

* add serialization test

* using property decorators

* add step control and custom driver

* small linting fix

* graph runner should not raise exception

* add mermaid support

* add test for mermaid

* implement and test auto-insertion of aux actions

* update poetry.lock

* mermaid generator automatically writes to file

* implement hook test and direct imports

* change runner to iteratively run nodes via BFS

* add mermaid support to docs

* simplify module imports

* rename test

* start documentation

* fix wording

* bump packages

* bump lock file, add typing marker

* fix type-checking for networkx

* fix CI pipeline

* migrate to current step control names

* reorder sections

* make docs consistent

* convert back to legacy names for plugging into backend

.github/workflows/pr.yml

@@ -60,4 +60,5 @@ jobs:
  id: type-checks
  run: |
  touch .venv/lib/python${{ matrix.python-version }}/site-packages/ruamel/py.typed
+ touch .venv/lib/python${{ matrix.python-version }}/site-packages/networkx/py.typed
  poetry run mypy --namespace-packages -p hammer

doc/Hammer-Use/Flowgraphs.rst

@@ -0,0 +1,157 @@
+.. _flowgraphs:
+
+Flowgraphs
+==========
+
+Hammer has **experimental** support for flowgraph constructions, similar to tools like `mflowgen <https:/mflowgen/mflowgen>`_.
+Their intention is to simplify the way flows are constructed and ran in Hammer.
+They can be imported via the ``hammer.flowgraph`` module.
+
+Construction
+------------
+
+Flowgraphs are nothing more than a collection of ``Node`` instances linked together via a ``Graph`` instance.
+Each ``Node`` "pulls" from a directory to feed in inputs and "pushes" output files to another directory to be used by other nodes.
+``Node`` instances are roughly equivalent to a single call to the ``hammer-vlsi`` CLI, so they take in similar attributes:
+
+* The action being called
+* The tool used to perform the action
+* The pull and push directories
+* Any *required* input/output files
+* Any *optional* input/output files
+* A driver to run the node with; this enables backwards compatibility with :ref:`hooks <hooks>`.
+* Options to specify steps within an action; this enables backwards compatibility with :ref:`flow control <flow-control>`.
+
+ * ``start_before_step``
+ * ``start_after_step``
+ * ``stop_before_step``
+ * ``stop_after_step``
+ * ``only_step``
+
+A minimal example of a ``Node``:
+
+.. code-block:: python
+ 
+ from hammer.flowgraph import Node
+
+ test = Node(
+ action="foo",
+ tool="nop",
+ pull_dir="foo_dir",
+ push_dir="/dev/null",
+ required_inputs=["foo.yml"],
+ required_outputs=["foo-out.json"],
+ )
+
+Each ``Node`` has the ability to be "privileged", meaning that a flow *must* start with this node.
+This only occurs when a flow is being controlled using any of the steps.
+
+Running a Flowgraph
+-------------------
+
+``Node`` instances are linked together using an `adjacency list <https://en.wikipedia.org/wiki/Adjacency_list>`_.
+This list can be used to instantiate a ``Graph``:
+
+.. code-block:: python
+
+ from hammer.flowgraph import Graph, Node
+
+ root = Node(
+ "foo", "nop", "foo_dir", "",
+ ["foo.yml"],
+ ["foo-out.json"],
+ )
+ child1 = Node(
+ "bar", "nop", "foo_dir", "bar_dir",
+ ["foo-out.json"],
+ ["bar-out.json"],
+ )
+ graph = Graph({root: [child1]})
+
+Using the Hammer CLI tool, separate actions are manually linked via an *auxiliary* action, such as ``syn-to-par``.
+By using a flowgraph, ``Graph`` instances by default *automatically* insert auxiliary actions.
+This means that actions no longer need to be specified in a flow; the necessary nodes are inserted by the flowgraph tool.
+This feature can be disabled by setting ``auto_auxiliary=False``.
+
+A ``Graph`` can be run by calling the ``run`` method and passing in a starting node.
+When running a flow, each ``Node`` keeps an internal status based on the status of the action being run:
+
+* ``NOT_RUN``: The action has yet to be run.
+* ``RUNNING``: The action is currently running.
+* ``COMPLETE``: The action has finished.
+* ``INCOMPLETE``: The action ran into an error while being run.
+* ``INVALID``: The action's outputs have been invalidated (e.g. inputs or attributes have changed).
+
+The interactions between the statuses are described in the diagram:
+
+.. mermaid::
+
+ stateDiagram-v2
+ [*] --> NOT_RUN
+ NOT_RUN --> RUNNING
+ RUNNING --> INCOMPLETE
+ RUNNING --> COMPLETE
+ INCOMPLETE --> NOT_RUN
+ COMPLETE --> INVALID
+ INVALID --> NOT_RUN
+
+Regardless of whether a flow completes with or without errors, the graph at the time of completion or error is returned, allowing for a graph to be "resumed" once any errors have been fixed.
+
+Visualization
+-------------
+
+A flowgraph can be visualized in Markdown files via the `Mermaid <https://mermaid.js.org/>`_ tool.
+Calling a ``Graph`` instance's ``to_mermaid`` method outputs a file named ``graph-viz.md``.
+The file can be viewed in a site like `Mermaid's live editor <https://mermaid.live/>`_ or using Github's native support.
+
+The flowgraph below would appear like this:
+
+.. code-block:: python
+
+ from hammer.flowgraph import Graph, Node
+
+ syn = Node(
+ "syn", "nop",
+ os.path.join(td, "syn_dir"), os.path.join(td, "s2p_dir"),
+ ["syn-in.yml"],
+ ["syn-out.json"],
+ )
+ s2p = Node(
+ "syn-to-par", "nop",
+ os.path.join(td, "s2p_dir"), os.path.join(td, "par_dir"),
+ ["syn-out.json"],
+ ["s2p-out.json"],
+ )
+ par = Node(
+ "par", "nop",
+ os.path.join(td, "par_dir"), os.path.join(td, "out_dir"),
+ ["s2p-out.json"],
+ ["par-out.json"],
+ )
+ g = Graph({
+ syn: [s2p],
+ s2p: [par],
+ par: []
+ })
+
+
+Here are the contents of ``graph-viz.md`` after calling ``g.to_mermaid()``:
+
+.. code-block:: markdown
+
+ ```mermaid
+ 
+ stateDiagram-v2
+ syn --> syn_to_par
+ syn_to_par --> par
+ ```
+
+Which would render like this:
+
+.. mermaid::
+
+ stateDiagram-v2
+ syn --> syn_to_par
+ syn_to_par --> par
+
+Note that the separators have been changed to comply with Mermaid syntax.

doc/Hammer-Use/index.rst

@@ -11,5 +11,6 @@ This documentation will walk through more advanced features of the Hammer infras
  Hammer-APIs
  Flow-Control
  Hooks
+ Flowgraphs
  Buildfile
  Hierarchical

doc/conf.py

@@ -17,12 +17,12 @@
 
 # -- Project information -----------------------------------------------------
 
-project = 'Hammer'
-copyright = '2023, Berkeley Architecture Research'
-author = 'Berkeley Architecture Research'
+project = "Hammer"
+copyright = "2023, Berkeley Architecture Research"
+author = "Berkeley Architecture Research"
 
 # The full version, including alpha/beta/rc tags
-release = '1.0.0'
+release = "1.0.0"
 
 
 # -- General configuration ---------------------------------------------------
@@ -33,30 +33,31 @@
 extensions = [
  "myst_parser",
  "sphinx-jsonschema",
- "sphinx_rtd_size"
+ "sphinx_rtd_size",
+ "sphinxcontrib.mermaid",
 ]
 
 # Add any paths that contain templates here, relative to this directory.
-templates_path = ['_templates']
+templates_path = ["_templates"]
 
 # List of patterns, relative to source directory, that match files and
 # directories to ignore when looking for source files.
 # This pattern also affects html_static_path and html_extra_path.
-exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
+exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"]
 
-source_suffix = '.rst'
+source_suffix = ".rst"
 
-master_doc = 'index'
+master_doc = "index"
 
 # -- Options for HTML output -------------------------------------------------
 
 # The theme to use for HTML and HTML Help pages. See the documentation for
 # a list of builtin themes.
 #
-html_theme = 'sphinx_rtd_theme'
+html_theme = "sphinx_rtd_theme"
 
 html_theme_options = {
- 'collapse_navigation': False,
+ "collapse_navigation": False,
 }
 
 sphinx_rtd_size_width = "1200px"

hammer/config/config_src.py

@@ -805,6 +805,7 @@ def get_config(self) -> dict:
  self.__config_cache_dirty = False
  return self.__config_cache
 
+ @property
  def get_config_types(self) -> dict:
  """
  Get the types for the configuration of a database.
@@ -906,9 +907,9 @@ def get_setting_type(self, key: str, nullvalue: Any = None) -> Any:
  :param nullvalue: Value to return for nulls.
  :return: Data type of key.
  """
- if key not in self.get_config_types():
+ if key not in self.get_config_types:
  raise KeyError(f"Key type {key} is missing")
- value = self.get_config_types()[key]
+ value = self.get_config_types[key]
  return nullvalue if value is None else value
 
  def set_setting_type(self, key: str, value: Any) -> None:
@@ -928,7 +929,7 @@ def has_setting_type(self, key: str) -> bool:
  :param key: Desired key.
  :return: True if the given setting exists.
  """
- return key in self.get_config_types()
+ return key in self.get_config_types
 
  def check_setting(self, key: str, cfg: Optional[dict] = None) -> bool:
  """
@@ -940,12 +941,12 @@ def check_setting(self, key: str, cfg: Optional[dict] = None) -> bool:
 
  if cfg is None:
  cfg = self.get_config()
- if key not in self.get_config_types():
+ if key not in self.get_config_types:
  #TODO: compile this at the beginning instead of emitting every instance
  #self.logger.warning(f"Key {key} is not associated with a type")
  return True
  try:
- exp_value_type = parse_setting_type(self.get_config_types()[key])
+ exp_value_type = parse_setting_type(self.get_config_types[key])
  except ValueError as ve:
  raise ValueError(f'Key {key} has an invalid outer type: perhaps you have "List" instead of "list" or "Dict" instead of "dict"?') from ve
 

hammer/flowgraph/__init__.py

@@ -0,0 +1,5 @@
+# Hammer logging code.
+#
+# See LICENSE for licence details.
+
+from .flowgraph import *