New features in Pipeline Manager: subgraphs, extended UI, full frontend support, and improved API

Proper visualization is key to better understanding of all kinds of complex systems and data flows. At Antmicro, we use graphs to visualize AI optimization pipelines and applications using Kenning, digital designs written in Verilog using Topwrap, entire hardware/software systems using System Designer, and more. For all these tools, we actively develop and use Pipeline Manager, an open source, application-agnostic library for creating interactive UIs with graphs.

Our previous blog note described how to connect CLI and server-based applications to the Pipeline Manager frontend, in order to support basic graph manipulation, run tasks in the app or forward logs or terminal views. Since then, we further improved the integration of graphs, not only in server-client and standalone scenarios, but also in the documentation and other frontend applications, providing convenience libraries and allowing two-way communication for and reporting and changing the graphs’ state. With further API improvements, it is now possible to modify all aspects of graphs, including their style, structure, specification and layout.

This blog note will guide you through various ways in which Pipeline Manager can be integrated in workflows, hopefully generating new ideas for using the tool in different contexts.

Integrating Pipeline Manager with external applications

As mentioned, Pipeline Manager has already been integrated into multiple applications and websites, such as Antmicro’s System Designer. For these integrations, we needed to control the appearance of Pipeline Manager from the (hosting) frontend and notify it when Pipeline Manager’s state is changed by the user.

Pipeline Manager already contained a mechanism to communicate with an external backend using JSON Remote Procedure Calls (JSON RPC), for use with CLI-level applications such as Kenning or Topwrap. To complement this, we now added the ability to communicate between Pipeline Manager and the containing website using postMessage cross-origin communication. This method can be used to exchange information between a website and e.g. Pipeline Manager’s iframe embedded in it, using the same JSON RPC format for calls to read and modify the state of Pipeline Manager.

Server-based applications can make use of Pipeline Manager in their UI, with help of Pipeline Manager’s Backend Communication library, providing a base asynchronous server to communicate with the frontend, as well as a set of convenience classes and methods to set up necessary callbacks. With this, you can implement sending requests and handling responses using a few simple methods in a class passed to the server (in Python), like so:

class RPCMethods:
    """Class implementing RPC methods."""

    def specification_get(self) -> dict:
        path = Path("sample-specification.json")
        specification = {}
        with open(path, "rt") as fd:
            specification = json.load(fd)
        return {
            "type": MessageType.OK.value, 
            "content": specification,
        }

    def dataflow_validate(self, dataflow: dict) -> dict:
        # validate the graph
        return {"type": MessageType.OK.value}

    def dataflow_run(self, dataflow: dict) -> dict:
        # run the application defined by the graph
        return {"type": MessageType.OK.value}

    # ...

The same API can be used for Javascript-level communication, where we can obtain an element containing Pipeline Manager using document.getElementById and call postMessage on that element to send a request to Pipeline Manager. Responses to requests, as well as handling requests from Pipeline Manager, can be done by listening to message events.

Below, you can find a code sample of such frontend-to-frontend communication:

const iframe = document.getElementById(
   pipelineManagerIframe
);

// receiving messages from Pipeline Manager
window.addEventListener('message',
   (event) => {
       const response = JSON.stringify(
          event.data
       );
       console.log(response);
       switch (event.data.method) {
           case 'specification_get':
               iframe.contentWindow.postMessage({type: 0, content: specification});
               break;
           // ...
       }
   }
);

// sending messages to Pipeline Manager
iframe.contentWindow.postMessage({ 
   method: 'specification_change',
   params: { specification } 
});

iframe.contentWindow.postMessage({
   method: "terminal_write",
   params: { 
      name: "Terminal",
      message: "Hello world!",
   }
});

Note that you can use the SpecificationBuilder to quickly create a specification in Python or Javascript without worrying too much about the internal format of the specification files.

What is more, we introduced numerous new API calls you can use to change the structure, appearance and view of the graph. With two-way communication, it is possible to immediately obtain information on what a user is modifying and how, what is the currently focused node, and change things in the UI with respect to such events (e.g. create visualization of a network where clicking on any node representing a device shows its log). For a detailed list of API calls, check the Pipeline Manager API documentation.

Test the communication with Pipeline Manager

In the mainline repository, there is a nice example of a frontend-only version of Pipeline Manager, which lets you observe communication from and to Pipeline Manager, as well as send custom requests to the frontend.

For instance, you can insert a specification or graph into the first terminal window (bottom left), and then click “Set specification” or “Set graph” to send the request to the frontend; you can also run any other request - insert the test API call provided below to change the position of one of the nodes, and click “Handle provided request”:

{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "position_change",
    "params": {
        "graph_id": "78cc86c4-9ad0-4a8f-88cb-71ee28c48659",
        "node_id": "6df5df60-3a6b-44a9-943d-54f3a2207ce7",
        "position": {
            "x": 500,
            "y": -400
        }
    }
}

Here are two other requests, for writing in the terminal and highlighting a node - you can copy-paste them into the terminal and send them to Pipeline Manager, too:

{
    "jsonrpc": "2.0",
    "id": 2,
    "method": "terminal_write",
    "params": {
        "name": "Terminal",
        "message": "Hello world!"
    }
}

{
    "jsonrpc": "2.0",
    "id": 3,
    "method": "nodes_highlight",
    "params": {
        "graph_id": "78cc86c4-9ad0-4a8f-88cb-71ee28c48659",
        "nodes": {
            "selected": [
                "5c398d8d-1ee2-41ab-afa5-1a07efac3f8d"
            ],
           "unselected": []
        }
    }
}

More customization options in diagram styles and contents

We introduced various improvements to styling and data representation, which allow you to create comprehensive graphs with easy-to-follow visualizations for all kinds of data.

It is now possible to add colors to nodes, either in node type or individual node instances in the graph. Along with interface and connection coloring, it can be used to mark the currently processed node in the flow, e.g. in a Kenning optimization flow, or to mark some specific path in the diagram. It can also be used to visually mark node types belonging to same group and more. Combine this with API calls, and you can have distinct nodes colored dynamically by their status.

Another node styling feature that we introduced are “pills” which you can add throughout your graph - you can think of these as tags with extra information.

Here’s the GaussianKernel node definition from the example presented above (in JSON):

{
   "name": "GaussianKernel",
   "layer": "kernel",
   "category": "Generators",
   "properties": [
       {
          "name": "size",
          "type": "integer",
          "default": 5
       },
       {
           "name": "sigma",
           "type": "number",
           "default": 1.0
       }
   ],
   "interfaces": [
       {
           "name": "kernel",
           "type": "Image",
           "direction": "output"
       }
   ],
   "color": "#6f091a",
   "pill": {"text": "kernel", "color": "#999999"}
},

Pipeline Manager automatically tracks which nodes are new or have been edited, and marks them as such, using configurable node styles. You can also use this feature to create your own styles, and apply them to any nodes of your choice. Colors and pills can also be combined into your custom styles, which control the visual aspects of nodes, making it easier for you to apply changes to many such nodes at once.

There is also a new multiline property type available, which allows you to add and edit text inside the nodes. The text is processed with Markdown, allowing the introduction of rich text with formatting, links, images and more, so you can include URLs, pictures and code snippets in the nodes, further enhancing Pipeline Manager’s visualization capabilities.

Easier customization of the specification in UI

Previous versions of Pipeline Manager only supported node types defined in the provided specification, and those node types could not be modified. Now, it is possible to modify the specification from the frontend level and then save it.

Pipeline Manager allows you to modify the specification from the UI in two ways, by:

• adding, removing and changing node types by using configuration menus (accessible through the node context menu).
• editing the node specification in YAML format using the editor (which features live validation of the code).

This aims to facilitate drafting diagrams on the fly, as now you can create a graph from scratch, without prior specification.

Once you enable “Modify node types” in Settings, you can right-click on a node header to evoke a context menu, from which you can add properties or interfaces, and modify the node in other ways - these changes will then be propagated to all nodes of the same type.

Pulling modular specification from external sources

Our recent improvements to the specification format allow you to import its parts from external sources, introducing a modular structure. This can be achieved with the include keyword, which holds an array of URLs pointing to JSON files. Node types defined in these files will be available in the editor. To be able to easily distinguish the affected nodes, you can also assign a style to all the nodes imported from one source.

Here is a sample specification which makes use of this mechanism:

    "include": [
        {
            "url": "https://raw.githubusercontent.com/antmicro/kenning-pipeline-manager/main/examples/sample-specification.json",
            "style": [
                "import",
                "miscellaneous"
            ]
        },
        "https://raw.githubusercontent.com/antmicro/kenning-pipeline-manager/main/examples/sample-loopback-specification.json",
        "https://raw.githubusercontent.com/antmicro/kenning-pipeline-manager/main/examples/sample-multiple-io-specification.json"
    ],
    "metadata": {
        "styles": {
            "import": {
                "icon": "Import",
                "pill": {"text": "Import", "color": "#ffffff"}
            },
            "miscellaneous": {
                "color": "#000000",
                "pill": {"text": "misc", "color": "#2271b3"}
            }
        }
    }

We have also added an includeGraphs keyword, which can be used as a substitute for graphs when defining subgraphs. You can use it to include a graph file, and link the graphs defined in it to your respective nodes.

Better navigation in multi-graph diagrams with subgraphs and graph links

Complex diagrams, especially those with overlapping layers, pose a challenge in building and maintaining without sacrificing their readability. One possible solution to this problem is a multi-graph, layered view. With this in mind, we have extended support for multiple graphs in the editor - subgraphs and related graphs - to facilitate creating diagrams for complex use cases.

Subgraphs help you introduce extra layers of complexity to your diagrams while keeping them intuitive and readable. This works by having the subgraphs embedded into nodes; once opened, a subgraph displays like any other graph, such as your main one, in a separate view. Subgraphs now come with improved flexibility: a subgraph node can be styled like any other node, and you can nest subgraphs, too.

For example, your main graph can be demonstrating the hardware overview of your board - the modules used and the connections between them - while for each module (node), you can define a subgraph, describing the individual components which the unit (represented by the node) is made of.

Likewise, you may wish to create related graphs - this is another new feature, which allows you to link a node to any graph, regardless of the subgraph hierarchy. What this allows is, for instance, demonstrating different aspects of the same node, e.g. for the aforementioned board(s) you can create a graph representing the software running there and the interactions between applications.

Your system, visualized in Pipeline Manager

Recent improvements to Pipeline Manager help visualize complex information and let you easily and conveniently create diagrams of many different systems, pipelines, ecosystems, and similar. Pipeline Manager does not assume anything about the data being presented, enabling all sorts of use cases. Notably, Pipeline Manager’s API can be used to embed it as a framework for creating (and managing) diagrams in your system, application, or website, and Antmicro can help incorporate it into new use cases based on the experience of using it for a variety of internal and customer applications.

We will be happy to demonstrate and discuss the possibilities that Pipeline Manager offers, in the context of your own project and use case. Contact us at contact@antmicro.com to start a conversation about this.