MyST Markdown editor component for easier collaboration

At Svevn, Antmicro’s web-oriented subsidiary, our goal is to provide our customers with beautiful and user-friendly gateways into the physical world, building specialized portals and UIs taking advantage of the latest developments in web technologies. The development of modern software, which is typically built from a large number of predominantly open source components, is a highly collaborative activity, not least because it typically revolves around version control systems such as git and portals like GitHub and GitLab.

While the idea of collaborative git-based code editing, being driven by developers themselves, is fairly well articulated in the form of various open source tools and frameworks by now, open source, collaborative git-backed editing of textual content has not been as widely pursued.

Perhaps this discrepancy partly originates from the fact that documentation is not the most beloved of tasks in a developer’s daily work. But good tools can turn difficult tasks into pleasant ones, and as a lot of our daily work involves lots of text - copywriting, documentation, READMEs, company documents, and blog notes like this one - we have been working on a collaborative editing MyST Markdown component to simplify our and our customers’ workflows around text.

Importance for workflow

It is natural for companies on the verge between technology and design to want complete control over how text is managed and presented to users and developers. Consistent styling as well as programmatic processing capability and complete edit history are essential to our workflows, which makes us strong believers in text-based markup languages such as Markdown and RestructuredText.

Using plain text allows you to separate content from form, version control your documents, and encourages reuse; markup languages put focus back on the text itself with uniform and syntactically-driven styling, as opposed to WYSIWYG approaches where keeping consistent styling works until it doesn’t. Keeping content/structure and style separate just like you do in web development with HTML and CSS is important e.g. when rolling out branding changes, working with a wide range of customers with different branding needs, designing documentation for diverse open source projects or repurposing text for other media.

The documentation use case: Sphinx and RST

Svevn’s documentation work predominantly relies on Sphinx - a well-established documentation system that allows us to generate both PDF and HTML from a single source. Sphinx in turn builds on the expressive power of ReStructuredText, or RST, which - with its extendible system of roles and directives provides a lot of flexibility.

RST and Sphinx are incredibly powerful as far as the breadth of capabilities and widespread use in documentation of major projects such as Zephyr or Linux is concerned, but the lack of interoperability with the modern Javascript stack and Markdown ecosystem is a serious limitation. The major hurdle originates from the fact that there is no mature browser-based RST parser, which implies a desktop-driven workflow.

Thus, the idea behind what eventually became the MyST editor component was to somehow overcome this limitation and move Svevn’s content editing workflow more towards the web and have a convenient tool with additional styling and PDF generation capabilities, keeping the easy versioning and comparison of purely textual documents. Additionally, to not disrupt existing workflows, the documents had to be interoperable with Sphinx and other Pythonic frameworks (Jupyter, etc.), and processing had to be done only on the client side.

Enter MyST

The major open source building block of the solution here (as conceived by the excellent Executable Books community which made it happen) involves a simple yet brilliant concept: take a widely recognized standard Markdown format, CommonMark, alongside it’s excellent parser and other tooling ecosystem, and extend it with best parts of RST, such as flexible and extendible directives and roles. At the same time, to keep the interoperability as tight as possible, also extend Sphinx (which is the most notable RST-based tool out there) to accept the resulting format - MarkedlyStructuredText, or MyST - for a best-of-both-worlds scenario.

Thus instead of using indented blocks as you normally would with RST’s directives:

.. directive:: required argument
   :optional argument: value

   body

MyST allows you to use the following syntax:

```{directive} required argument
:optional argument: value

body

```

With Markdown’s handy heading syntax, intuitive links, and other well-known syntactic elements, MyST is easy to write for both RST and MD users. The capabilities of the two formats are pretty much identical, and a Docutils parser for MyST allows you to plug it directly into Sphinx with all its capabilities, plugins, etc. Migration from RST is thus very natural and only requires one Sphinx plugin, and trying MyST out actually feels like magic: rewriting content between the two formats is a breeze, and the two can even be mixed together on a per-file basis.

And of course, most importantly, basing MyST on CommonMark meant that implementing support in the browser was possible by means of extending well-established JavaScript Markdown parsers, and thus markdown-it-docutils was born.

The fun begins in the browser

Having a JS parser opens up the door to a world of possibilities where you do not need a server to build a great user experience for editing content. Discovering markdown-it-docutils pushed us to start working on tools that we had been discussing for a long time, but since we had to cover a variety of workflows that used different syntaxes and tools, there was never a clear way of building a single tool for all of the use cases.

MyST is a means of unifying various internal efforts under a single syntax and toolset, allowing us to create a single editing component that can be used to meet the diverse internal needs as well as those of our clients. We also wanted to experiment with pure browser-based experiences, leveraging the new browser module capabilities and tools like Preact and HTM, to cut down on the typically extremely heavy scaffolding surrounding even the simplest of front-end projects. The result of our efforts and experience so far is the MyST content editor, which strives to build on top of MyST’s strengths to offer some unique capabilities for seamless content editing in various scenarios.

MyST content editor

The MyST content editor uses a classical two-pane edit/preview system with an (optional) toolbar, deliberately refraining from adding more complex panel layouts to allow for easy integration with Svevn’s various internal systems. While the editor as such is not necessarily strictly limited to MyST, we believe that focusing on that format will allow us to build the most cohesive experience which transcends beyond individual file editing and into the realm of complex documents, such as technical documentation.

The editor component as such is fairly minimalistic, offering the ability to toggle between source / preview or both, including of course live preview of your rendered Markdown content. Additionally, the content can be easily exported to a styled PDF (a custom callback can be used for more advanced PDF and report generation capabilities which we use at Svevn) or displayed in full-screen preview mode for a better viewing experience.

Another feature is a plugin manager which allows you to load additional syntax constructs like TeX-style math, CriticMarkup etc. The basic MyST syntax is of course included by default.

Building the MyST content editor as a component that needs to “live” within various other internal systems was also an opportunity to play around with some new capabilities of the web in terms of packaging, support of ESM modules etc. The component is using the Preact/HTM framework combination, with natively-supported tagged templates instead of JSX to allow for importing it as a single JS source file, without any bundling whatsoever.

By deliberately limiting the feature set of the component, allowing for easy prototyping and integration and basing it on the concept of plugins, we want myst-editor to be a good starting point for adding more custom features and integrating with other services.

Upcoming work

We believe that the natural next step in the development of the MyST content editor is to allow users to create content collaboratively using WebRTC as well as add more capabilities into the source pane. Building a performant, collaborative content editing platform is no easy task, but fortunately there are frameworks such as yjs and CodeMirror which provide a very good starting point. Integrating those two frameworks into myst-editor is now in progress, while simultaneously we are integrating the component into more of our internal workflows. These new features, once available, should greatly increase the pace of adoption!

Develop your next web app with Svevn, Antmicro’s web-oriented subsidiary

If you need commercial help in developing your next web application, especially one that involves both content creation, design and new web technologies, Svevn brings Antmicro’s expertise and open source-based approach to front-end solutions. We are always curious about your projects, so feel free to reach out to us at contact@svevn.com.