Show HN: Parse LLM Markdown streams incrementally on the server or client

Notifications You must be signed in to change notification settings

Fork 0

Star 8

BranchesTags

Open more actions menu

Folders and files

NameName

Last commit message

Last commit date

Latest commit

History

22 Commits

22 Commits

demo

demo

packages

packages

.gitignore

.gitignore

LICENSE

LICENSE

README.md

README.md

biome.json

biome.json

package-lock.json

package-lock.json

package.json

package.json

turbo.json

turbo.json

Repository files navigation

A markdown parser with streaming support, suitable for incrementally parsing LLM markdown streams. Parses markdown into a structured fully typed tree of nodes, following the CommonMark specification. It supports streaming/incremental parsing, so you can feed it growing input and emit only the blocks that have become finalized.

Installation

npm install markdown-parser

Usage

import { MarkdownParser } from "markdown-parser";

const parser = new MarkdownParser();

// Parse complete markdown const nodes = parser.parse("# Hello World\nThis is a paragraph."); // [ // { type: "heading", level: 1, children: [{ type: "text", text: "Hello World" }] }, // { type: "paragraph", children: [{ type: "text", text: "This is a paragraph." }] } // ]

// Parse with streaming mode (for incremental content) const partialNodes = parser.parse("# Hello World\nThis", { stream: true }); // Emits heading, but not the paragraph (still open) // [ // { type: "heading", level: 1, children: [{ type: "text", text: "Hello World" }] }, // ]

// Continue parsing as more content arrives const moreNodes = parser.parse(" is a paragraph\n\nThis is another paragraph.", { stream: true }); // Emits the paragraph // [ // { type: "paragraph", children: [{ type: "text", text: "This is a paragraph." }] } // ]

// Access all finalized blocks parsed so far const stableNodes = parser.nodes;

// Optionally preview the currently open block while streaming const openNode = parser.experimental_openNode;

const finalNodes = parser.parse("", { stream: false }); // Closes anything still open and emits remaining blocks // [ // { type: "paragraph", children: [{ type: "text", text: "This is another paragraph." }] } // ]

When stream is false (default), the parser finalizes all open blocks at the end of the input and returns the full set of blocks (for that input). When you parse in streaming mode, the parser keeps internal state across calls and returns only blocks that have become closed and stable since the last call.

API

MarkdownParser

The main parser class that converts markdown text into a structured block AST (headings, paragraphs, lists, etc.).

parse(text: string, options?: { stream: boolean }): BlockNode[]

Parses markdown text and returns an array of block nodes.

text - The markdown text to parse

options.stream - When true, enables streaming mode which buffers incomplete blocks until they can be fully parsed. Defaults to false.

nodes: BlockNode[]

Returns all finalized top-level block nodes parsed so far. In streaming mode, this does not include the currently open block.

experimental_openNode: OpenBlockNode | null

Returns a best-effort preview of the currently open top-level block while streaming, or null when there is no open block. The open node has not finalized yet, so it may change, disappear, or be reinterpreted as more input arrives.

This property is experimental and subject to change.

Supported Nodes

The parser provides 100% support for the CommonMark specification, and includes full support for GitHub Flavored Markdown (GFM) tables.

Block nodes

Heading (ATX and setext style)

Paragraph

Code block (fenced and indented)

Thematic break (horizontal rule)

HTML block

Blockquote

List (ordered and unordered)

Link reference definitions

Table (GFM)

Inline nodes

Text

Code span

Hard break

Soft break

HTML (inline)

Autolink

Link

Image

Emphasis

Strong

Some notes on the implementation

The implementation is inspired by various other markdown parsers, including commonmark.js, markdown-it, and marked.js. In fact, the implementation is structurally very similar to how commonmark.js goes about parsing; the only major difference is how we decide which lines to parse when streaming is set to true. I started with a much simpler and a lot more readable implementation for the parser, but it became complex when adding block container (blockquote and lists) support, so I ended up going for a slightly complex solution but a more robust and extensible one.

Commonmark specification allows link reference definitions to appear after the links that use them. Therefore, when streaming is enabled, it is important to consider that a link reference might not resolve, since its definition could arrive in a later chunk of the input.

About

A streaming-capable markdown parser, written in TypeScript

markdownparser.vercel.app/experimental

Resources

Readme

License

MIT license

Uh oh!

There was an error while loading. Please reload this page.

Activity

Stars

8 stars

Watchers

0 watching

Forks

0 forks

Report repository

Releases

No releases published

Packages 0

Uh oh!

There was an error while loading. Please reload this page.

Contributors

Uh oh!

There was an error while loading. Please reload this page.

Languages

TypeScript 100.0%