herringtondarkholme/vue-compiler

vue template compiler in Rust

code generation and ast parsing in the same data structure

https://github.com/vuejs/rfcs/discussions/369#discussioncomment-1192421

Maybe in the long run we want the whole transform (and even the Vue compiler!) to be implemented in native Go/Rust so performance would no longer be a concern ;)

Future is now!

Architecture

The original design in vue-next mixes . As we can see, the transform pass will in-place mutate ast nodes, leaving the node with both code generation node and ssr code generation node.

This is typically a sign of leaky abstraction.

So in the Rust version I decided to take another approach.

The design targets at three different levels of developers in Vue ecosystem:

  • Lib/App creator: every Vue developers who write component library or application code.
  • Platform developer: Devs who write compiler implementation for DOM/SSR/Native platform.
  • Framework author: Vue core lib author a.k.a Evan.

The core framework targets multiple platforms and can be extended to support more. Core framework components span all platforms and are hardwired to the core lib runtime.

Platforms are usually DOM or SSR environment. Hosts are browser and node, respectively. Developing a platform needs to write code for both vue-compiler and vue-runtime. Optionally platform developer can write code in host, e.g. in hybrid app or mini-program.

And finally lib or app creators can write component library, business code or application components targeted to certain platforms.

The compilation has several phases:

  • Scan (output: Tokens): Hardwired in the compiler at framework level.
  • Parse (output: template AST): Hardwired in the compiler at framework level.
  • Convert (output: intermediate representation): Customizable for platform developers with sensible default.
  • Transform (input/output: customizable IR): Customizable with default by using generic/traits.
  • Code Generate (customizable output: e.g. JS/TS): Customizable with default.

Other Design different from the original compiler

  • Directive parsing is implemented manually instead of by regexp.
  • nodeTransforms is not supported. It's too hard for app creator to use and maintain IR invariants. Platform devs can still customize by implementing converter/transformer.
  • directiveTransforms now can returns not only Props but also SimpleExpression. The extra flexibility makes a more cohesive v-bind/v-on conversion: the logic for processing the directives now resides in one single file without regard to the presence of an argument.
  • Runtime helper collection context.helper/helperString is moved out from convert and tracked in transform phase, avoiding several methods and reducing HashMap to a bitflag.

Intended Usage

  • Rust library
  • CLI binary
  • napi based nodejs library
  • wasm based npm package: a fallback if napi fails to work and a toy for browser.
  • No Browser build No support since most features in full build are additional except for browser based expression checking or HTML escaping. Browser build removed them for size. But template compiler in browser is already for toy project. For browser specific summary see this google sheet.

Implementation Detail

  • Plenty of debug_asserts to maintain compiler state invariants.
  • The library seeks minimal allocation by using &str, Cow<'_, str> and smallvec.
  • A customized VStr is used to minimize string manipulation.
  • Fxhash is preferred over default hasher since hash collision is not a concern.
  • The bitflags crate is used to represent runtime helper and vnode patch flags.
  • Use heavily optimized routines for string search primitives. (Perf reference)
  • Benchmark with criterion.rs.
  • Test compiler output by snapshot test.
  • Use alternative allocator like wee_alloc or mi_malloc.
  • Use Box<[T]> instead of Vec to reduce type size.
  • Use Arean to minimize allocation.
  • A Future like stack-allocated transformation Pass composition.
  • Use Rc to manage error handler. Don't optimize wrong code.
  • Parallelized conversion with Rayon.

Reference

  • vue-next: ご本家様
  • html spec is the definitive guide for parsing HTML-like files.
  • Vue Template Explorer gives instant results for code generation and error reporting.
  • Nu html checker is the official html validator from W3C. This is the canonical error reporter for html parsing, when there is a discrepancy between the framework and the spec.
  • AST explorer can inspect AST nodes interactively.

Roadmap

Todo tasks grouped by scopes.

[util]

  • VStr
    • string intern
    • camel/pascal cache
    • str ops

[core]

  • scanner
    • UTF8 support
  • parser
  • IR converter
    • v-if
    • v-for
    • v-slot
    • v-model
    • slot outlet
    • element
    • build props
  • transformer
    • SWC RSLint integration
    • Rewrite MergePass struct
  • code generator
    • module preamble
  • wrap error handler in Rc
  • compile option
  • Arena allocation
  • Parallelization

[dom]

  • IR converter
    • v-on
    • v-once
    • v-memo
  • transformer
  • code generator

[ssr]

  • TODO

[sfc]

  • TODO

[test]

  • scanner test
  • parser test
    • dir parser test
  • Add insta snapshot
  • Move snapshot outside of src

[bench]

  • Add benchmark framework
  • Micro benchmarks for scanner
  • Micro benchmarks for parser
  • Micro benchmarks for converter
  • Micro benchmarks for transformer
  • Micro benchmarks for codegen
  • Integrated benchmarks using repos like Element-Plus

[infra]

  • Add pre-commit hooks.
  • Add Github Actions for various checks.
  • Change single lib to cargo workspaces.

[community]

  • TODO. not ready for contribution for now.
Issues

Collection of the latest Issues

HerringtonDarkholme

HerringtonDarkholme

Comment Icon0

The experimentation shows that rayon has at least 100us overhead for thread pool initialization. However, most template can be compiled within 100us. The parallelization, at least in practice, does not pay off.

The example is the most complicated SFC in element-plus, el-table.vue. The compilation takes about 100us without thread pool.

But after the thread pool is introduced the compilation time comes to 200us+.

HerringtonDarkholme

HerringtonDarkholme

Comment Icon4

TLDR: Rslitn is still better than swc.

The code changes are in swc branch and rslint branch.

swc contains a lot of dependencies regardless if they are relevant to core parsing. Such bloated dependencies place huge burden to compiling, slowing rust-analyzer to almost freezing. swc's docs and examples re scarce. The only working example is https://rustdoc.swc.rs/swc_ecma_parser/. Looking to the code is also hard. The abstraction and module organization is, well, hazy at least to the uneducated. Peeking the definition is hard, if possible, given the massive usage of macros. Alas, the macro is also the perpetrator of the sluggish compilation. :/ Using swc is not a nice journey, actually. Looking at the example above, it immediately requires several crates other than the parse. common, ast, atom, visit and blahblah. And the core impl Visitor has 200+ macro generated methods to implement without one single line of documentation. The usability is poor... And the output is large, merely importing swc pushes the binary size to 33MB.

Rslint at least has more comments and documentation than swc. It's underlying crate, rowan, also has docs. So it might be a better choice. Rslint's dependencies are also more lightweight. Rslint's source code is also simple and clear, compared to swc. Understanding Green/Red tree does require some learning. But it is fine. The binary size is 10MB after importing Rslint, one third of swc.

renovate[bot]

renovate[bot]

Comment Icon0

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

This repository currently has no open or pending branches.

Detected dependencies

cargo
benches/Cargo.toml
  • criterion 0.3
  • compiler
  • glob 0.3.0

crates/cli/Cargo.toml

  • anyhow 1.0.58
  • clap 3.2.8
  • compiler
  • dom
  • path-clean 0.1.0
  • codespan-reporting 0.11.1
  • serde_yaml 0.8.25

crates/compiler/Cargo.toml

  • smallvec 1.9.0
  • bitflags 1.3
  • rustc-hash 1.1.0
  • serde 1.0
  • rslint_parser 0.3.1
  • phf 0.10
  • insta 1.15.0
  • lazy_static 1.4.0

crates/dom/Cargo.toml

  • compiler
  • phf 0.10

crates/ref_transform/Cargo.toml

  • tree-sitter 0.20.8
  • cc *

crates/sfc/Cargo.toml

  • smallvec 1.9.0
  • rustc-hash 1.1.0
  • rslint_parser 0.3.1
  • compiler

crates/ssr/Cargo.toml

  • compiler

crates/wasm/Cargo.toml

  • compiler
  • wasm-bindgen 0.2.81
  • wee_alloc 0.4.5

napi/Cargo.toml

  • napi 2.6.2
  • napi-derive 2.6.0
  • compiler
  • dom
  • napi-build 2
  • mimalloc 0.1

github-actions

.github/workflows/baseline.yml
  • actions/checkout v3
  • actions/setup-node v3
  • actions/cache v3
  • pnpm/action-setup v2.2.2
  • benchmark-action/github-action-benchmark v1

.github/workflows/benchmark.yml

  • actions/checkout v3
  • actions-rs/toolchain v1
  • actions/cache v3

.github/workflows/ci.yml

  • actions/checkout v3
  • actions-rs/toolchain v1
  • actions/cache v3

.github/workflows/coverage.yml

  • actions/checkout v3
  • codecov/codecov-action v3

.github/workflows/gh-pages.yml

  • actions/checkout v3
  • actions-rs/toolchain v1
  • jetli/wasm-pack-action v0.3.0
  • actions/cache v3
  • actions/setup-node v3
  • pnpm/action-setup v2.2.2
  • peaceiris/actions-gh-pages v3

npm

benches/package.json
  • @types/benchmark ^2.1.1
  • @types/glob ^7.1.4
  • @vue/compiler-core ^3.2.19
  • benchmark ^2.1.4
  • glob ^8.0.0
  • microtime ^3.0.0
  • ts-node 10.8.2
  • typescript 4.7.4

napi/npm/android-arm64/package.json

  • node >= 10

napi/npm/darwin-arm64/package.json

  • node >= 10

napi/npm/darwin-x64/package.json

  • node >= 10

napi/npm/freebsd-x64/package.json

  • node >= 10

napi/npm/linux-arm-gnueabihf/package.json

  • node >= 10

napi/npm/linux-arm64-gnu/package.json

  • node >= 10

napi/npm/linux-arm64-musl/package.json

  • node >= 10

napi/npm/linux-x64-gnu/package.json

  • node >= 10

napi/npm/linux-x64-musl/package.json

  • node >= 10

napi/npm/win32-arm64-msvc/package.json

  • node >= 10

napi/npm/win32-ia32-msvc/package.json

  • node >= 10

napi/npm/win32-x64-msvc/package.json

  • node >= 10

napi/package.json

  • @node-rs/helper ^1.2.1
  • @napi-rs/cli 2.10.1
  • @swc-node/register 1.5.1
  • @types/node 16.11.43
  • @vue/compiler-core 3.2.37
  • @vue/compiler-sfc 3.2.37
  • @typescript-eslint/eslint-plugin 5.30.5
  • @typescript-eslint/parser 5.30.5
  • ava 4.3.0
  • benny 3.7.1
  • chalk 5.0.1
  • eslint 8.19.0
  • eslint-config-prettier 8.5.0
  • eslint-plugin-import 2.26.0
  • eslint-plugin-prettier 4.2.1
  • eslint-plugin-sonarjs 0.13.0
  • npm-run-all 4.1.5
  • prettier 2.7.1
  • typescript 4.7.4
  • node >= 10

playground/package.json

  • vue 3.2.37
  • @babel/types 7.18.8
  • @vitejs/plugin-vue 2.3.3
  • typescript 4.7.4
  • vite 2.9.14
  • vue-tsc 0.38.3
  • rusty-vue-compiler file:../crates/wasm/pkg/

  • Check this box to trigger a request for Renovate to run again on this repository
HerringtonDarkholme

HerringtonDarkholme

Comment Icon0

Hard to migrate but possible

  • compiler-core/src/transforms/vFor.ts|304 col 15| const forAliasRE = /([\s\S]*?)\s+(?:in|of)\s+([\s\S]*)/
  • compiler-core/src/transforms/vFor.ts|307 col 18| const forIteratorRE = /,([^,\}\]]*)(?:,([^,\}\]]*))?$/
  • compiler-core/src/transforms/vOn.ts|19 col 12| const fnExpRE = /^\s*([\w$_]+|\([^)]*?\))\s*=>|^\s*function(?:\s+[\w$]+)?\s*\(/
  • compiler-core/src/utils.ts|79 col 17| const whitespaceRE = /\s+[.[]\s*|\s*[.[]\s+/g
  • shared/src/normalizeProp.ts|30 col 20| const listDelimiterRE = /;(?![^(]*\))/g // ; followed by no single )

Easy to migrate

use RE in SFC crates

Information - Updated Jul 10, 2022

Stars: 504
Forks: 19
Issues: 6

A fantasy deathcrawl in Rust

To run, with Rust compiler and Cargo package manager installed:

A fantasy deathcrawl in Rust

MIRAI is an abstract interpreter for the Rust compiler's mid-level intermediate

MIRAI is an abstract interpreter for the mid-level intermediate

MIRAI is an abstract interpreter for the Rust compiler's mid-level intermediate

Rust compiler toolkit for WebAssembly apps in the WasmEdge Runtime

Developers: Getting started with the Tencent Serverless Functions for AI inference, or WasmEdge Runtime

Rust compiler toolkit for WebAssembly apps in the WasmEdge Runtime

guessing_game_rust

A repo used to learn rust using the Rust compiler

guessing_game_rust

owner-thing-rust

A repo used to learn rust using the Rust compiler

owner-thing-rust

enums_thing_rust

A repo used to learn rust using the Rust compiler

enums_thing_rust

collections-rust

A repo used to learn rust using the Rust compiler

collections-rust

A snake game written in Rust

Download Rust compiler from

A snake game written in Rust

C Compiler in Rust

A basic C compiler written in Rust, roughly following the tutorial official Rust compiler was taken as inspiration

C Compiler in Rust

It is rust bindings and wrapper around libconfig library

It is rust bindings and wrapper around Rust Compiler

It is rust bindings and wrapper around libconfig library

Toy Rust Compiler

A compiler can be broken down into 4 parts

Toy Rust Compiler
Facebook Instagram Twitter GitHub Dribbble
Privacy