vue template compiler in Rust

https://github.com/vuejs/rfcs/discussions/369#discussioncomment-1192421

Maybe in the long run we want the whole transform (and even the Vue compiler!) to be implemented in native Go/Rust so performance would no longer be a concern ;)

Future is now!

Design

The original design in vue-next mixes . As we can see, the transform pass will in-place mutate ast nodes, leaving the node with both code generation node and ssr code generation node.

This is typically a sign of leaky abstraction. So in the Rust version I decided to take another approach.

The compilation has several phases:

Scan (output: Token)
Parse (output: template AST)
intermediate representation
transformation/optimization pass
output generation

Intended Usage

Rust library
CLI binary
napi based nodejs library
wasm based npm package

Implementation Detail

The library seeks minimal allocation by using &str, Cow<'_, str> and smallvec.
Fxhash is preferred over default hasher since hash collision is not a concern.
The bitflags crate is used to represent runtime helper and vnode patch flags.
Use heavily optimized routines for string search primitives.
Benchmark with criterion.rs.
Test compiler output by snapshot test.

Reference

vue-next: ご本家様
html spec is the definitive guide for parsing HTML-like files.
Vue Template Explorer gives instant results for code generation and error reporting.
Nu html checker is the official html validator from W3C. This is the canonical error reporter for html parsing, when there is a discrepancy between the framework and the spec.
AST explorer can inspect AST nodes interactively.

Performance Related Reference

https://lise-henry.github.io/articles/optimising_strings.html

Roadmap

Todo tasks grouped by scopes.

[core]

tokenizer
parser
IR converter
transformer
code generator

[dom]

transformer
code generator

[ssr]

TODO

[sfc]

TODO

[test]

Add unit test
Add insta snapshot

[bench]

Add benchmark framework
Micro benchmarks for compiler components
Integrated benchmarks using repos like Element-Plus

[infra]

Add pre-commit hooks.
Add Github Actions for various checks.

[community]

TODO. not ready for contribution for now.

Issues

Collection of the latest Issues

trueLoving

The experimentation shows that rayon has at least 100us overhead for thread pool initialization. However, most template can be compiled within 100us. The parallelization, at least in practice, does not pay off.

The example is the most complicated SFC in element-plus, el-table.vue. The compilation takes about 100us without thread pool.

But after the thread pool is introduced the compilation time comes to 200us+.

HerringtonDarkholme

SWC or Rslint ?

TLDR: Rslitn is still better than swc.

The code changes are in swc branch and rslint branch.

swc contains a lot of dependencies regardless if they are relevant to core parsing. Such bloated dependencies place huge burden to compiling, slowing rust-analyzer to almost freezing. swc's docs and examples re scarce. The only working example is https://rustdoc.swc.rs/swc_ecma_parser/. Looking to the code is also hard. The abstraction and module organization is, well, hazy at least to the uneducated. Peeking the definition is hard, if possible, given the massive usage of macros. Alas, the macro is also the perpetrator of the sluggish compilation. :/ Using swc is not a nice journey, actually. Looking at the example above, it immediately requires several crates other than the parse. common, ast, atom, visit and blahblah. And the core impl Visitor has 200+ macro generated methods to implement without one single line of documentation. The usability is poor... And the output is large, merely importing swc pushes the binary size to 33MB.

Rslint at least has more comments and documentation than swc. It's underlying crate, rowan, also has docs. So it might be a better choice. Rslint's dependencies are also more lightweight. Rslint's source code is also simple and clear, compared to swc. Understanding Green/Red tree does require some learning. But it is fine. The binary size is 10MB after importing Rslint, one third of swc.

renovate[bot]

Dependency Dashboard

This issue provides visibility into Renovate updates and their statuses. Learn more

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

chore(deps): update dependency @napi-rs/cli to v2.9.0
chore(deps): update typescript-eslint monorepo to v5.25.0 (@typescript-eslint/eslint-plugin, @typescript-eslint/parser)

Detected dependencies

cargo

benches/Cargo.toml

criterion 0.3

compiler

glob 0.3.0

crates/cli/Cargo.toml

anyhow 1.0.57

clap 3.1.18

compiler

dom

path-clean 0.1.0

codespan-reporting 0.11.1

serde_yaml 0.8.24

crates/compiler/Cargo.toml

smallvec 1.8.0

bitflags 1.3

rustc-hash 1.1.0

serde 1.0

rslint_parser 0.3.1

phf 0.10

insta 1.14.0

lazy_static 1.4.0

crates/dom/Cargo.toml

compiler

phf 0.10

crates/ref_transform/Cargo.toml

tree-sitter 0.20.6

cc *

crates/sfc/Cargo.toml

smallvec 1.8.0

rustc-hash 1.1.0

rslint_parser 0.3.1

compiler

crates/ssr/Cargo.toml

compiler

crates/wasm/Cargo.toml

compiler

wasm-bindgen 0.2.80

wee_alloc 0.4.5

napi/Cargo.toml

napi 2.4.3

napi-derive 2.4.1

compiler

dom

napi-build 2

mimalloc 0.1

github-actions

.github/workflows/baseline.yml

actions/checkout v3

actions/setup-node v3

actions/cache v3

pnpm/action-setup v2.2.1

benchmark-action/github-action-benchmark v1

.github/workflows/benchmark.yml

actions/checkout v3

actions-rs/toolchain v1

actions/cache v3

.github/workflows/ci.yml

actions/checkout v3

actions-rs/toolchain v1

actions/cache v3

.github/workflows/coverage.yml

actions/checkout v3

codecov/codecov-action v3

.github/workflows/gh-pages.yml

actions/checkout v3

actions-rs/toolchain v1

jetli/wasm-pack-action v0.3.0

actions/cache v3

actions/setup-node v3

pnpm/action-setup v2.2.1

peaceiris/actions-gh-pages v3

npm

benches/package.json

@types/benchmark ^2.1.1

@types/glob ^7.1.4

@vue/compiler-core ^3.2.19

benchmark ^2.1.4

glob ^8.0.0

microtime ^3.0.0

ts-node 10.7.0

typescript 4.6.4

napi/npm/android-arm64/package.json

node >= 10

napi/npm/darwin-arm64/package.json

node >= 10

napi/npm/darwin-x64/package.json

node >= 10

napi/npm/freebsd-x64/package.json

node >= 10

napi/npm/linux-arm-gnueabihf/package.json

node >= 10

napi/npm/linux-arm64-gnu/package.json

node >= 10

napi/npm/linux-arm64-musl/package.json

node >= 10

napi/npm/linux-x64-gnu/package.json

node >= 10

napi/npm/linux-x64-musl/package.json

node >= 10

napi/npm/win32-arm64-msvc/package.json

node >= 10

napi/npm/win32-ia32-msvc/package.json

node >= 10

napi/npm/win32-x64-msvc/package.json

node >= 10

napi/package.json

@node-rs/helper ^1.2.1

@napi-rs/cli 2.8.0

@swc-node/register 1.5.1

@types/node 16.11.36

@vue/compiler-core 3.2.33

@vue/compiler-sfc 3.2.33

@typescript-eslint/eslint-plugin 5.23.0

@typescript-eslint/parser 5.23.0

ava 4.2.0

benny 3.7.1

chalk 5.0.1

eslint 8.15.0

eslint-config-prettier 8.5.0

eslint-plugin-import 2.26.0

eslint-plugin-prettier 4.0.0

eslint-plugin-sonarjs 0.13.0

npm-run-all 4.1.5

prettier 2.6.2

typescript 4.6.4

node >= 10

playground/package.json

vue 3.2.33

@babel/types 7.17.12

@vitejs/plugin-vue 2.3.3

typescript 4.6.4

vite 2.9.9

vue-tsc 0.34.15

rusty-vue-compiler file:../crates/wasm/pkg/

Check this box to trigger a request for Renovate to run again on this repository

HerringtonDarkholme

Should RE used in the crate?

Hard to migrate but possible

compiler-core/src/transforms/vFor.ts|304 col 15| const forAliasRE = /([\s\S]*?)\s+(?:in|of)\s+([\s\S]*)/
compiler-core/src/transforms/vFor.ts|307 col 18| const forIteratorRE = /,([^,\}\]]*)(?:,([^,\}\]]*))?$/
compiler-core/src/transforms/vOn.ts|19 col 12| const fnExpRE = /^\s*([\w$_]+|$[^)]*?$)\s*=>|^\s*function(?:\s+[\w$]+)?\s*\(/
compiler-core/src/utils.ts|79 col 17| const whitespaceRE = /\s+[.[]\s*|\s*[.[]\s+/g
shared/src/normalizeProp.ts|30 col 20| const listDelimiterRE = /;(?![^(]*\))/g // ; followed by no single )