sourcegraph/syntect_server

This is an HTTP server that exposes the Rust Syntect syntax highlighting library for use...

This is an HTTP server that exposes the Rust Syntect -> Syntax highlighting of code

🚨 Note: this repo has been archived and its contents are now in the sourcegraph/sourcegraph repo.

Syntect Server

This is an HTTP server that exposes the Rust Syntect syntax highlighting library for use by other services. Send it some code, and it'll send you syntax-highlighted code in response.

Technologies:

  • Syntect -> Syntax highlighting of code.
  • Rocket.rs -> Web framework.
  • Serde -> JSON serialization / deserialization .
  • Rayon -> data parallelism for SyntaxSet across Rocket server threads.
  • lazy_static -> lazily evaluated static ThemeSet (like a global).

Usage

You can then e.g. GET http://localhost:9238/health or #404 to confirm it is working.

API

  • POST to / with Content-Type: application/json. The following fields are required:
    • filepath string, e.g. the/file.go or file.go or Dockerfile, see "Supported file extensions" section below.
    • theme string, e.g. Solarized (dark), see "Embedded themes" section below.
    • code string, i.e. the literal code to highlight.
  • The response is a JSON object of either:
    • A successful response (data field):
      • data string with syntax highlighted response. The input code string is properly escaped and as such can be directly rendered in the browser safely.
      • plaintext boolean indicating whether a syntax could not be found for the file and instead it was rendered as plain text.
    • An error response (error field), one of:
      • {"error": "invalid theme", "code": "invalid_theme"}
      • {"error": "resource not found", "code": "resource_not_found"}
  • GET to /health to receive an OK health check response / ensure the service is alive.

Client

gosyntect is a Go package + CLI program to make requests against syntect_server.

Configuration

By default on startup, syntect_server will list all features (themes + file types) it supports. This can be disabled by setting QUIET=true in the environment.

Development

  1. Install Rust nightly.
  2. git clone this repository anywhere on your filesystem.
  3. Use cargo run to download dependencies + compile + run the server.

Building

Invoke cargo build --release and an optimized binary will be built (e.g. to ./target/release/syntect_server).

Building docker image

./build.sh will build your current repository checkout into a final Docker image.

Publishing docker image

Run ./publish.sh after merging your changes.

Updating Sourcegraph

Once published, the image version will need to be updated in the following locations to make Sourcegraph use it:

  • sourcegraph/sourcegraph > dev/syntect_server.sh
  • sourcegraph/sourcegraph > docker-images/syntax-highlighter/build.sh
  • sourcegraph/sourcegraph > cmd/server/Dockerfile
  • sourcegraph/sourcegraph > sg.config.yaml

Additionally, it's worth doing a search for other uses in case this list is stale.

Code hygiene

  • Use cargo fmt or an editor extension to format code.

Adding themes

  • Copy a .tmTheme file anywhere under ./syntect/testdata (make a new dir if needed) in our fork.
  • cd syntect && make assets
  • In this repo, cargo update -p syntect.
  • Build a new binary.

Adding languages:

1) Find an open-source .tmLanguage or .sublime-syntax file and send a PR to our package registry

https://github.com/slimsag/Packages is the package registry we use which holds all of the syntax definitions we use in syntect_server and Sourcegraph. Send a PR there by following these steps

2) Update our temporary fork of syntect

We use a temporary fork of syntect as a hack to get our Packages registry into the binary. Update it by creating a PR with two commits like:

  • https://github.com/slimsag/syntect/commit/9976d2095e49fd91607026364466cd7b389b938e
  • https://github.com/slimsag/syntect/commit/1182dd3bd7c82b6655d8466c9896a1e4f458c71e

3) Update syntect_server to use the new version of syntect

Send a PR to this repository with the result of running cargo update -p syntect.

4) Publish a new image, use it in Sourcegraph

Run ./publish.sh to build and release a new image of syntect_server, and then send a PR to the main Sourcegraph repository like this. Once merged, it will automatically rollout to Sourcegraph.com and go in the next Sourcegraph release.

Embedded themes:

  • InspiredGitHub
  • Monokai
  • Solarized (dark)
  • Solarized (light)
  • Sourcegraph
  • Sourcegraph (light)
  • TypeScript
  • TypeScriptReact
  • Visual Studio
  • Visual Studio Dark
  • base16-eighties.dark
  • base16-mocha.dark
  • base16-ocean.dark
  • base16-ocean.light

Supported file extensions:

  • Plain Text (txt)
  • ASP (asa)
  • HTML (ASP) (asp)
  • ASP vb.NET (vb)
  • HTML (ASP.net) (aspx, ascx, master)
  • ActionScript (as)
  • AppleScript (applescript, script editor)
  • Batch File (bat, cmd)
  • NAnt Build File (build)
  • C# (cs, csx)
  • C++ (cpp, cc, cp, cxx, c++, C, h, hh, hpp, hxx, h++, inl, ipp)
  • C (c, h)
  • CMake Cache (CMakeCache.txt)
  • CMake Listfile (CMakeLists.txt, cmake)
  • ACUCOBOL (``)
  • COBOL (cbl, cpy, cob, dds, ss, wks, pco)
  • OpenCOBOL (``)
  • jcl (jcl)
  • CSS (css, css.erb, css.liquid)
  • Cap’n Proto (capnp)
  • Cg (cg)
  • Clojure (clj, cljc, cljs, cljx, edn)
  • Coq (v)
  • Crontab (crontab)
  • CUDA C++ (cu, cuh)
  • D (d, di)
  • DMD Output (``)
  • Dart Doc Comments (``)
  • Dart (dart)
  • Diff (diff, patch)
  • Dockerfile (Dockerfile)
  • DM (dm, dme)
  • Elixir (EEx) (ex.eex, exs.eex)
  • Elixir (ex, exs)
  • HTML (EEx) (html.eex, html.leex)
  • Regular Expressions (Elixir) (ex.re)
  • SQL (Elixir) (ex.sql)
  • Elm (elm)
  • Erlang (erl, hrl, Emakefile, emakefile, escript)
  • HTML (Erlang) (yaws)
  • Solidity (sol)
  • Vyper (vy)
  • F Sharp (fs)
  • friendly interactive shell (fish) (fish)
  • Forth (frt, fs)
  • ESSL (essl, f.essl, v.essl, _v.essl, _f.essl, _vs.essl, _fs.essl)
  • GLSL (vs, fs, gs, vsh, fsh, gsh, vshader, fshader, gshader, vert, frag, geom, tesc, tese, comp, glsl)
  • Git Attributes (attributes, gitattributes, .gitattributes)
  • Git Commit (COMMIT_EDITMSG, MERGE_MSG, TAG_EDITMSG)
  • Git Common (``)
  • Git Config (gitconfig, .gitconfig, .gitmodules)
  • Git Ignore (exclude, gitignore, .gitignore)
  • Git Link (.git)
  • Git Log (gitlog)
  • Git Mailmap (.mailmap, mailmap)
  • Git Rebase Todo (git-rebase-todo)
  • Go (go)
  • GraphQL (graphql, graphqls, gql, graphcool)
  • Graphviz (DOT) (dot, DOT, gv)
  • Groovy (groovy, gvy, gradle, Jenkinsfile)
  • HLSL (fx, fxh, hlsl, hlsli, usf)
  • HTML (html, htm, shtml, xhtml)
  • Haskell (hs)
  • Literate Haskell (lhs)
  • INI (cfg, conf, ini, lng, url, .buckconfig, .flowconfig, .hgrc)
  • REG (reg)
  • JSON (json, sublime-settings, sublime-menu, sublime-keymap, sublime-mousemap, sublime-theme, sublime-build, sublime-project, sublime-completions, sublime-commands, sublime-macro, sublime-color-scheme, ipynb, Pipfile.lock)
  • Java Server Page (JSP) (jsp)
  • Java (java, bsh)
  • Javadoc (``)
  • Java Properties (properties)
  • JS Custom - Default (js, htc)
  • JS Custom - React (js, jsx)
  • Regular Expressions (Javascript) (``)
  • JS Custom (Embedded) (``)
  • Julia (jl)
  • Kotlin (kt, kts)
  • LESS (less)
  • BibTeX (bib)
  • LaTeX Log (``)
  • LaTeX (tex, ltx)
  • TeX (sty, cls)
  • Lisp (lisp, cl, clisp, l, mud, el, scm, ss, lsp, fasl)
  • Lua (lua)
  • MSBuild (proj, targets, msbuild, csproj, vbproj, fsproj, vcxproj)
  • Make Output (``)
  • Makefile (make, GNUmakefile, makefile, Makefile, makefile.am, Makefile.am, makefile.in, Makefile.in, OCamlMakefile, mak, mk)
  • Man (man)
  • Markdown (md, mdown, markdown, markdn)
  • MultiMarkdown (``)
  • MATLAB (matlab)
  • Maven POM (pom.xml)
  • Mediawiki (mediawiki, wikipedia, wiki)
  • Move (move)
  • Ninja (ninja)
  • Nix (nix)
  • OCaml (ml, mli)
  • OCamllex (mll)
  • OCamlyacc (mly)
  • camlp4 (``)
  • Objective-C++ (mm, M, h)
  • Objective-C (m, h)
  • PHP Source (``)
  • PHP (php, php3, php4, php5, php7, phps, phpt, phtml)
  • Regular Expressions (PHP) (``)
  • Pascal (pas, p, dpr)
  • Perl (pl, pc, pm, pmc, pod, t)
  • Property List (XML) (``)
  • Postscript (ps, eps)
  • PowerShell (ps1, psm1, psd1)
  • Protocol Buffer (proto)
  • Puppet (pp, epp)
  • Python (py, py3, pyw, pyi, pyx, pyx.in, pxd, pxd.in, pxi, pxi.in, rpy, cpy, SConstruct, Sconstruct, sconstruct, SConscript, pyst, pyst-include, gyp, gypi, Snakefile, vpy, wscript, bazel, bzl)
  • Regular Expressions (Python) (``)
  • R Console (``)
  • R (R, r, Rprofile)
  • Rd (R Documentation) (rd)
  • HTML (Rails) (rails, rhtml, erb, html.erb)
  • JavaScript (Rails) (js.erb)
  • Ruby Haml (haml)
  • Ruby on Rails (rxml, builder)
  • SQL (Rails) (erbsql, sql.erb)
  • Regular Expression (re)
  • reStructuredText (rst, rest)
  • Ruby (rb, Appfile, Appraisals, Berksfile, Brewfile, capfile, cgi, Cheffile, config.ru, Deliverfile, Fastfile, fcgi, Gemfile, gemspec, Guardfile, irbrc, jbuilder, Podfile, podspec, prawn, rabl, rake, Rakefile, Rantfile, rbx, rjs, ruby.rail, Scanfile, simplecov, Snapfile, thor, Thorfile, Vagrantfile)
  • Cargo Build Results (``)
  • Rust Enhanced (rs)
  • Sass (sass, scss)
  • SQL (sql, ddl, dml)
  • Scala (scala, sbt, sc)
  • Bourne Again Shell (bash) (sh, bash, zsh, ash, .bash_aliases, .bash_completions, .bash_functions, .bash_login, .bash_logout, .bash_profile, .bash_variables, .bashrc, .profile, .textmate_init, .zlogin, .zlogout, .zprofile, .zshenv, .zshrc, PKGBUILD, .ebuild, .eclass)
  • Shell-Unix-Generic (``)
  • commands-builtin-shell-bash (``)
  • Smalltalk (st)
  • Smarty (tpl)
  • Starlark (build_defs, BUILD.in, BUILD, WORKSPACE, bzl, sky, star, BUILD.bazel, WORKSPACE, WORKSPACE.bazel)
  • Stylus (styl, stylus)
  • Swift (swift)
  • HTML (Tcl) (adp)
  • Tcl (tcl)
  • TOML (toml)
  • Terraform (tf, tfvars, hcl)
  • Textile (textile)
  • Thrift (thrift, frugal)
  • TypeScript (ts)
  • TypeScriptReact (tsx)
  • VimL (vim, .vimrc, .gvimrc)
  • Vue Component (vue)
  • XML (xml, xsd, xslt, tld, dtml, rng, rss, opml, svg)
  • YAML (yaml, yml, sublime-syntax)
  • Zig (zig)
Issues

Collection of the latest Issues

slimsag

slimsag

Comment Icon1

Today we use babel-sublime for jsx support (see https://github.com/sourcegraph/syntect_server/issues/3) but babel-sublime is not well-maintained anymore, and upon updating sublimehq/Packages it appears it is much more incompatible with the base Packages now and e.g. breaks JS embedded in PHP/HTML.

https://github.com/sublimehq/Packages/issues/860 makes note of a new project, https://github.com/Thom1729/Sublime-JS-Custom, which appears to add JSX on top of the base JavaScript syntax and may be better in the near term.

slimsag

slimsag

Comment Icon1

We should move all env vars from the Dockerfile into the binary itself so that consumers using just the binary (e.g. Sourcegraph) do not have to separately maintain their own list of env vars to configure Rocket.

Should be as simple as setting them if not already set.

slimsag

slimsag

Comment Icon3

The ./syntect folder is a copied / slightly modified version of upstream syntect. I should figure out the correct way to send these changes upstream.

EDIT: there is now much more changes in terms of Packages etc. than just the above that we would want to send upstream^

Information - Updated Oct 19, 2021

Stars: 84
Forks: 12
Issues: 5

When you need to pretend you are a REST API, but you're really not

Facade is a Rust-based HTTP transformation layer to seamlessly convert REST calls into GraphQL calls for piecemeal API migrations

When you need to pretend you are a REST API, but you're really not

Rust helpers for serving HTTP GET and HEAD responses with

This crate supplies two ways to respond to HTTP GET and HEAD requests:

Rust helpers for serving HTTP GET and HEAD responses with

When you need to pretend you are a REST API, but you're really not

Vostok is a Rust-Based HTTP Transformation Layer To Seamlessly Convert REST Calls Into GraphQL Calls For Piecemeal API Migrations

When you need to pretend you are a REST API, but you're really not

Eve-rs is a middleware-based rust http framework which is inspired from ExpressJS framework

Let's create a demo application using eve-rs, using an intuative approach to get a better understanding of the framework

Eve-rs is a middleware-based rust http framework which is inspired from ExpressJS framework
Http

375

HTTP mocking for Rust!

Before upgrading, make sure to check out the rustfmt as a general code style

HTTP mocking for Rust!

Rust Urbit HTTP API

This library wraps the Urbit ship http interface exposing it as an easy-to-use Rust crate

Rust Urbit HTTP API

HTTP Server with Rust

Rust http server from scratch

HTTP Server with Rust

ehttp: a minimal Rust HTTP client for both native and WASM

If you want to do HTTP requests and are targetting both native and web (WASM), then this is the crate for you!

ehttp: a minimal Rust HTTP client for both native and WASM

Here lies my attempt to create a multi thread webserver based off of Rust and...

Here lies my attempt to create a multi thread webserver based off of Rust and using HTTP and TCP networking protocols :)

Here lies my attempt to create a multi thread webserver based off of Rust and...

rust-http-server

Multithread HTTP Server developed with Rust

rust-http-server
Facebook Instagram Twitter GitHub Dribbble
Privacy