For anyone interested in similar tools, Microsoft recently published a protocol[1] for this type of service, and an implementation[2] of it for VSCode+NodeJS.
Swift also has a library[3] called SourceKit which does the same thing for Swift – it can run as an out-of-process daemon, and people have written integrations for Atom, Emacs, etc. to get autocomplete, syntax highlighting, def links, etc.
Thanks, I totally missed that. That's pretty great – I thought it was a neat idea from MS, but I was worried it would get any traction. Glad to have been wrong!
I'm imaging more along the lines of serializing editor state with each key press, encoding it, sending it to another process, having that process unserialize it, lex it, encode all of the cascading token changes and information about those tokens back into JSON, sending it back to the editor process which then has to decode the JSON and apply all of the token information to the in-memory representation as real overhead.
It's great that Rust will decode and encode the JSON quickly (I've been keeping an eye on Rust's MSVC progress before I consider using it for a real-world project), but the project I work on cares about optimizations on the level of in-memory data layout and memory allocations. Moving syntax highlighting into a pluggable system would be a huge performance regression.
> I'm imaging more along the lines of serializing editor state with each key press, encoding it, sending it to another process, having that process unserialize it, lex it, encode all of the cascading token changes and information about those tokens back into JSON, sending it back to the editor process which then has to decode the JSON and apply all of the token information to the in-memory representation as real overhead.
The latency of both Unix IPC calls and serialization/deserialization of small amounts of JSON should be measured in single- to double-digit microseconds. A frame at 60fps is 16,000 microseconds, so if this is causing even a single dropped frame, something is seriously wrong.
The lexing itself might or might not be a performance worry, depending on the architecture of the whole system, but that's no different from the lexing built into the editor (except perhaps in that people writing editors tend to care more about latency)...
(Well, this seems academic, since according to other comments RLS doesn't aim to put syntax highlighting out of process.)
There was a talk at RustConf by Raph Levien about his editor, Xi. It almost entirely uses JSON-RPC to communicate through a distributed architecture, and next-level speed is one of his design goals: https://confreaks.tv/videos/rustconf2016-a-modern-editor-bui...
> I've been keeping an eye on Rust's MSVC progress before I consider using it for a real-world project
Ah! What things are you looking for in its maturity? In some ways, it's actually the most mature backend; stack probes work with it, for example. Windows isn't my personal area of expertise though.
I've been following https://github.com/rust-lang/rfcs/issues/1061 for a while. It seems that high-level pieces are there, but that certain elements useful in day-to-day software engineering still need some more time.
If you have the time, you should really consider helping to improve the Rust experience on Windows. There's a huge shortage of Windows developers in the Rust community.
Unfortunately I am already overcommitted with open source projects. For now I am content to wait and see how it all plays out. It will be great to have another language where Windows support is really first-rate.
Probably the biggest drawback of the MSVC backend is that its debugging symbols only have line numbers, no variable/type information. The rest is mostly Windows-specific issues like manifests and subsystems.
Actually, the PDB debug info has variable and type information and a lot of it actually works. There's just a few minor issues here and there with stuff like fat pointers.
It might have been due to either https://github.com/rust-lang/rust/pull/31319 or an LLVM upgrade. But stuff like looking at the values of basic integer types, the fields of structs, tuple fields, thin pointers, and even many enums works fine. You just have to make sure you have debug info enabled.
Keep in mind std in the Rust distribution was built without debuginfo, so if you step into functions from that, things don't work too well.
MSVC is commonly used to refer to Visual Studio, more specifically the C++ compiler that's part of it (originally "Microsoft Visual C++"), which would explain the parent confusing the two.
Perhaps I'm not understanding you, but how does the speed of syntax highlighting (or, more generally, the editor) affect your project's performance? I'm trying and failing to imagine some sort of realtime system controlled by someone coding away.
(Also, you wouldn't necessarily need to transmit the whole editor state--just changes).
Syntax highlighting is (usually) done by the editor, not the server in the LS protocol. The changes sent from editor to server are pretty minimal, and so the whole thing is pretty quick. The overhead, never caused us any problems.
I was referring to the paragraph about Swift and Source kit. To my knowledge, the majority of editors do not do syntax highlighting outside of the main editor process.
Swift also has a library[3] called SourceKit which does the same thing for Swift – it can run as an out-of-process daemon, and people have written integrations for Atom, Emacs, etc. to get autocomplete, syntax highlighting, def links, etc.
[1]https://github.com/Microsoft/language-server-protocol [2]https://github.com/Microsoft/vscode-languageserver-node [3]https://github.com/apple/swift/tree/master/tools/SourceKit