Editor plugins

[kuon]

I think that a language success is very dependent on the tooling. And having good editor support is step one.

This is not a rhai issue per se, but I think it is good to have a place to track the progress of the ecosystem, it can also serve as reference when people search for it.

To my knowledge, the main editors used nowadays are vim, vscode, emacs and sublime text, so it would be nice to have plugins for those.

I started a vim plugin here https://github.com/kuon/rhai.vim, but this is my first syntax plugin and there are a few features I have not implemented yet.

[schungx]

I have been wanting a plugin for Visual Studio Code as well...

Actually a good way to do this is to build a language server which can work with multiple IDE's. Short of that, a TextMate grammar may be better than nothing...

I'm wondering - does your vim plugin do anything other than syntax highlighting? There is an online playground available with syntax highlighting that you can look at.

It leverages internal Rhai data structures to hook into the tokenizer.

[kuon]

Yeah, a language server would be require for more advanced features. And yes, my plugin only does basic syntax highlighting for now. -- Nicolas Goy

[schungx]

@kuon how is your vim plugin coming along? How does it work out?

Would you like to publish it in the official org itself? I can make a rhai-plugins repo...

[kuon]

I didn't really do any more work on it, it provides really minimal features for syntax highlight.

At present, I am working on another project, and I don't have time to maintain it. But if someone wants to take over, you can fork it and I'll redirect my repo to yours. But to play nice with plugin manager, it's better in its own repository.

[schungx]

Ah, so something like rhai-vim-plugin

[schungx]

@kuon Just wondering if you still have your vim plugin for Rhai?

Rhai now has official VS Code syntax highlighting: https://marketplace.visualstudio.com/items?itemName=rhaiscript.vscode-rhai

Would be nice to add a vim-rhai to it as well.

Would you be interested to transfer it to the rhaiscript org?

[tamasfe]

Hi @schungx, I'm looking for projects for my university thesis for this semester and writing an LSP server for rhai came up as an idea. It would be a completely personal project that would be eventually open-sourced at the end.

I have a few concerns/questions before I commit to it:

• Has anyone started working on something like this? I have to work completely alone before it is open-sourced, so I cannot contribute unfortunately, and I'd rather not have 2 competing implementations.
• Most likely I'd have to write my own parser as well, how "hard" is parsing rhai? (especially since custom keywords can be defined) Is there a syntax specification somewhere?

What I currently have in mind:

• Since rhai is very context-dependent, I'd rather implement the LSP as a library, where people can provide their own Engine that can be queried for modules and custom keywords
• then figure out a way to create a context for the LSP so that these could be mapped to files (or even get/construct the source code from the user-provided engine).

I haven't started digging deeper, I thought I'd ask here first. Do you think this would be feasible, or do you see any major blocker why I wouldn't be able to implement something like this in the following 3-4 months?

[schungx]

This, now, would be wonderful. I've always wanted an LSP server but never learned how to write one.

Has anyone started working on something like this?

Not really...

Most likely I'd have to write my own parser as well, how "hard" is parsing rhai?

It is not hard at all. You can simply "borrow" the parser from the Rhai project itself. It builds an entire AST.

If you need any special functionalities, let me know, and I'll see if I can work with you to add it.

I'd recommend leaving custom syntax out of it because it is user-defined and you won't be able to handle it in an separate LSP server anyway.

implement the LSP as a library, where people can provide their own Engine that can be queried for modules and custom keywords

This sounds interesting, meaning that it can handle user-defined functions and custom syntax. However, I'd say a "standard" build that is added to the VS Code Rhai extension, for example, would work great already.

any major blocker why I wouldn't be able to implement something like this in the following 3-4 months?

Nothing that I can think of... but I have never written an LSP before so I won't really know!

[tamasfe]

It is not hard at all. You can simply "borrow" the parser from the Rhai project itself. It builds an entire AST.

This was also my first reaction, and it makes perfect sense technically, but the thesis needs enough original work done by me, so I probably cannot reuse it. I'll see if I can work something out.

If you need any special functionalities, let me know, and I'll see if I can work with you to add it.

Thank you, I'll dig deeper this week.

However, I'd say a "standard" build that is added to the VS Code Rhai extension, for example, would work great already.

How useful would this be though? From my understanding Rhai is perfect for embedding domain-specific scripts into existing applications, and a generic LSP would simply would not know about all the things that automagically appear at runtime.

My idea was to create something embeddable with a similar mindset, so that people who use Rhai in their applications would be able to provide their own LSP with minimal effort as well.

Another issue with a generic LSP would be module resolution since there is no file structure specified (unless I missed it), and I'd rather not specify any in the LSP either (thus people won't rely on it). I see that there is a default file resolver, this could work as a start. Is there support planned for multiple resolvers at the same time?

Nothing that I can think of... but I have never written an LSP before so I won't really know!

Awesome, thank you for the approval. No promises, but this is definitely my top choice right now that I'll explore further.

[schungx]

This was also my first reaction, and it makes perfect sense technically, but the thesis needs enough original work done by me, so I probably cannot reuse it. I'll see if I can work something out.

Rhai uses a hand-crafted recursive-descent parser which is actually quite simple to understand. I'd say the parser is actually a very small portion of an LSP server, you may be able to get away with it.

On the other hand, it would be interesting to write a grammar for Rhai and put it into an LALR generator or something... or some PEG generator... That would be original!

How useful would this be though?

That's true... but at least we have jump-to-definition for variables and functions etc. Probably no type checking and function arguments checking... However, I suppose it would still be useful at least...

[Hodkinson]

I'd like to create an IntelliJ IDEA plugin for supporting (initially) Rhai syntax highlighting. Is there any grammar definition for Rhai that I would be able to convert into an antlr grammar?

Edit: I see from the above, that it might be defined only in the code, so I guess the starting point would be to look at https://github.com/rhaiscript/rhai/blob/master/src/parse.rs and convert this into a grammar.

[schungx]

I'd suggest you start with a standard-issue JavaScript grammar, as Rhai is very similar to JavaScript in syntax.

Just a few things to remember:

1. There can be number digit separators _ but they cannot be in front of the number or the decimal point...
2. 0o and 0b numbers
3. Names like _123, _0 etc. are not valid identifiers while in JavaScript they are.
4. JavaScript has no statement expressions, so you need to consider them
5. Semicolon requirements for JavaScript are different

I built the vscode and textmate syntax highlighting by starting off with the JavaScript files and then simply deleted the unnecessary stuff.

[tamasfe]

I've made progress with the LSP and submitted my BSc thesis, I uploaded it here. It was mostly written out of necessity so it might be vague/incomplete at parts and probably even inaccurate, but it should be enough to skim through and get an idea of what I've been up to if you're interested.

I don't want to make repo completely public yet as it needs more documentation and some refactoring to which I'll get to at some point in January, but I can invite anyone as a collaborator until then.

[schungx]

Wow, this looks wonderful and a lot of work! Especially on the LSP implementation, rewritten parser, fault recovery, and syntax highlighting.

I'm wondering if it is possible at all to eventually merge your parser implementation into Rhai, replacing the existing implementation. This way, we have one uniform way to parse for both the compiler as well as the language server. I suppose the CST would need to be new (as Rhai's AST doesn't really keep a lot of source text info), but the AST can be regenerated from the CST.

[tamasfe]

Thanks!

I'm wondering if it is possible at all to eventually merge your parser implementation into Rhai, replacing the existing implementation.

I wouldn't do this for the following reasons:

• I don't think my parser is superior in any way, and is not yet widely tested
• it also can't yet handle user-defined syntax and tokenization
• I don't think there would be any benefit when the goal is to execute the code as fast as possible, skipping comments and bailing on errors is the way to go

So I believe the two parsers serve different purposes, and uniformity can be mostly achieved with common compliance tests. Currently I use the scripts in the repository for this, but they will need to be extended as there are some language constructs they don't cover.

Even if my parser turns to be somewhat better, I'm sure the benefits will only be marginal and wouldn't warrant a complete rewrite.

Using a common tokenizer on the other hand should be possible, and I'm definitely for it as Logos is awesome.

[tamasfe]

I decided to actually open up the lsp repository in its current state so that you can look at it and discussions can be done there.

It's rather messy and I don't have too much time right now, but there is little point in keeping it hidden.

@schungx I also invited you as a collaborator.

[schungx]

Thanks! Will be peeking into it this weekend...

BTW, any interest to move it into rhaiscript/lsp? Or would you prefer to get it into a stable form before putting it under the org?

[tamasfe]

BTW, any interest to move it into rhaiscript/lsp? Or would you prefer to get it into a stable form before putting it under the org?

Yes, it was my intention from the beginning. Although I'd definitely like to get your approval first to see if the content/quality etc. aligns with the rest of Rhai. But it's up to you really, as soon as you tell me, I can start a transfer of the repo (actually I'm not sure how to do this, as I haven't transferred repositories before on github).

[schungx]

Well, I guess there is a button somewhere in Github to transfer a repo to an org... but I haven't done it myself... See if you can do it yourself... if you need authentication or something, let me know and I'll figure it out.

EDIT: There IS such a button...

It depends on you when you'd want to do the move. Some people I see wait until it is at least in a usable form before putting a repo under an org, while others will start off right in the org.

Doing development under the org itself has the benefit of making more people aware of the work, so they're more likely to help out than the chance that they discover the repo under your name.

[tamasfe]

If it's up to me, then I'd definitely transfer it, then keep working on it there.

According to github docs:

To transfer repositories to an organization, you must have repository creation permissions in the receiving organization.

I wouldn't necessarily want to get any permissions in the org, see if you have the rights in rhai-lspto transfer it.

[schungx]

Done. I just sent you a member invitation.

[schungx]

You may want to decide whether to call it rhaiscript/rhai-lsp or just rhaiscript/lsp...

[tamasfe]

I've transferred it, but I seem to have lost rights to it in the process.

There was no option to rename it during transfer, but we can rename it regardless if you wish (I have no opinion on it to be honest).

[schungx]

I've transferred it, but I seem to have lost rights to it in the process.

Fixed.

[schungx]

There was no option to rename it during transfer, but we can rename it regardless if you wish (I have no opinion on it to be honest).

It is renamed to rhaiscript/lsp which is better naming.