1. Introduction
Language Server Protocol (LSP) is the de facto standard for providing rich editor experience these days. Given its popularity, surprisingly little content can be found on the internet about how to implement your own language server from scratch. Even when the material does exist, it either only talks about the specification itself with no implementation, or it implements a very basic server that provides almost no useful functionality. This series of posts is going to be my attempt to fill this void. In this series of posts, I will implement a simple language server for Azure Resource Manager templates. I will try to utilize maximum number of LSP features that make sense for this case. This first post is going to set some context about what we will be doing and what technologies we will be using. Full source code of this server is available in this repository.
2. Brief Introduction to Language Server Protocol (LSP)
Roughly, LSP is a protocol to mediate the communication between an editor and a language server, over JSON-RPC. A language server implements the language server protocol to provide rich editing experience for some set of related files, such as a C# project, or a Ruby project. By implementing the language server protocol, a single LSP server can provide rich editing to all editors that support LSP and an editor that supports LSP can support all language servers.
LSP describes the set of messages clients and servers are supposed to exchange, along with the data required to be in those messages, to provide various features. LSP supports a wide range of features, some of which (in layman terms) are:
- Hovering on symbols
- Providing auto-complete suggestions
- Jumping to definition
- Finding references
- … and many more
2.1. An Exchange between Client and Server
Let's see an example of an exchange between a client and a server to implement the Hover functionality.
In this figure above, we can see an interaction between the language client and the language server. As the user brings their cursor over a variable name, named noOfCols, the language client will construct a HoverParams struct, filling out the relevant information about where exactly in the file the cursor is pointing. It will then call the method textDocument/Hover on the language client through JSON-RPC, passing HoverParams struct as argument. The language server takes that position and maps it to a token that is currently under the cursor. It then looks up the information about that token and sends a HoverResult struct containing the hover information, like the type of the variable and the documentation about what that variable refers to. The language client can choose to display this information however it is configured, like as a tooltip or in the echo area. This is more or less how all features in LSP work.
In most cases, you will not have to implement the specifics of the protocol by yourself. Official website for LSP contains a list of SDKs for a variety of languages here. You can use the SDK for your preferred language and let the SDK handle the communication with the language client. The library will expose clean functions/bindings relating to all functionalities offered by LSP that you will implement to provide the functionalities for your particular technology, in this case ARM templates. I will be using C# and LSP library from OmniSharp project to implement the language server. In order to follow along with this post, you can clone the Armls repository and checkout commit with ID b794cd3. armls directory contains the code for the language server, while armls-ext contains the code and VSIX for a VS Code extension that can utilize Armls. In order to use the extension, you will have to install the VSIX as explained here and set the path to compiled armls binary.
3. Bird's Eye View of Armls
Armls comprises of 3 components primarily:
- C#-LSP library to handle interactions with language client
- TreeSitter to parse ARM template JSON
- Domain knowledge of ARM templates to implement LSP functionalities
The choice of language here is somewhat arbitrary and mostly dependent on what you are comfortable with. LSP SDKs are available for a large number of languages and they all provide the same functionality, idiomatic to the patterns in that language. You are relatively constrained in the choice of a parser though. It is critical that whatever parser you use (or write yourself) is fault tolerant. As the user writes code and modifies the file, the syntax tree is bound to have errors in it. Your parser must be comfortable parsing faulty and incomplete code to be able to highlight errors. Lastly, you will need to domain knowledge of the technology you are providing rich editing experience for.
4. C# Project
Start by creating a blank C# project for Armls by opening your terminal and running something like:
dotnet new console --name armls
Edit armls.csproj to add dependencies for C#-LSP library and Microsoft's popular dependency injection library which we will use to cleanly inject dependencies in our handlers.
<ItemGroup>
<PackageReference Include="Microsoft.Extensions.DependencyInjection" Version="10.0.0-preview.3.25171.5" />
<PackageReference Include="OmniSharp.Extensions.LanguageServer" Version="0.19.9" />
</ItemGroup>5. Minimum Viable Server
Add the following to your Program.cs:
public static void Main()
{
MainAsync().Wait();
}
private static async Task MainAsync()
{
var server = await LanguageServer.From(options =>
options
.WithInput(Console.OpenStandardInput())
.WithOutput(Console.OpenStandardOutput())
);
await server.WaitForExit;
}
This code simply creates a language server using the APIs from C#-LSP and connects the Standard IO of console application as IO streams for the language server. Believe it or not, you just created your own language server. This is all it takes to create a language server that does absolutely nothing. C#-LSP exposes various base classes for Handlers that provide functionalities for rich editing. There's TextDocumentSyncHandlerBase for handling the file change notifications coming from language client. There's CompletionHandlerBase to provide completion candidates. And so on.
6. Managing Buffers
A buffer, for this discussion, roughly refers to a file, either open in the editor or on the file system. Language servers cannot only rely on the files on the file system because the servers need to provide diagnostics, like errors and warnings, for changes that haven't been saved yet. Hence LSP clients convey all changes made to a file to the server. We need to cache these changes in the server to be able to run analysis on them. To that end, we will create a class called BufferManager that is responsible to carry the latest state of all buffers.
public class BufferManager
{
private readonly IDictionary<string, Buffer> buffers;
public BufferManager()
{
buffers = new ConcurrentDictionary<string, Buffer>();
}
public void Add(DocumentUri uri, Buffer buf)
{
Add(uri.GetFileSystemPath(), buf);
}
public void Add(string path, Buffer buf)
{
buffers[path] = buf;
}
public IReadOnlyDictionary<string, Buffer> GetBuffers()
{
return buffers.AsReadOnly();
}
}
BufferManager contains a simple dictionary that maps a path to an instance of Buffer class. Buffer is a very simple class that just has the text of a buffer, for now. Overtime, it will grow to cache all information related to a buffer, like the concrete syntax tree of the parsed text.
public class Buffer
{
public string Text;
public Buffer(string text)
{
Text = text;
}
}7. Text Document Synchronization
In order to sync with all the text changes happening inside the editor, we need to provide an implementation of ITextDocumentSyncHandler. The interface provides various callbacks received from the editor about what the user is doing.
public interface ITextDocumentSyncHandler
{
public abstract TextDocumentAttributes GetTextDocumentAttributes(DocumentUri uri);
public abstract Task<Unit> Handle(DidOpenTextDocumentParams request, CancellationToken cancellationToken);
public abstract Task<Unit> Handle(DidChangeTextDocumentParams request, CancellationToken cancellationToken);
public abstract Task<Unit> Handle(DidSaveTextDocumentParams request, CancellationToken cancellationToken);
public abstract Task<Unit> Handle(DidCloseTextDocumentParams request, CancellationToken cancellationToken);
}
You can extend a base implementation provided by C#-LSP which handles some boilerplate, named TextDocumentSyncHandlerBase.
public class TextDocumentSyncHandler : TextDocumentSyncHandlerBase
{
private readonly BufferManager bufManager;
private readonly ILanguageServerFacade languageServer;
public TextDocumentSyncHandler(BufferManager manager,
ILanguageServerFacade languageServer)
{
bufManager = manager;
this.languageServer = languageServer;
}
// ...
}
To start with, the sync handler will need access to the BufferManager to cache all the changes we will receive from language client. We will also get an instance of ILanguageServerFacade which, among other things, is the interface to communicate with the language client.
public class TextDocumentSyncHandler : TextDocumentSyncHandlerBase
{
// ...
public override TextDocumentAttributes GetTextDocumentAttributes(DocumentUri uri)
{
// Language ID of json and jsonc are just their names
// which are also the extensions of the files.
return new TextDocumentAttributes(uri, Path.GetExtension(uri.Path));
}
protected override TextDocumentSyncRegistrationOptions CreateRegistrationOptions(
TextSynchronizationCapability capability,
ClientCapabilities clientCapabilities
)
{
return new TextDocumentSyncRegistrationOptions(TextDocumentSyncKind.Full);
}
private Buffer.Buffer CreateBuffer(string text)
{
return new Buffer(text);
}
// ...
}
We then implement GetTextDocumentAttributes which is supposed to provide some information about the file. We just provide the URI to the document as well as the language ID. We override CreateRegistrationOptions where we note that we want to get the full content of the file with every change, instead of just getting the changed region. We also create a utility method to create an instance of Buffer from the given text of the file.
public class TextDocumentSyncHandler : TextDocumentSyncHandlerBase
{
// ...
public override Task<Unit> Handle(
DidOpenTextDocumentParams request,
CancellationToken cancellationToken
)
{
bufManager.Add(request.TextDocument.Uri, CreateBuffer(request.TextDocument.Text));
return Unit.Task;
}
public override Task<Unit> Handle(
DidChangeTextDocumentParams request,
CancellationToken cancellationToken
)
{
var text = request.ContentChanges.FirstOrDefault()?.Text;
if (text is not null)
{
bufManager.Add(request.TextDocument.Uri, CreateBuffer(text));
}
return Unit.Task;
}
// ...
}
We then override the callbacks we get from the language client whenever a new document is opened (DidOpenTextDocumentParams) and whenever an open document is changed (DidChangeTextDocumentParams). In both cases, we simply get the latest content of the file and cache in our BufferManager to be analyzed. We don't need to do anything on document save and document close so we won't override those methods.
8. Activating the Handler
Finally we need to add the handler to the language server for the server to utilize it. We do it by injecting it during the construction of the language server.
var server = await LanguageServer.From(options =>
options
.WithInput(Console.OpenStandardInput())
.WithOutput(Console.OpenStandardOutput())
.WithServices(s => s.AddSingleton(new BufferManager()))
.WithHandler<TextDocumentSyncHandler>()
);9. Conclusion
At this point, you have created a basic language server that will be able to receive all text changes from the editor and cache it for analysis. That's all for this post. We will cover how to parse and analyze the text we just stored in our BufferManager in the next post.