21 posts tagged with "MoonBit"

In the past two years, MoonBit has demonstrated significant performance advantages across WebAssembly, JavaScript, and native backends. We believe that for a new language to be truly valuable, it must offer a generational leap in both core performance and developer experience. Today, we are excited to introduce MoonBit LLVM backend support with 8× the performance of Java in the FFT benchmark, and out-of-the-box debugging.

Why do we need an LLVM backend?​

Compiling to C is a common way to bootstrap a programming language through native compilation pipelines. The C programming language is mature, and widely-supported in a large number of platforms, making it the de facto "portable assembly code".

However, using C as a compilation target comes at a cost. While C is a standardized language, its history of more than 50 years had led to a fragmented ecosystem, where each C compiler has its own interpretation of and extensions to the language. The C standard that all compilers agree on, as a result, has many undefined or implementation-defined behaviors that can easily trap programmers. Examples of these include integer overflow, bitfields, enum underlying types, and many more. So, when transpiling a program to C, maintaining consistent semantics across platforms becomes a significant challenge.

LLVM, on the other hand, is a modern compilation infrastructure designed specifically for code optimizations, and generating consistent code across platforms. It's the workhorse powering many successful programming languages including the C-family (C, Objective-C, C++, though Clang), Rust, Julia, Swift, Zig.

LLVM's intermediate representation (IR) -- which is essentially a low-level programming language designed to represent the program during compiler optimization -- is a well-defined language with far less undefined behaviors than C. Via bindings to the LLVM library, compilers can construct the program in LLVM IR, optimize the program, and generate binary object files, all without ever leaving the process. This makes the compilation pipeline simpler, and without having to deal with the fragmented world of C compilers.

By having direct access to LLVM's compilation infrastructure, we can maintain full control over optimization and code generation, preserving important program information that would be lost when going through C. The LLVM infrastructure helps us handle the heavy lifting of applying low-level code optimizations (the same ones that make C and Rust fast), generating debug information, and generating code for different platforms, so that we can focus more on improving MoonBit as a programming language.

By adding LLVM to the list of our backends, we also gain several important capabilities. LLVM's support for debugging information makes it easier for us to map lines to source code, enhancing debugging experience of native binaries through gdb or lldb. In the future, it will also provide us with debugging information down to values of variables and data structures. LLVM also has support for garbage collectors and exceptions, which will make it easier for us to experiment with different runtime choices.

The pipeline of compiling to LLVM​

It's now good time for us to introduce the compilation pipeline of the MoonBit language.

Overview​

The compilation pipeline of a MoonBit program can be split into two parts.

The first part is on-demand and incremental, compiling each individual package from the source code, and outputs an intermediate representation which we call Core IR, via the command moonc build-package. Core IR is an intermediate representation in ANF-style (Administrative Normal Form) that we use to perform compiler optimizations. In this part, we perform various optimizations on the IR.

The second part is linking the packages into the final binary, via the command moonc link-core. Here, Core IRs from different packages are linked together and optimized again, using dead code elimination, inlining and other optimization techniques. The result is then passed down the compilation pipeline, ultimately generating an output binary.

Compilation of packages​

Within each package's compilation process, the compiling starts with the abstract syntax tree (AST) -- the original code represented in a structured way in memory.

Then, the types within each expression of the code are inferred from code structure and imported libraries. This gives us the Typed AST (TAST) -- AST with each expression annotated with its type.

From TAST, we then lower it into Core IR, the main intermediate representation, and perform several high-level transforms and optimizations. These optimization include desugaring async functions, unboxing data structures that do not escape, and removing dead code not referenced by others. The result Core IR from these optimizations becomes the compiled form of each library.

Linking and codegen​

Unlike many other compiled programming languages that don't do much after linking, there's a stack of lowering and optimizing passes after the linking of Core IRs that gradually turns the IR into target code.

Upon compilation, Core IR from different packages are first linked together. Then, it goes through optimization and monomorphization (removing generic functions) into MCore IR, and then through another pass removing closures and lowering into CLambda IR. Garbage collection intrinsics are also added during this stage.

The CLambda IR is the last step before emitting code for different backends. For WASM, JS and C backends, the output code is self-contained, so there's only one more lowering step to generate the output code, and then emit it into the output file.

For LLVM, things are a little more complex than them: CLambda IR is converted to LLVM IR and passed through more optimization passes in the LLVM library. After all LLVM optimizations, the code is emitted to object file, and finally linked with the prebuilt runtime into a working executable.

Performance​

The LLVM backend of MoonBit has achieved impressive performance results. For example, using a classic FFT algorithm benchmark, we ran the same Cooley-Tukey algorithm in both MoonBit and Java. MoonBit demonstrated an over 8x performance improvement compared to Java and was more than 3x faster than Java compiled with the GraalVM static compiler.

FFT (Fast Fourier Transform) is an efficient algorithm for computing the Discrete Fourier Transform of a sequence. It is widely used in fields such as signal processing and data compression. The FFT algorithm involves extensive numerical computations and array operations, making it an excellent benchmark for evaluating a language’s fundamental performance.

Debugging support​

For the JavaScript backend targeting the virtual machine, MoonBit provides an "out-of-the-box" debugging experience via Chrome Debugger. However, the debugging experience for the native C backend is not yet as seamless. In the C backend, users can debug the compiled C files, but there is a lack of precise mapping between the generated C code and the original source files.

With the help of LLVM, the MoonBit compiler now supports injecting DWARF format symbol tables into the final binary. This means MoonBit can now insert debugging information into the output binary, allowing instructions in the binary to be directly mapped to the source file, including line and column numbers. This significantly improves the accuracy of source code mapping in the compiled output.

Thanks to this debugging information mechanism, the MoonBit toolchain now enables developers to perform source-level debugging using standard debugging tools like LLDB. The debugger can automatically resolve symbol mappings. When execution reaches a predefined source location, breakpoints are triggered accurately. For example, developers can insert breakpoints in LLDB to pause execution at specific locations in their code.

The current debug info support is still preliminary, but already pretty usable. We have plans to bring more detailed information to it, including better symbols and name mangling, and more importantly, support showing the values of local variables and data structures.

Future prospects​

The most significant improvement brought by LLVM is that MoonBit can compile directly to native binary files without relying on a C compiler. This eliminates various compatibility issues associated with C compilers. Once support for distributing precompiled runtime libraries is implemented, MoonBit programs will be completely independent of the C compiler, generating native executables using only a linker.

With LLVM doing heavier lifting, we can also spare more time on experimenting new changes to the runtime. As LLVM has full integrated support for exception handling, we can convert the implementation of functions with error into try-catches and get better performance. LLVM also has integrated support for tracing garbage collection, which means we can also try using a tracing GC instead of Perceus and RC for memory management.

Conclusion​

The addition of LLVM backend marks an important milestone for MoonBit, expanding our compilation capabilities while complementing our existing C backend.

LLVM brings additional strengths to our ecosystem--from industrial-strength optimizations to improved debugging support. While there is still work to be done, this new backend opens up exciting possibilities for the future With both C and LLVM backends in our hands, MoonBit is now better equipped to serve diverse use cases and deployment scenarios.

We're excited to see what our users will build with these new capabilities, and we look forward to leveraging LLVM's rich feature set to make MoonBit an even more powerful and developer-friendly language.

New to MoonBit?

• Download MoonBit
• Explore MoonBit Beginner's Guide
• Play with MoonBit Language Tour
• Check out MoonBit Docs
• Join our Discord community

Both the MoonBit homepage and Language Tour feature a component that allows you to write MoonBit code directly in the browser and compile it in real-time. This component is Moonpad, which we developed and has now been released on npm. This blog post will guide you on how to use Moonpad on your own website.

All the code referenced in this post has been uploaded to GitHub, and you can view it at https://github.com/moonbit-community/moonpad-blog-examples.

What is Moonpad?​

MoonBit plugin provides many features that greatly benefit developers. In addition to supporting syntax highlighting, autocompletion, and error diagnostics for MoonBit syntax, it also provides an out-of-the-box debugger, real-time value tracing, testing, and a built-in MoonBit AI assistant, all of which help reduce the workload outside of core development, creating a highly efficient and smooth development environment. If you haven't experienced the complete MoonBit plugin, you can install our VS Code plugin.

Moonpad, as described in this tutorial, is an online MoonBit editor based on the Monaco editor. It supports syntax highlighting, autocompletion, and error diagnostics and more for MoonBit. Moreover, it can compile MoonBit code in real-time in the browser, making it essentially a simplified MoonBit plugin running in the browser. In this tutorial, we'll walk you through how to implement Moonpad, which you might have tried it in the MoonBit homepage and language tour.

How to Use Moonpad​

Setup​

First, create a new JS project:

mkdir moonpad
cd moonpad
npm init -y

Install dependencies:

npm i @moonbit/moonpad-monaco esbuild monaco-editor-core

Here’s a brief explanation of each dependency:

• @moonbit/moonpad-monaco is a Monaco editor plugin that provides syntax highlighting, autocompletion, error diagnostics, and compilation features for MoonBit.
• esbuild is a fast JavaScript bundling tool used for packaging source code.
• monaco-editor-core is the core library for the Monaco editor. We use this library instead of monaco-editor because it doesn’t include unnecessary syntax highlighting and semantic support for languages like HTML, CSS, and JS, making the bundled output smaller.

Writing Code​

Next, we need to write code for using Moonpad and Monaco editor along with a build script.

The following code is written in the moonpad directory and contains extensive comments to help you better understand it.

Here’s the code for index.js:

import * as moonbitMode from "@moonbit/moonpad-monaco";
import * as monaco from "monaco-editor-core";

// Monaco editor requires global variables; for more details: https://github.com/microsoft/monaco-editor
self.MonacoEnvironment = {
  getWorkerUrl: function () {
    return "/moonpad/editor.worker.js";
  },
};

// moonbitMode is an extension of Monaco Editor, which we call Moonpad.
// moonbitMode.init initializes various MoonBit functionalities and returns a simple MoonBit build system for compiling and running MoonBit code.
const moon = moonbitMode.init({
  // A URL string that can request onig.wasm
  // URL for onig.wasm (the WASM version of oniguruma for MoonBit syntax highlighting)
  onigWasmUrl: new URL("./onig.wasm", import.meta.url).toString(),
  // A worker running the MoonBit LSP server for language support
  lspWorker: new Worker("/moonpad/lsp-server.js"),
  // A factory function for creating a worker to compile MoonBit code in the browser
  mooncWorkerFactory: () => new Worker("/moonpad/moonc-worker.js"),
  // A configuration function to filter which codeLens should be displayed
  // Return false to disable any codeLens for simplicity
  codeLensFilter() {
    return false;
  },
});

// Note: All paths are hardcoded here, so later when writing the build script, we need to ensure these paths are correct.

// Mount Moonpad

// Create an editor model and specify its languageId as "moonbit". We can initialize the code content here.
// Moonpad only provides LSP services for models with languageId "moonbit".

const model = monaco.editor.createModel(
  `fn main {
  println("hello")
}
`,
  "moonbit",
);

// Create a Monaco editor and mount it to the `app` div element, displaying the model we just created.
monaco.editor.create(document.getElementById("app"), {
  model,
  // Moonpad provides its own theme with better syntax highlighting, compared to the default Monaco theme
  // Moonpad also provides a dark theme "dark-plus", which you can try replacing with "dark-plus"
  theme: "light-plus",
});

Create an esbuild.js file and enter the following code for our build script that bundles index.js:

const esbuild = require("esbuild");
const fs = require("fs");

// The dist folder is our output directory. Here we clear it to ensure esbuild always starts from scratch.
fs.rmSync("./dist", { recursive: true, force: true });

esbuild.buildSync({
  entryPoints: [
    "./index.js",
    // Bundle Monaco Editor worker required for providing editing services, which is also the return value of MonacoEnvironment.getWorkerUrl in `index.js`.
    // As previously mentioned, all paths are hardcoded, so we use `entryNames` to ensure the worker is named `editor.worker.js`.
    "./node_modules/monaco-editor-core/esm/vs/editor/editor.worker.js",
  ],
  bundle: true,
  minify: true,
  format: "esm",
  // The output directory corresponds to the hardcoded paths in `index.js`.
  outdir: "./dist/moonpad",
  entryNames: "[name]",
  loader: {
    ".ttf": "file",
    ".woff2": "file",
  },
});

fs.copyFileSync("./index.html", "./dist/index.html");

// Copy various worker files needed to initialize Moonpad, since they are already bundled, we don’t need to bundle them again with esbuild.
fs.copyFileSync(
  "./node_modules/@moonbit/moonpad-monaco/dist/lsp-server.js",
  "./dist/moonpad/lsp-server.js"
);
fs.copyFileSync(
  "./node_modules/@moonbit/moonpad-monaco/dist/moonc-worker.js",
  "./dist/moonpad/moonc-worker.js"
);
fs.copyFileSync(
  "./node_modules/@moonbit/moonpad-monaco/dist/onig.wasm",
  "./dist/moonpad/onig.wasm"
);

Finally, create a simple index.html file:

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>Moonpad Demo</title>
    <link rel="stylesheet" href="/moonpad/index.css" />
  </head>
  <body>
    <div id="app" style="height: 500px"></div>
    <script type="module" src="/moonpad/index.js"></script>
  </body>
</html>

Build and Start the Server​

Run the build script:

node esbuild.js

Your dist folder should now contain:

dist/
├── index.html
└── moonpad
    ├── codicon-37A3DWZT.ttf
    ├── editor.worker.js
    ├── index.css
    ├── index.js
    ├── lsp-server.js
    ├── moonc-worker.js
    └── onig.wasm

2 directories, 8 files

The codicon-37A3DWZT.ttf file hash at the end may not match exactly in your case, but that's not a problem.

Now, start a simple HTTP server in the dist folder::

python3 -m http.server 8081 -d ./dist

Open http://localhost:8081 in your browser, and you should see Moonpad successfully rendered.

Using Moonpad on Your Website​

Let’s see how to integrate Moonpad into your website, using Jekyll and Marp as examples.

Principle​

In the previous section, we created a moonpad folder containing all the files necessary to use Moonpad. Any webpage can import moonpad/index.js and moonpad/index.css and modify the mount logic in index.js to use Moonpad.

To use Moonpad on your webpage, you need to:

1. Modify the mount logic in moonpad/index.js based on your website framework.

2. Import moonpad/index.js and moonpad/index.css on the page where you want to use Moonpad.

3. Place the moonpad folder in the website’s static resource directory and ensure its path is /moonpad.

Using Moonpad in Jekyll​

Jekyll is a simple, blog-aware static site generator. It uses Markdown or Textile, along with the Liquid template engine, to generate static web pages. Jekyll combines content with templates to generate static files that can be directly deployed to any web server. It is particularly well-suited for GitHub Pages, allowing users to easily create and maintain blogs or websites.

Observing the Structure of Code Blocks Rendered by Jekyll​

For the following Markdown code block in Jekyll:

```moonbit
fn main {
  println("hello, moonbit")
}
```

It generates the following HTML structure:

<pre><code class="language-moonbit">fn main {
  println("hello, moonbit")
}
</code></pre>

If we want all MoonBit code blocks in Jekyll to be rendered in Moonpad, we need to replace the generated pre element with a div and mount Moonpad on this div.

Here is the implementation of this logic:

// Loop through all moonbit code blocks
for (const pre of document.querySelectorAll("pre:has(code.language-moonbit)")) {
  // Get the content of the code block
  const code = pre.textContent;
  // Create a div element, which will serve as the mount point for Monaco editor
  const div = document.createElement("div");
  // Set the div's height based on the code content
  const height = code.split("\n").length * 20;
  div.style.height = `${height}px`;
  // Replace the code block with the div
  pre.replaceWith(div);
  // Create a Monaco editor model using the obtained code content
  const model = monaco.editor.createModel(code, "moonbit");
  // Create Monaco editor and mount it to the div, displaying the previously created model
  monaco.editor.create(div, {
    model,
    theme: "light-plus",
  });
}

Simply replace the mounting logic in index.js with the above code.

Importing Moonpad​

Jekyll supports using HTML directly in Markdown, so we can import Moonpad directly in the Markdown. Just add the following code at the end of the Markdown file where Moonpad is needed.

<link rel="stylesheet" href="/moonpad/index.css" />
<script type="module" src="/moonpad/index.js"></script>

Place the Moonpad Folder in Jekyll’s Static Directory​

After running jekyll build, Jekyll will place all static assets in the _site directory. We only need to copy the moonpad folder into the _site directory.

The directory structure of the _site folder should look like this after copying:

_site/
├── ... # Other static assets
└── moonpad
    ├── codicon-37A3DWZT.ttf
    ├── editor.worker.js
    ├── index.css
    ├── index.js
    ├── lsp-server.js
    ├── moonc-worker.js
    └── onig.wasm

Result​

After completing the above steps, you can use Moonpad in Jekyll. The result will look like this:

Using Moonpad in Marp​

Marp is a Markdown conversion tool that can convert Markdown files into slides. It is based on the Marpit framework, supports custom themes, and allows you to create and design slides using simple Markdown syntax, making it ideal for technical presentations.

Observing the Structure of Code Blocks Rendered by Marp​

For the same code block as in the previous Jekyll section, Marp renders the HTML structure as follows:

<marp-pre>
  ...
  <code class="language-moonbit">fn main { println("hello, moonbit") } </code>
</marp-pre>

Clearly, if we want all MoonBit code blocks to use Moonpad for rendering, we need to replace marp-pre with div and mount Moonpad on this div.

Here is the implementation of this logic, which is very similar to the Jekyll example:

for (const pre of document.querySelectorAll(
  "marp-pre:has(code.language-moonbit)",
)) {
  const code = pre.querySelector("code.language-moonbit").textContent;
  const div = document.createElement("div");
  const height = code.split("\n").length * 20;
  div.style.height = `${height}px`;
  pre.replaceWith(div);
  const model = monaco.editor.createModel(code, "moonbit");
  monaco.editor.create(div, {
    model,
    theme: "light-plus",
  });
}

Just replace the mounting logic in index.js with the above code.

Importing Moonpad​

Marp also supports using HTML in Markdown, but this feature must be explicitly enabled. Add the following code at the end of the Markdown file where Moonpad is needed:

<link rel="stylesheet" href="/moonpad/index.css" />
<script type="module" src="/moonpad/index.js"></script>

And enable the html option when building:

marp --html

Place the Moonpad Folder in Marp’s Static Directory​

There are two common ways to preview slides in Marp: one is by using the built-in server feature, and the other is by exporting the Markdown file as HTML and setting up your own server.

For the built-in server feature, place the moonpad folder in the folder specified by the marp --server command.

For the case where you export Markdown as HTML, ensure that the moonpad folder and the exported HTML file are in the same directory.

Result​

After completing the above steps, you can use Moonpad in Marp. The result will look like this:

Unfortunately, in Marp, the hover tooltip position in Monaco editor is not correct. We are currently unsure how to resolve this issue.

How to Use Moonpad to Compile MoonBit Code​

As mentioned earlier, moonbitMode.init returns a simple build system, which we can use to compile and run MoonBit code. It exposes two methods: compile and run, which are used to compile and run MoonBit code, respectively. For example:

// compile also accepts more configuration via parameters, such as compiling files with tests. Here, we only compile a single file.
const result = await moon.compile({ libInputs: [["a.mbt", model.getValue()]] });
switch (result.kind) {
  case "success":
    // If compilation is successful, the JavaScript code compiled by the backend compiler will be returned.
    const js = result.js;
    // You can use the run method to execute the compiled JS and get the standard output stream.
    // Note that the smallest unit of this stream is not characters, but each line of string output from the standard output.
    const stream = moon.run(js);
    // Collect the content from the stream into a buffer and output it to the console.
    let buffer = "";
    await stream.pipeTo(
      new WritableStream({
        write(chunk) {
          buffer += `${chunk}\n`;
        },
      }),
    );
    console.log(buffer);
    break;
  case "error":
    break;
}

Open the console, and you will see the output: hello.

For more usage of the compile function and how to display the code output on the page, refer to the language tour code: moonbit-docs/moonbit-tour/src/editor.ts#L28-L61

New to MoonBit?

• Download MoonBit.
• Explore MoonBit Beginner's Guide.
• Play with MoonBit Language Tour.
• Check out MoonBit Docs.
• Join our Discord community.

In one of our previous blog posts, we have introduced how to consume a MoonBit-powered Wasm library, Cmark, within frontend JavaScript. In this post, we will explore how to profile the same library directly from the Chrome browser. Hopefully, this will help you gain insights and eventually achieve better overall performance with MoonBit in similar use cases.

For this blog post in particular, we will focus on making minimal changes to our previous "Cmark for frontend" example, so that we may apply Chrome's built-in V8 profiler to Cmark's Wasm code.

Profiling the Cmark library​

We can easily refactor the original frontend application to include a new navigation bar with two links, "Demo" and "Profiling". Clicking the first will tell the browser to render the HTML for the original A Tour of MoonBit for Beginners example, and clicking the second will lead it to our new document for profiling. (If you are curious about the actual implementation, you can find a link pointing to the final code towards the end of this post.)

Now we are ready to write some code to actually implement Wasm profiling. Are there any particularities involved in terms of profiling Wasm compared to JavaScript?

As it turns out, we can use the very same APIs for profiling Wasm as we do for JavaScript code. There is an article in the Chromium documentation that describes them in more detail, but in short:

• When we call console.profile(), the V8 profiler will start recording a CPU performance profile;
• After that, we can call the performance-critical function we would like to analyze;
• Finally, when we call console.profileEnd(), the profiler will stop the recording and then visualizes the resulting data in Chrome's Performance Tab.

With that in mind, let's have a look at the actual implementation of our profiling functionality:

async function profileView() {
  const docText = await fetchDocText("../public/spec.md");
  console.profile();
  const res = cmarkWASM(docText);
  console.profileEnd();
  return (
    `<h2>Note</h2>
<p>
  <strong
    >Open your browser's
    <a
      href="https://developer.chrome.com/docs/devtools/performance"
      rel="nofollow"
      >Performance Tab</a
    >
    and refresh this page to see the CPU profile.</strong
  >
</p>` + res
  );
}

As you can see, we have to minimize the scope of the code being executed when the profiler is activated. Thus, the code is written so that the call to the cmarkWASM() function is the only thing within that scope.

On the other hand, we have chosen the 0.31.2 version of the CommonMark Specification (a.k.a. spec.md in the code above) as the input document for the profiling mode. This is notably because of the document's richness particularly in the employment of different Markdown features, in addition to its sheer length which can cause trouble for many Markdown parsers:

> wc -l spec.md  # line count
    9756 spec.md
> wc -w spec.md  # word count
   25412 spec.md

We have reorganized our frontend application so that clicking on the "Profiling" link in the navigation bar will trigger the profileView() function above, giving the following:

If you have ever dug into performance optimization, this flame graph above should look pretty familiar...

Wait, what are wasm-function[679], wasm-function[367] and so on? How are we supposed to know which function corresponds to which number?

It turns out we need to retain some debug information when building our Wasm artifact. After all, we have been using the following command to build our MoonBit project:

> moon -C cmarkwrap build --release --target=wasm-gc

... and stripping is the standard behavior of moon when producing a release build.

Fortunately, there is an extra flag we can use to keep the symbol information without having to resort to a slow debug build: --no-strip. Let's rebuild our project with it:

> moon -C cmarkwrap build --release --no-strip --target=wasm-gc

Note

Similarly, if we would like to use wasm-opt on the resulting Wasm artifact, we can use the --debuginfo (or -g) flag of wasm-opt to preserve the function names in the optimized output.

With the function names retained, we can finally see what is really going on in the Performance Tab!

Analyzing the Flame Graph​

The flame graph as shown in the previous screenshot can provide a nice summary of function calls and their respective execution times in our code. In case you are not familiar with it, the main ideas behind it are as follows:

• The Y-axis represents the call stack, with the topmost function being the one that was called first;
• The X-axis represents the time spent in execution, with the width of each box-shaped node corresponding to the total time spent in a certain function call and its children.

Since we are investigating the performance of the Cmark library in particular, we should move downwards and concentrate on the node for @rami3l/cmark/cmark_html.render(). Here, for example, we can clearly see that the execution of render() is divided into two main parts, as represented by two children nodes on the graph:

• @rami3l/cmark/cmark.Doc::from_string(), which stands for the conversion of the input Markdown document into a syntax tree;
• @rami3l/cmark/cmark_html.from_doc(), which stands for the rendering of the syntax tree into the final HTML document.

To get a better view, let's highlight the render() node in the flame graph with a single click. This will tell Chrome to update the "Bottom-up" view to show only the functions that are transitively called by render(), and we will get something like the following:

After sorting the items by self time (i.e. total time excluding child time) in the "Bottom-up" view, we can easily find out the functions that have consumed the most time on their own. This suggests that their implementation might be worth a closer look. Meanwhile, we would also want to try eliminating deep call stacks when possible, which can be found by looking for the long vertical bars in the flame graph.

Achieving High Performance​

During its development process, Cmark has been profiled hundreds of times using the very method we have seen above in pursuit of satisfactory performance, but how does it actually perform against popular JavaScript Markdown libraries?

For this test, we have chosen Micromark and Remark---two well-known Markdown libraries in the JavaScript ecosystem---as our reference. We have used a recent build of Chrome 133 in this test as our JS and Wasm runtimes, and have imported Tinybench to measure the average throughput of each library.

Below is the average throughput of these libraries converting a single copy of the CommonMark spec to HTML on a MacBook M1 Pro:

The results are quite clear: thanks to the continuous profiling-and-optimizing process, Cmark is now significantly faster than both JavaScript-based libraries by a factor of about 12x for Micromark and 13x for Remark. Furthermore, wasm-opt's extra optimization passes can give Cmark another performance boost, bringing these factors up to about 13x and 14x respectively.

In conclusion, the performance of Cmark is a testament to the power of MoonBit in providing visible efficiency improvements in an actual frontend development scenario.

If you are interested in the details of this demo, you may check out the final code on GitHub. The code based used in benchmarking is also available here.

New to MoonBit?

• Download MoonBit.
• Explore MoonBit Beginner's Guide.
• Play with MoonBit Language Tour.
• Check out MoonBit Docs.
• Join our Discord community.

In one of our previous blog posts, we have already started exploring the use of JavaScript strings directly within MoonBit's Wasm GC backend. As we have previously seen, not only is it possible to write a JavaScript-compatible string-manipulating API in MoonBit, but once compiled to Wasm, the resulting artifact is impressively tiny in size.

In the meantime, however, you might have wondered what it will look like in a more realistic use case. That is why we are presenting today a more realistic setting of rendering a Markdown document on a JavaScript-powered web application, with the help of the MoonBit library Cmark and Wasm's JS String Builtins Proposal.

Motivation​

Cmark is a new MoonBit library for Markdown document processing, which makes it possible to parse both vanilla CommonMark and various common Markdown syntax extensions (task lists, footnotes, tables, etc.) in pure MoonBit. Furthermore, open to external renderers since its early days, it comes with a ready-to-use official HTML renderer implementation known as cmark_html.

Given Markdown's ubiquitous presence in today's cyberspace and the web world in particular, a conversion pipeline from Markdown to HTML remains an important tool in virtually every JavaScript developer's toolbox. As such, it also constitutes a perfect scenario for showcasing the use of MoonBit's Wasm GC APIs in frontend JavaScript.

Wrapping over Cmark​

For the sake of this demo, let's start with a new project directory:

> mkdir cmark-frontend-example

In that very directory, we will first create a MoonBit library cmarkwrap that wraps Cmark:

> cd cmark-frontend-example && moon new cmarkwrap

This extra project cmarkwrap is required mostly because:

• Cmark in itself doesn't expose any API across the FFI boundary, which is the common case for most MoonBit libraries;
• We will need to fetch the Cmark project from the mooncakes.io repository and compile it locally to Wasm GC anyway.

cmarkwrap's structure is simple enough:

• cmark-frontend-example/cmarkwrap/src/lib/moon.pkg.json:

  {
    "import": ["rami3l/cmark/cmark_html"],
    "link": {
      "wasm-gc": {
        "exports": ["render", "result_unwrap", "result_is_ok"],
        "use-js-builtin-string": true
      }
    }
  }
  

  This setup is pretty much identical to the one we have seen in the previous blog, with the use-js-builtin-string flag enabled for the Wasm GC target, and the relevant wrapper functions exported.

• cmark-frontend-example/cmarkwrap/src/lib/wrap.mbt:

  ///|
  typealias RenderResult = Result[String, Error]
  
  ///|
  pub fn render(md : String) -> RenderResult {
    @cmark_html.render?(md)
  }
  
  ///|
  pub fn result_unwrap(res : RenderResult) -> String {
    match res {
      Ok(s) => s
      Err(_) => ""
    }
  }
  
  ///|
  pub fn result_is_ok(res : RenderResult) -> Bool {
    res.is_ok()
  }
  

  This is where things start to get interesting. The render() function is a wrapper of the underlying @cmark_html.render() function that, instead of being a throwing function, returns a RenderResult type.

  Unfortunately, being a Wasm object (instead of a number or a string), a RenderResult is opaque to JavaScript, and thus cannot be directly consumed by our JavaScript caller. As a result, we also need to provide means to destruct that very type from within MoonBit as well: the result_unwrap() and result_is_ok() functions are there exactly for this purpose, and that is why they accept a RenderResult input.

Integrating with JavaScript​

Now is the time to write the web part of this project. At this point, you are basically free to choose any framework or bundler you prefer. This demo in particular has chosen to initialize a minimal project skeleton under the cmark-frontend-example directory with no extra runtime dependencies. For now, let's focus on the HTML and JS parts of the project:

• cmark-frontend-example/index.html:

  <!doctype html>
  <html lang="en">
    <head>
      <meta charset="UTF-8" />
      <meta name="viewport" content="width=device-width, initial-scale=1.0" />
      <title>Cmark.mbt + JS</title>
    </head>
    <body>
      <div id="app"></div>
      <script type="module" src="/src/main.js"></script>
      <link rel="stylesheet" href="/src/style.css" />
    </body>
  </html>
  

  This simple HTML file includes a single div of id="app", which will become the target to which we render the Markdown document later.

• cmark-frontend-example/src/main.js:

  const cmarkwrapWASM = await WebAssembly.instantiateStreaming(
    fetch("../cmarkwrap/target/wasm-gc/release/build/lib/lib.wasm"),
    {},
    {
      builtins: ["js-string"],
      importedStringConstants: "_",
    },
  );
  const { render, result_is_ok, result_unwrap } =
    cmarkwrapWASM.instance.exports;
  
  function cmarkWASM(md) {
    const res = render(md);
    if (!result_is_ok(res)) {
      throw new Error("cmarkWASM failed to render");
    }
    return result_unwrap(res);
  }
  
  async function docHTML() {
    const doc = await fetch("../public/tour.md");
    const docText = await doc.text();
    return cmarkWASM(docText);
  }
  
  document.getElementById("app").innerHTML = await docHTML();
  

  As it turns out, integrating cmarkwrap into JavaScript is fairly straightforward. After fetching and loading the Wasm artifact, we can call the wrapper functions right away. The result_is_ok() function helps us identify if we are on the happy path: if we are, we can unwrap the HTML result across the FFI boundary with result_unwrap(); otherwise, we can throw a JavaScript error. If everything goes as well, we can finally use the aforementioned <div id="app"></div> as our output target.

We can now compile the MoonBit Wasm GC artifact and launch the development server:

> moon -C cmarkwrap build --release --target=wasm-gc
> python3 -m http.server

And voilà! You can now read A Tour of MoonBit for Beginners rendered by the Cmark MoonBit library in a JavaScript frontend application at http://localhost:8000.

You may find the code of this demo on GitHub.

New to MoonBit?

• Download MoonBit.
• Explore MoonBit Beginner's Guide.
• Play with MoonBit Language Tour.
• Check out MoonBit Docs.
• Join our Discord community.

We’re excited to announce that the MoonBit compiler, complete with a Wasm backend, is now open source and available on GitHub! Initially developed as a private project over the past two years, this approach allowed us to move quickly and focus on building a strong foundation. Now, we’re ready to gradually open the doors and invite the community to help shape its future. The source code is released under a permissive variant of the Server Side Public License, and we aim to embrace even greater openness as the community grows.

You can find the code on GitHub.

Motivations​

Why are we open-sourcing the Wasm backend? WebAssembly has great potential to be safe, fast, and cross-platform, but not fully delivered by existing languages. MoonBit aims to be a Wasm-optimized language in the first place. In our first blog post, we launched MoonBit as a fast, compact, and user-friendly language for WebAssembly. In the past year, while MoonBit launched JS and native backends, WebAssembly has been our priority and most polished backend. MoonBit has supported the Wasm component model and reduced the code size to 27kB in the http-hello-world example—much smaller than other languages, making MoonBit-wasm ready for production usage.

Why a Relaxed SSPL​

We currently use a modified SSPL (Server Side Public License) with two key exceptions:

1. Artifacts produced by the MoonBit compiler may be licensed by the user under any license of their choosing, MoonBit users have the freedom to choose the license for their own MoonBit source code and generated artifacts.

2. Modifications to the compiler are allowed for non-commercial purposes.

While we value openness, we chose this adjusted SSPL instead of a fully permissive license for two main reasons:

1. MoonBit is still in beta-preview. Introducing forks at this point could risk destabilizing the project. We aim to reach a more mature and stable status before welcoming community contributions.

2. We want to safeguard against large cloud vendors leveraging our work for commercial purposes in a way that could undermine our efforts.

In the past two years, our team worked hard to improve MoonBit and its toolchain, staying true to our vision of creating a fast, simple, and efficient language. By open sourcing MoonBit, we would like to reassure our users that our team remains dedicated to MoonBit's pace of growth and innovation. We also want to ensure our users that MoonBit is not going to adopt open-core model, all MoonBit users will get the best-developed compiler and IDE support. MoonBit team will try to generate revenue through hosting services and hardware SDKs over the long term.

Visit MoonBit Public Source License.

Roadmap​

Building a programming language is a long journey. It took Rust 9 years and Go 5 years to reach 1.0. Made by a young and driven team, MoonBit is steadily moving forward. We understand that community adoption and expansion are key to a new language, and we’re fully committed to nurturing an engaged and collaborative community around MoonBit. So far, we have open-sourced the core library and most tools, including build tools, lex, markdown, and more to come. Having the compiler source available is important for security measures. Open-sourcing the Wasm backend is another major step, and it is on our roadmap to open source more in the future.

New to MoonBit?

• Download MoonBit.
• Explore MoonBit Beginner's Guide.
• Play with MoonBit Language Tour.
• Check out MoonBit Docs.
• Join our Discord community.

Wasm has long been known for its high performance, but its interoperability costs with JavaScript have limited its full potential. With Chrome (131) now supporting Wasm's new JS string builtins proposal by default, MoonBit has fully adapted the JS string builtins proposal in the compiler. Benchmark shows that the file generated by MoonBit is 60 times smaller than the one generated by Rust.

Background​

Since the Wasm GC proposal was implemented last year, many programming languages have supported compilation to Wasm code using the standard GC. MoonBit was one of the first to release the wasm-gc backend. However, the standard GC only provides basic data types like structs and arrays, and there is no standardized implementation for complex types such as strings. Considering the different encoding formats of strings and the various implementation approaches, standardizing strings in Wasm is an extremely complex task. In this context, there are two main approaches for handling string types:

• Custom String Type Using Structs or Arrays

One approach is to use the basic data types (like structs or arrays) provided by the GC standard to implement a custom string type. For example, languages that use UTF-16 encoding could implement strings as an (array 16). This approach offers high flexibility and can be optimized for better string operation performance. However, when interacting with JavaScript host Web APIs (such as fetch) or the DOM APIs, the custom string type often needs to be converted into a JavaScript string, which typically requires multiple memory copies, leading to performance overhead.

• External Reference to JavaScript String

The second approach is to use JavaScript strings via external references (externref). String-related operations (such as substring and concatenation) are handled by importing JavaScript functions. This method avoids the need for conversion when working with JavaScript Web APIs, but it introduces the overhead of indirect function calls every time a string operation is invoked. Indirect calls are less optimized by the Wasm engine and have higher performance costs compared to direct function calls.

Clearly, neither of the two approaches is ideal. In this context, the JS string builtins proposal was introduced and has already been implemented in major browser environments. The proposal aims to address the performance issues with the second approach by allowing Wasm modules to import functions from a predefined module called wasm:js-string. This module includes common string operations like length, concat, and substring. Unlike regular functions, the Wasm engine is aware of these functions and can handle them more efficiently, sometimes inlining them to improve performance.

How to use​

MoonBit now supports the JS string builtins proposal. Below are two examples demonstrating how to use JS string builtins in both Deno and browser environments. Full code examples are available at the MoonBit demo repository.

Deno Environment​

First, create a new project using moon new --lib demo-js-builtin-string. Next, create a package called palindrome and write a function in MoonBit to check if a string is a palindrome:

// demo-js-builtin-string/palindrome/palindrome.mbt
pub fn palindrome(s : String) -> String {
  let len = s.length()
  for i in 0..<(len / 2) {
    if s[i] != s[len - i - 1] {
      return "false"
    }
  }
  return "true"
}

This function has the type (String) -> String. When compiled to the wasm-gc backend, the MoonBit String type will be translated into an array type in Wasm. This array type is not directly compatible with JavaScript's String object, the exported palindrome function cannot be used directly in the JavaScript environment. However, after enabling js-string-builtins, MoonBit's String type will be compiled as a JavaScript String object, allowing for seamless interoperability. To enable this feature, add the following to the moon.pkg.json file:

// demo-js-builtin-string/palindrome/moon.pkg.json
{
  "link": {
    "wasm-gc": {
      "exports": ["palindrome"],
      "use-js-builtin-string": true
    }
  },
}

Then, build the package with moon build and test this function in a JavaScript environment:

// demo-js-builtin-string/a.js
const { instance } = await WebAssembly.instantiateStreaming(
  fetch(
    new URL(
      "./target/wasm-gc/release/build/palindrome/palindrome.wasm",
      import.meta.url
    )
  ),
  {},
  {
    builtins: ["js-string"],
    importedStringConstants: "_",
  }
);
console.log(instance.exports.palindrome("hello"));
console.log(instance.exports.palindrome("ada"));

By setting builtins: ["js-string"], JS-string-builtins is enabled in the JavaScript environment. The importedStringConstants field is used for string literals, where "_" is the default name used by MoonBit's compiler for string constants (we will talk more about string constants in later sections).

Run the program using the latest Deno with:

$ deno run -A --v8-flags=--experimental-wasm-imported-strings a.js
false
true

Browser Environment​

Now, let’s use MoonBit to interact with the DOM API in a browser environment. Create a package called dom, define the DOM object as an abstract type, and import a set_css function from the JavaScript environment to manipulate the DOM. For instance, we can use this function to implement a change_color function.

// demo-js-builtin-string/dom/dom.mbt
type DOM
fn set_css(dom : DOM, key : String, value : String) -> Unit = "dom" "set_css"
pub fn change_color(dom : DOM) -> Unit {
  set_css(dom, "color", "red")
}

Enable the Js-string-builtins feature in the moon.pkg.json file:

// demo-js-builtin-string/dom/moon.pkg.json
{
  "link": {
    "wasm-gc": {
      "exports": ["change_color"],
      "use-js-builtin-string": true
    }
  }
}

In the JavaScript environment, import the set_css function to modify the DOM and call the change_color function to change the color of a text element:

// demo-js-builtin-string/b.mjs
const { instance } = await WebAssembly.instantiateStreaming(
  fetch(
    new URL("./target/wasm-gc/release/build/dom/dom.wasm", import.meta.url)
  ),
  {
    dom: {
      set_css: (dom, key, value) => {
        dom.style[key] = value;
      },
    },
  },
  {
    builtins: ["js-string"],
    importedStringConstants: "_",
  }
);
document.getElementById("hello").addEventListener("click", () => {
  instance.exports.change_color(document.getElementById("hello"));
});

Meanwhile, we use the following HTML file to demonstrate this example. In this HTML file, we create a simple <div> element and include our previously written JavaScript module using the <script> tag. This way, when the user clicks on the "hello" text, it triggers the change_color function we defined in MoonBit, changing the text color to red.

<!-- demo-js-builtin-string/index.html -->
<html>
  <body>
    <div id="hello">hello</div>
    <script src="b.mjs" type="module"></script>
  </body>
</html>

We can use a simple HTTP server to host our HTML and JavaScript files. For example, we can quickly create a local server using Python's http.server module by running the following command:

python -m http.server 8000

Then, open the browser at http://localhost:8000 to see the example. Clicking the "hello" text will change its color to red, demonstrating how MoonBit with JS-string-builtins can interact with the DOM.

More Details​

Using moon build --output-wat, you can view the generated textual Wasm code. For example, you’ll see that the palindrome function uses the length and charCodeAt functions from the wasm:js-string module, and string literals like "true" are declared with import "_" "true", the same with "false":

;; target/wasm-gc/release/build/palindrome/palindrome.wat
(func $moonbit.js_string.length (import "wasm:js-string" "length")
 (param externref) (result i32))
(func $moonbit.js_string.charCodeAt (import "wasm:js-string" "charCodeAt")
 (param externref) (param i32) (result i32))
(global $moonbit.js_string_constant.0 (import "_" "true") (ref extern))
(global $moonbit.js_string_constant.1 (import "_" "false") (ref extern))

Two things to note here:

• The special import module "_" corresponds to the importedStringConstants field provided in JavaScript when loading a Wasm module. This field is used to declare a special module name. By using this specific module name, the JS builtin string proposal overrides the import syntax to declare string literals, as shown in the above .wat file. MoonBit uses "_" as the default value for importedStringConstants, in order to reduce the size of the generated Wasm binary file. However, if "_" conflicts with other modules that need to be imported, this field can be configured in the moon.pkg.json file. For example:

// moon.pkg.json
{
  "link": {
    "wasm-gc" : {
      "use-js-builtin-string": true,
      "imported-string-constants": "some-other-name"
    }
    ...
  }
}

In this case, the same module name should be passed in JavaScript as the value of importedStringConstants:

  {
    builtins: ["js-string"],
    importedStringConstants: "some-other-name",
  }

• In MoonBit, println function directly uses console.log. However, this function is not included in the original proposal, so when using Wasm code generated by a MoonBit program with println in a JavaScript environment, console.log needs to be provided via an import object, like this:

const { instance } = await WebAssembly.instantiateStreaming(
  ...,
  {
    console: {
      log: console.log,
    },
  },
  {
    builtins: ["js-string"],
    importedStringConstants: "_",
  }
);

Size Comparison​

By using the JS string builtins proposal, the binary size of Wasm generated by MoonBit for the wasm-gc backend is further reduced. Here, we compare it with Rust. For example, in MoonBit, we can implement the following print_and_concat function:

// demo-js-builtin-string/print_and_concat/print_and_concat.mbt
pub fn print_and_concat(a : String, b : String) -> String {
  let c = a + b
  println(c)
  return c
}

The string printing and concatenation functionality is provided by the JavaScript host environment, so the generated code is very small. After compiling with MoonBit, the resulting Wasm binary is only 182 bytes. After compression with wasm-opt, its size is reduced to just 117 bytes:

$ wc -c target/wasm-gc/release/build/print_and_concat/print_and_concat.wasm
182 target/wasm-gc/release/build/print_and_concat/print_and_concat.wasm
$ wasm-opt -Oz target/wasm-gc/release/build/print_and_concat/print_and_concat.wasm -o - -all | wc -c
117

Next, we implement similar functionality in Rust:

// demo-js-builtin-string/rust-example/src/lib.rs
use wasm_bindgen::prelude::*;

#[wasm_bindgen]
pub fn print_and_concat(a: &str, b: &str) -> String {
    let c = format!("{}{}", a, b);
    println!("{}", c);
    c
}

We use wasm-pack to compile this Rust program to Wasm and measure its size:

$ wasm-pack build --target web
$ wc -c ./pkg/rust_example_bg.wasm
20727 ./pkg/rust_example_bg.wasm

The resulting Wasm binary is approximately 20 kB. To further reduce the size, we can add some optimization options to the Cargo.toml file:

// demo-js-builtin-string/rust-example/Cargo.toml

[profile.release]
lto = true
opt-level = 'z'

Using the nightly Rust compiler, we can run the following command to trim some unnecessary modules during compilation and obtain a Wasm file:

$ wasm-pack build --target web --release --manifest-path ./Cargo.toml -Z build-std=panic_abort,std -Z build-std-features=panic_immediate_abort
$ wc -c ./pkg/rust_example_bg.wasm                                                                                    ~/playground/moonbit-playground/js-builtin-string/demo-js-builtin-string/rust-example
12534 ./pkg/rust_example_bg.wasm
$ wasm-opt -Oz pkg/rust_example_bg.wasm -o - -all | wc -c                                                             ~/playground/moonbit-playground/js-builtin-string/demo-js-builtin-string/rust-example
12706

The resulting Wasm file size from Rust is about 12 kB, which is 60 times larger than the file generated by MoonBit. Although much of this size difference is due to memory management code, Rust does not provide a toolchain adapted to Wasm's native GC. In contrast, MoonBit allows users to choose whether to use Wasm’s native GC when targeting the Wasm platform so as to generate very compact and efficient binary code.

More resources：

• Get started with MoonBit.
• Check out the MoonBit Docs.
• Join our Discord community.
• Explore MoonBit demos in the Gallery.

MoonBit, launched in August 2023, is an AI and cloud-native language toolchain with multi-backend support, delivering exceptional performance on both WebAssembly and JavaScript.

Today, we are happy to announce the release of MoonBit native backend that compiles MoonBit directly to machine code. The MoonBit native backend allows for native compilation and direct execution on hardware without the need for a virtual machine. We are also excited to see that MoonBit native backend showcases a significant performance advantage over traditional languages, running 15x faster compared to Java in some scenarios.

Why Native Backend​

Programming languages are generally divided into two categories: virtual machine based and native. Virtual machine-based languages, such as Java, Kotlin, and most scripting languages, require a VM at runtime to interpret programs dynamically. Native languages, such as C/C++, Rust, and Go, compile to machine code directly and can be executed directly on hardware, without the need for a VM. In most cases, the performance and development experience of a programming language largely depend on the quality of the compiler. Native compilation offers more performance potential, but it also imposes more challenges on compiler implementation.

Expanded Ecosystem​

Currently, MoonBit supports WebAssembly and JavaScript backends, which both have broad ecosystem support. However, in scenarios where high performance is critical, these backends may face limitations. By supporting the native backend, MoonBit can meet the demand of high-performance applications, further enhancing its performance and expanding its application scenarios.

Balance Performance & Developer Experience​

Balancing performance and developer experience is a common challenge in the design of native languages. In typical native languages like C, C++, and Rust, users must invest significant effort in managing memory and other resources, manually controlling many low-level details, which decreases the development experience and efficiency. Some managed languages, such as Java, Kotlin, and Swift, also offer options to compile to native, but their execution time and memory consumption still lag significantly behind true native languages. In MoonBit native backend, we adopt a more modern top-level design, achieving a dual advantage of performance and developer experience.

Benchmark​

Time to show the data. In actual performance comparisons, MoonBit native backend significantly outperforms programming languages like Java and Swift in numerical computation and memory management performance, while effectively balancing performance advantages with developer experience.

Numerical Computations: 15x faster than Java​

The performance advantage of native languages over virtual machine-based languages is particularly evident in numeric computing, due to native languages' ability to optimize memory layout and utilize hardware acceleration instructions.

Below is a comparison of MoonBit and Java implementations of FFT (Fast Fourier Transform) using the Cooley-Tukey algorithm. FFT is widely used in signal processing and compression. In the benchmark, MoonBit shows over 15x performance improvement compared to Java and over 4x compared to the most advanced commercial AOT Java compiler, GraalVM. The algorithm involves heavy numeric computations and array operations, making it a good indicator of language performance baseline:

MoonBit achieves significant performance gains without sacrificing development experience. MoonBit automatically manages memory just like Java, so users are not burdened with manual memory management. Moreover, MoonBit's modern language design offers more advanced features than Java to enhance developer experience.

For numeric computation, performance depends primarily on two factors: optimization of the computations themselves and overhead from the language itself. C/C++ excels in this domain, not only due to compiler optimizations but also because the language itself imposes minimal overhead. For higher-level languages with automatic memory management, eliminating the overhead of advanced language features is key to improving performance. Additionally, due to the cache mechanism of modern CPU, data layout is also crucial to performance. MoonBit achieves both high runtime performance and an excellent development experience through our modern top-level design:

• MoonBit's language design is concise, minimizing additional performance overhead, and its data structures are more compact and cache friendly.

• MoonBit uses a multi-layer IR (Intermediate Representation) architecture with global optimization in its compiler, making it capable of eliminating the overhead of advanced features like generics, and optimizing data layout globally.

Memory Management: Outperforming Java & Swift​

While numeric computation tasks reflect the performance baseline of a language, real-world applications often involve a lot of memory allocation and operations on in-memory data structures. Thanks to a customized memory management system, MoonBit native backend also excels in memory operation performance and usage, outperforming Java with its high-performance garbage collector (GC) and Swift, another language with automatic memory management that compiles to native binary.

The benchmarks here come from the paper Perceus: Garbage Free Reference Counting with Reuse. The benchmark programs contain heavy memory allocation, showcasing memory management performance well.

Automatic memory management systems generally fall into two categories: tracing GC and reference counting. Tracing GC tends to offer better throughput but consumes more memory, while reference counting has lower memory consumption and better responsiveness, though its throughput is usually bad.

Due to MoonBit's multi-layer optimizations and custom memory management system, MoonBit achieves performance better than or close to Java's high-performance tracing GC in most scenarios, and far surpasses Swift's reference counting. Moreover, MoonBit's memory consumption is slightly better than Swift's, and uses way less memory than Java in some benchmarks.

In embedded and IoT devices, both computation and memory resources are very limited. Therefore, programming languages need to perform well in both excution time and memory consumption to suit embedded/IoT development well. Due to hardware constraints, developers often resort to low-level languages like C/C++ for better performance. However, this significantly increases cognitive load and reduces development efficiency. MoonBit native presents a new possibility to balance execution performance, memory consumption, and your development experience.

From the two tasks above, we can see that although MoonBit native backend is still in its early stages, it holds enormous potential to fully unleash hardware performance in the future.

Integrate Low-Level Ecosystems with Security and Reliability​

The native ecosystem, dominated by C/C++, plays a crucial role in the whole software ecosystem. Low-level APIs in operating systems, high-performance computing libraries, and AI tensor libraries like llama.cpp and ggml all rely on the C ABI for interfacing. MoonBit native can seamlessly integrate with third-party libraries using C ABI, unlocking new application scenarios for MoonBit.

MoonBit native employs a multi-layer IR design, with the last IR layer being a subset of C. This allows MoonBit to interoperate smoothly with C without heavy FFI (Foreign Function Interface) overhead, taking full advantage of existing high-performance C libraries. In the future, MoonBit will offer standardized C language calling conventions, enabling the integration of existing open-source AI inference libraries for high-performance computing and edge AI inference, as well as for writing high-performance applications and services that directly call operating system APIs.

Compared to system programming languages like C/C++, MoonBit focuses on memory safety and reliability while maintaining high performance. MoonBit ensures memory safety primarily through compiler optimizations that remove the overhead of GC without compromising the developer experience. In key scenarios, MoonBit gradually introduces modality and other cutting-edge designs to enhance determinism. This approach lowers MoonBit’s learning curve, allowing more developers to benefit from its ecosystem.

Try MoonBit Native Today​

Download MoonBit, or install the MoonBit plugin in VS Code and follow the prompts for one-click installation.

Create a new project.

moon new hello-native
cd hello-native

Next, execute:

moon build --target native

And get target/native/release/build/main/main.exe, a binary executable file. This file can be executed directly.

$ ./target/native/debug/build/main/main.exe
Hello, World!

Or you can directly execute moon run --target native main.

What's next​

MoonBit native backend completes the final piece of its ecosystem. With support for native, WebAssembly, and JavaScript backends, MoonBit can now cater to the multi-scenario needs of most developers. Notably, with MoonBit’s modular toolchain design, all three backends share most of the foundational framework, meaning most of our frontend optimizations can benefit users across all three backends.

The MoonBit team will continue working on improving the user experience and performance optimziations. Starting next month, we will also start working on async support. We aim to reach the beta level of quality in the middle of next year. Follow us on X to get our latest updates!

More resources：

• Get started with MoonBit.
• Check out the MoonBit Docs.
• Join our Discord community.
• Explore MoonBit projects in the Gallery.

Recently, there has been a growing demand from the community for file I/O in MoonBit. In response, we have introduced a basic fs package in moonbitlang/x. The package, moonbitlang/x/fs, supports Wasm, Wasm-GC, and JS backends, and includes the following commonly used APIs:(Note: This library is currently experimental, and its API may undergo further updates and adjustments.)

• write_string_to_file, write_bytes_to_file
• read_file_to_string, read_file_to_bytes
• path_exists
• read_dir
• create_dir
• is_dir, is_file
• remove_dir, remove_file

Usage Example​

1. Declare Dependency

Execute the following command in the command line:

moon update
moon add moonbitlang/x

Or manually declare the dependency in moon.mod.json:

"deps": {
    "moonbitlang/x": "0.4.10"
}

2. Import the dependency in the moon.pkg.json of the relevant package:

"import": [
    "moonbitlang/x/fs"
]

test "write_and_read" {
  let path = "1.txt"
  // write content to the file
  @fs.write_string_to_file(
    path="1.txt",
    content=
      #|target/
      #|.mooncakes/
      #|
    ,
  )
  // make sure the file has been create
  assert_true!(@fs.path_exists(~path))
  let byte = @fs.read_file_to_bytes!(~path)
  // verify file content
  inspect!(
    byte,
    content=
      #|b"\x74\x61\x72\x67\x65\x74\x2f\x0a\x2e\x6d\x6f\x6f\x6e\x63\x61\x6b\x65\x73\x2f\x0a"
    ,
  )
  @fs.remove_file!(~path)
  assert_false!(@fs.path_exists(~path))
  try {
    @fs.read_file_to_string!(~path) |> ignore
  } catch {
    @fs.IOError::NotFound(_) as e =>
      inspect!(e, content="`1.txt` does not exist")
    _ => return
  }
  let bytes = Bytes::from_array([65, 97].map(fn(x) { x.to_byte() }))
  @fs.write_bytes_to_file(~path, content=bytes)
  assert_true!(@fs.path_exists(~path))
  let content = @fs.read_file_to_string!(~path)
  inspect!(content, content="Aa")
  @fs.remove_file!(~path)
  assert_false!(@fs.path_exists(~path))
  try {
    @fs.remove_file!(~path) |> ignore
  } catch {
    @fs.IOError::NotFound(_) as e =>
      inspect!(e, content="`1.txt` does not exist")
    _ => return
  }
}

For more examples, refer to fs/fs_test.mbt, which contains black-box tests of the fs library. These test cases intuitively demonstrate how external users should interact with the library.

Background: MoonBit Project's Testing Mechanism​

A MoonBit project can currently have three types of tests: white-box tests, black-box tests, and inline tests.

White-box Tests​

Written in *_wbtest.mbt. The build system will package and compile *.mbt and *_wbtest.mbt together from the current package, allowing access to the private members of the current package in *_wbtest.mbt. These tests can use dependencies declared in the import and wbtest-import fields in moon.pkg.json. The wbtest-import dependencies are only used in white-box tests and are not included in the final build artifact.

Black-box Tests​

Written in *_test.mbt. When compiling *_test.mbt, the build system will automatically treat its containing package as a dependency. *_test.mbt can only access the public members of its package (simulating the perspective of an external user). These tests can use dependencies declared in the import and test-import fields of moon.pkg.json (including the package itself, which does not need to be explicitly listed in test-import). The test-import dependencies are only used in black-box tests and are not included in the final build artifact.

Inline Tests​

These can be written directly in *.mbt (excluding the previously mentioned *_wbtest.mbt and *_test.mbt). They can access the private members of the current package and use dependencies declared in the import field of moon.pkg.json.

Additional resources:

• Get started with MoonBit.
• Check out the MoonBit Docs.
• Learn MoonBit with the open course.
• Join our Discord community.
• Explore MoonBit programming projects in the MoonBit Gallery.

In our previous blog on MoonBit supporting WASM-4, we introduced how to write a mini retro game using the WASM-4 framework.

The hardware specifications simulated by WASM-4 include a 160x160 pixel display, 64 KB of linear memory, support for keyboard, mouse, touchscreen, and up to four game controllers as input devices, audio output, and 1 KB of storage. A natural idea is to port WASM-4 to microcontrollers, replacing the simulated I/O devices with real hardware. In this article, we will explore how to run these mini-games written in MoonBit on actual hardware, using the ESP32-C6 chip, which features a RISC-V 32-bit single-core processor with a frequency of up to 160 MHz, 320 KB of ROM, and 512 KB of SRAM.

Below is a video showcasing the Snake game, developed with MoonBit, running on the ESP32-C6 development board.

The Porting Process​

The WASM-4 project repository provides two runtime environments: one for web applications and another for native execution. The WASM-4 runtime is an environment that can run games compliant with the WASM-4 specifications. The native runtime, implemented in C, can be modified to run on a microcontroller by removing unnecessary components.

LCD Display​

The WASM-4 native runtime comes with a graphical user interface (GUI) for displaying game graphics on the screen. However, when porting to hardware, we can replace this GUI with a real LCD screen, eliminating the need for the GUI components.

The first step is to remove GUI-related components. WASM-4 has an internal framebuffer for temporarily storing display data, which is rendered by MiniFB. We only need to keep this framebuffer for storing image data while removing the MiniFB-related code. Before debugging on the microcontroller, we can output the data from the framebuffer as image files. Once the images are confirmed to be correct, our task simplifies to displaying an array of pixel data on the screen. The display device used here is a 1.54-inch LCD driven by ST7789, with a resolution of 240x240, as shown below:

When centering the 160x160 image from WASM-4, there will be a 40-pixel margin around it. It’s also important to note that since the selected LCD only supports RGB565 color format, there may be color discrepancies.

Game Controller​

WASM-4 supports various input devices, including keyboards, mice, touchscreens, and controllers. In this article, silicone buttons simulate the game controller.

The controller has directional buttons (up, down, left, right) and two action buttons (X and Z). WASM-4 uses one byte in its linear memory to represent the controller state, where the 0th and 1st bits correspond to the X and Z buttons, while bits 4 to 7 correspond to the directional buttons. Each bit indicates whether the corresponding button is pressed. We set the corresponding bits in the button press event. For simplicity, each button is assigned a GPIO interface.

Replacing the wasm3 Interpreter with WAMR​

As the wasm3 project is no longer actively maintained and lacks some needed features, we will replace the WASM-4 native runtime's wasm execution engine with WAMR. Similar to wasm3, we only need to provide the same FFI in the WAMR environment. For example, functions for drawing lines (line, hline, vline), rectangles (rect), and blitting images (blit, blitSub) are required.

Conclusion​

At this point, we have implemented basic game display and control functions. You can view the complete code here. There are many exciting directions to explore further; for instance, this article has not fully utilized the rich wireless interfaces of the ESP32 chip. We could implement WiFi for hot loading of WASM game modules. Additionally, external speakers for audio output have not yet been integrated.

Additional resources:

• Get started with MoonBit.
• Check out the MoonBit Docs.
• Learn MoonBit with the open course.
• Join our Discord community.
• Explore MoonBit programming projects in the MoonBit Gallery.

If you have been dropping in MoonBit's repositories, you might have noticed a wierd bot called "peter-jerry-ye-code-review" that comments on every new pull request. It will also update the comment if any new commits are pushed. That's right, it's our own experimental code review bot written in MoonBit. Today we will explain how it works.

You can find a snapshot of the full code sources in its repository, and the README should explain the detail of the techniques.

Motivation​

When pull requests come in, we want them to be reviewed promptly. It would be helpful if some preliminary checks could be provided in real-time so that the creator can get instant feedback and update their work.

To achieve this, we use a bot that automatically reviews the pull request. It turns out that this is a perfect serverless task: the server gets notified by the pull request, retrieves the necessary information, and post a comment. Nothing needs to be stored. So, we're using MoonBit with components, as WebAssembly fits perfectly in these scenarios, with a bonus of a fast cold startup.

How it works​

The Architecture​

We develop this example based on the open-source runtime, Spin. It is built on the wasi-http standard and provides additional capabilities such as interfaces for connecting to Redis or relational databases. We are deploying our example on Fermyon Cloud, which offers a free plan for starters. Alternatively, one may choose the open-source Spin Kube and self-host it on a Kubernetes cluster.

With the wit-bindgen support for MoonBit, we can integrate Spin functionalities seeminglessly by grabbing the WIT files and generating the binding code.

The workflow​

The following diagram illustrates the workflow.

After creating an GitHub App and having it installed by the user, we can receive webhook events. GitHub sends webhook events based on configuration. Upon receiving the payload, we can verify it and gather the necessary information: the repository name, the installation ID, the pull request number, and the actual event.

With the installation ID, we can obtain an access token for our App, allowing us to access what the user has granted us, such as read/write pull request permissions, especially if it were installed in a private repository.

We can determine our actions based on the event. For example, "opened" means a new PR and "synchronize" means an update to an existing PR. For the former, we can create a fresh comment, while in the latter, we update an existing comment. The meaning and payloads are described in the document.

With the information in the payload, we can retrieve, from GitHub, what happened with the pull request. For example, we can obtain its changes in diff or patch format. We can list commits and retrieve the respective changes and messages, and so on.

We then send the gathered information to the LLM provider we've chosen via an HTTP request. The response is used to create or update a comment on the pull request.

Conclusion​

MoonBit, with Component Model support, can fit in many scenarios. With the increasing support for the standard from various runtimes, MoonBit can be widely applied to develop serverless applications or edge-computing applicaitons. This AI Agent is just one example of such applications, and there are more to come.

Additional resources:

• Get started with MoonBit.
• Check out the MoonBit Docs.
• Learn MoonBit with the open course.
• Join our Discord community.
• Explore MoonBit programming projects in the MoonBit Gallery.

On the 18th of August 2023 we launched MoonBit: a Wasm-first language toolchain focused on efficiency and simplicity. We received lots of positive feedback and worked hard to improve MoonBit and its toolchain.

Throughout this year, we've refined the language design, stabilized major features, and prepared for deep vertical integration of the toolchain with AI, with a focus on cloud and edge computing use cases. As we celebrate MoonBit's first birthday, we're proud to introduce the MoonBit Beta preview version, positioning MoonBit as an AI and cloud-native development platform.

Major Language Features Stabilized​

After a year of rapid iteration, MoonBit has established foundational language features on par with those of most mainstream languages at their 1.0 versions. With the release of the beta preview version, the language has reached a level of stability, with only minor changes expected. This stability paves the way for broader developer participation in building the ecosystem. Here are some of MoonBit's core and most challenging features:

Modern Generic System​

The design of type system is one of the most complex parts in modern language. Many mainstream industrial languages, like Java and Go, gradually introduced generics support many years after their 1.0 release, causing fragmentation within their ecosystems. MoonBit, however, has completed its generics and trait support in the beta version, offering zero-cost generics while maintaining extremely fast compilation speed.

Precise Error Handling​

If you’ve been following our recent updates, you'll notice lots of efforts on crafting rigorous error handling. In most programming languages, error handling isn't adequately supported during static analysis, resulting in untracked exceptions that can lead to potential reliability issues in software. MoonBit, through precise control flow analysis, tracks function errors entirely at compile-time. This process is almost entirely inferred by the compiler, sparing developers from cumbersome error-handling annotations, while preserving error safety.

Efficient Iterators​

Traditional programming languages often experience performance degradation with iterators due to frequent boxing operations, resulting in significantly worse performance compared to regular loops. MoonBit, however, introduces zero-cost iterators, enabling developers to write elegant code without compromising on performance.

Highlights​

MoonBit has been committed to leveraging its strengths in language performance, output size, and data processing.

Fast & Smallest Components​

MoonBit aims for full-stack efficiency, including both compilation and runtime performance. In our benchmark test, MoonBit compiles 626 packages in just 1.06 seconds, nearly nine times faster than Rust in terms of compilation speed, while the compute time is 1/35 of Go.

MoonBit has a significant advantage in reducing the size of Wasm code output. In our recent support for the Wasm component model, MoonBit achieves notable size optimizations in generated code. While compiling a simple "Hello World" HTTP server, MoonBit’s output file size is just 27KB, which is significantly smaller compared to 100KB for Rust, 8.7MB for TypeScript, and a massive 17MB for Python in WasmCloud's http-hello-world example.

Data-Oriented​

As a multi-paradigm programming language, MoonBit maintains simplicity while providing an optimal data processing experience. With native support for JSON processing, the Iter type, and pattern matching, MoonBit achieves safe and efficient data processing while the flexibility of dynamic languages and the safety and efficiency of static languages, enabling intuitive and concise data parsing and transformation.

MoonBit’s syntax design for data processing aims to resolve performance issues caused by generating multiple intermediate arrays. In the performance comparison of Iter, MoonBit’s data processing speed is 25 times faster than JavaScript.

AI-Native IDE​

In our initial release, MoonBit offered an all-in-one solution for development, debugging, and deployment. Beyond multi-backend support, a general-purpose language design, MoonBit provides a toolset including the compiler, build system, IDE, debugger, and deployment tools. In the past year, we refined debugging and testing support, open-sourced the standard library and build system, launched a package manager, and enhanced our cloud IDE with an AI assistant. Here are the key updates to our toolchain.

The initial release of MoonBit offered a Cloud IDE running directly on the edge. With a highly parallelized architecture and native support for separate compilation, MoonBit IDE can handle large-scale codebases without relying on containers, making it well-suited for cloud-native environments and edge computing needs.

In addition to traditional IDE features, the MoonBit AI Assistant is now integrated into the MoonBit IDE, providing automated test generation, documentation generation, and code explanation features. This comprehensive support for software development, testing, and documentation streamlines the development process, enabling developers to focus on more critical and creative aspects of their work without the burden of maintenance.

MoonBit has provided debugging tools typically available only in the mature stages of other languages. In the beta preview version, MoonBit supports out-of-the-box debugging in IDE. Users can instantly start debugging by executing moon run --target js --debug in the JavaScript Debug Terminal.

Application Scenarios​

MoonBit is a development platform covering all scenarios with a focus on cloud and edge computing. In each field, MoonBit aims for excellence, ensuring performance that outpaces other languages by at least an order of magnitude.

Cloud Computing​

With the recent component model support, our community member has developed an experimental MoonBit-Spin SDK. By breaking applications into independent, reusable components, MoonBit better utilizes computing resources in cloud computing development, enhances system flexibility and security, simplifies the scaling and integration process, and significantly improves cloud computing development efficiency while reducing operational costs.

Edge Computing​

With community support, MoonBit now has a real-world use case in edge computing through the Extism PDK plugin. The PDK support allows MoonBit to more efficiently utilize hardware resources in edge computing applications, enabling distributed computing and local processing. This improves performance, response speed, device compatibility, and data security, significantly enhancing development and deployment efficiency to meet the demands for low latency and high performance.

Roadmap​

MoonBit currently supports both Wasm and JS backends, with future plans to add support for native backends, aiming to cover all possible application scenarios. Whether it's UI development, client-side applications, edge computing, or system programming, users will find the right solution on MoonBit.

The MoonBit beta preview is less of an endpoint and more of a beginning. While there's still plenty of work ahead, we believe the most exciting developments won't come from the MoonBit team alone. Rather, we believe that this stable foundation will allow the MoonBit community and ecosystem to grow even faster than before.

Additional resources

If you're not already familiar with MoonBit, here is our quick guide:

• Get started with MoonBit.
• Check out the MoonBit Docs.
• Explore MoonBit projects in Gallery.
• Play with MoonBit IDE.
• Join our Discord community.

Wasm Component​

WebAssembly is a new low-level virtual instruction set standard for a sandbox model. It is low-level, meaning it is close to native speed. It is virtual, meaning it can be run on many runtimes, including the browsers or on the operating sytems with projects such as wasmtime or wamr. It is a sandbox model, meaning that it can not interact with outside world unless using FFI. The FFI, however, can only return numbers, making the usage through linear memory a more efficient way. Many programming languages can compile to it, including Java, JavaScript/TypeScript, Python, Rust and, of course, MoonBit.

So how can we combine the Wasm components that are implemented in different programming languages? Enter the Component Model, a proposal to unify the surface. With the Component Model, we define a high-level API, and components can be combined with other components as long as the interfaces match.

This blog will follow a step-by-step guide on writing a small HTTP server that prints "Hello World" with MoonBit and demonstrate how MoonBit achieves high compatibility and interoperability while significantly reducing output size.

How-to​

We will write a small HTTP server that prints "Hello World" with MoonBit. The prerequisites are:

• wit-deps

• wit-bindgen with MoonBit (You will need to build it from source for now, so Rust toolchain is also needed)

• wasm-tools

• wasmtime

• and of course, the MoonBit toolchain.

Define WIT​

First, you need to define the interface with WIT. See the manual for detailed explanations.

We specify our dependencies under wit/deps.toml. Today we are just using wasi-http version 0.2.0.

http = "https://github.com/WebAssembly/wasi-http/archive/v0.2.0.tar.gz"

You need to update the dependencies with wit-deps, and you'll see all the dependencies in the wit/deps folder.

Then we specify our "world", corresponding to the resulting Wasm, under wit/world.wit:

package moonbit:example;

world server {
  export wasi:http/incoming-handler@0.2.0;
}

A world may include other worlds, or import/export interfaces. Here we export the interface incoming-handler of wasi:http version 0.2.0, since as an HTTP server, exporting an incoming handler is needed so that the runtime can use it to treat the incoming requests and generate responses.

Generate code​

You need to generate the code with wit-bindgen. Currently, you need to install the project with:

cargo install wit-bindgen-cli --git https://github.com/peter-jerry-ye/wit-bindgen/ --branch moonbit

After having the wit-bindgen command, simply execute it with the proper subcommand (moonbit) and location of the WIT files (wit). There are also arguments to specify whether the types should derive the Show trait or the Eq trait.

wit-bindgen moonbit wit --derive-show --derive-eq --out-dir .

And here's what you will get:

.
├── ffi
│  ├── moon.pkg.json
│  └── top.mbt
├── gen
│  ├── ffi.mbt
│  ├── interface_exports_wasi_http_incoming_handler_export.mbt
│  ├── moon.pkg.json
│  └── worlds_server_export.mbt
├── interface
│  ├── exports
│  │  └── wasi
│  │     └── http
│  │        └── incomingHandler
│  │           ├── moon.pkg.json
│  │           ├── README.md
│  │           └── top.mbt
│  └── imports
│     └── wasi
│        ├── clocks
│        │  └── monotonicClock
│        │     ├── moon.pkg.json
│        │     ├── README.md
│        │     └── top.mbt
│        ├── http
│        │  └── types
│        │     ├── moon.pkg.json
│        │     ├── README.md
│        │     └── top.mbt
│        └── io
│           ├── error
│           │  ├── moon.pkg.json
│           │  └── top.mbt
│           ├── poll
│           │  ├── moon.pkg.json
│           │  ├── README.md
│           │  └── top.mbt
│           └── streams
│              ├── moon.pkg.json
│              ├── README.md
│              └── top.mbt
├── moon.mod.json
├── wit // contents ignored here
└── worlds
   └── server
      ├── import.mbt
      ├── moon.pkg.json
      └── top.mbt

The generated project has four folders:

• ffi and gen are the generated files to help with the Wasm bindings that you can safely ignore. The gen directory contains the entrance of the project.

• interface contains all the interfaces that are imported into the selected world. It is divided to imports and exports where the imports provide all the imported functions and types, while the exports is where you need to fill the stub functions that will be exported.

• worlds contains the world. Similar to interface, it contains a import.mbt, providing the imported functions and types of the world level, and top.mbt, containing the stub export functions.

Then you can develop as a normal MoonBit application, and moon check --target wasm should finish successfully. To see what APIs you have and the documentation of the types or functions, you can run moon doc --serve to see the documentation. Don't forget to execute moon fmt to make the code look better.

Develop​

Here's our code for a minimum Hello-World server for demonstration:

pub fn handle(
  request : @types.IncomingRequest,
  response_out : @types.ResponseOutparam
) -> Unit {
  let response = match request.path_with_query() {
      None | Some("/") => make_response(b"Hello, World")
      _ => make_response(b"Not Found", status_code=404)
    }
    |> Ok
  response_out.set(response)
}

fn make_response(
  body : Bytes,
  ~status_code : UInt = 200
) -> @types.OutgoingResponse {
  ...
}

See the full example on Github.

Componentize​

What we have achieved is a core Wasm, namely a Wasm following the WebAssembly standard. However, we need to turn it into a component so that it can be shared along with necessary information.

You need to use wasm-tools to embed the core Wasm into a component. First, we embed the WIT information into the custom section of our core Wasm. You need to specify the encoding as UTF-16 at this step. Then we turn the core Wasm into a component Wasm.

moon build --target wasm
wasm-tools component embed wit target/wasm/release/build/gen/gen.wasm -o target/wasm/release/build/gen/gen.wasm --encoding utf16
wasm-tools component new target/wasm/release/build/gen/gen.wasm -o target/wasm/release/build/gen/gen.wasm

You may also use JCO for this step if you prefer using npm or pnpm.

Use​

With the Wasm we created, you can serve it with Wasmtime:

wasmtime serve target/wasm/release/build/gen/gen.wasm

You may also use JCO to serve it on Node.js or Deno, or serve it with WasmCloud or Spin.

Compare​

We implement a simple HTTP server that simply replies "Hello World". Comparing with the http-hello-world provided by WasmCloud, we have the following sizes:

Conclusion​

We've demonstrated how to create a Wasm following the Component Model standard with MoonBit. The Component Model provides a new standard for creating interoperable Wasm. By pulling the WIT files from Spin, for example, we can easily build a serverless AI application in 5 minutes.

By supporting the WebAssembly component model, MoonBit enhances its application scenarios in microservice architectures and cloud-native applications with high compilation performance and compact code size, allowing for quick deployment and execution in various environments.

On August 18, MoonBit will reach the beta preview version, indicating language stability for broader testing and real-world use. In the future, we will continue to expand the MoonBit ecosystem and optimize documentation and toolchain for a better user experience. Stay tuned!

Additional resources:

• Get started with MoonBit.
• Check out the MoonBit Docs.
• Learn MoonBit with the open course.
• Join our Discord community.
• Explore MoonBit programming projects in the MoonBit Gallery.

If you have been visiting mooncakes or our gallery, you may have noticed that we've published a package called wasm4 and a new demo called Wasm4 Snake. Today I'd like to introduce you to this amazing game development framework.

What is Wasm4​

WASM-4 is a framework for building retro games using WebAssembly. It provides a gaming console with 160 x 160px, and less then 64K memory. Using WebAssembly, the new web standard of an instruction set, the game is able to be run on all the Web browser and even some low-end devices, and any language that compiles to WebAssembly can be used to develop the game, including MoonBit. We are proud to announce the MoonBit Wasm4 SDK.

How to develop​

Our gallery supports live reload where you can have a taste via our cloud IDE.

For developing locally, we expect Node.js and MoonBit toolchain to be downloaded.

Creating project​

We first create a new project with MoonBit in the current directory, and we install the wasm4 with npm

moon new --user moonbit --name demo --lib --path .
npm install -D wasm4

We will have the following directory structure (node_modules not included):

.
├── .gitignore
├── lib
│  ├── hello.mbt
│  ├── hello_test.mbt
│  └── moon.pkg.json
├── moon.mod.json
├── moon.pkg.json
├── package-lock.json
├── package.json
├── README.md
└── top.mbt

The moon.mod.json provides the definition for the whole project, and the moon.pkg.json provides the definition for each package. The top.mbt will be the main entrance of the game, and we can write helper functions in the lib, while the hello_test.mbt provides a blackbox testing example. We will not be using lib for this example.

Adding Wasm4 dependency​

We also need to add the dependency moonbitlang/wasm4:

moon update && moon add moonbitlang/wasm4

This will result in the following moon.mod.json at the time of writing:

{
  "name": "moonbit/demo",
  "version": "0.1.0",
  "deps": {
    "moonbitlang/wasm4": "0.2.0"
  },
  "readme": "README.md",
  "repository": "",
  "license": "Apache-2.0",
  "keywords": [],
  "description": ""
}

We need to modify the moon.pkg.json as follows to meet the requirements:

{
  "import": [
    "moonbitlang/wasm4"
  ],
  "link": {
    "wasm": {
      "exports": [
        "start",
        "update"
      ],
      "import-memory": {
        "module": "env",
        "name": "memory"
      },
      "heap-start-address": 6560
    }
  }
}

There are a few things to notice here:

• We import the pacakge moonbitlang/wasm4/lib as wasm4, so we will be using the functions and types with @wasm4 qualifier.

• We declare that we treat this package as a linking target for the Wasm backend with a few configurations:

  • We export functions start and update as required by Wasm4.

  • We import the Wasm memory to meet Wasm4's ABI, and the memory will come from module env with name memory.

  • We define that the heap for MoonBit will be starting from 6560 to meet Wasm4's ABI. The space lower than 6560 (0x19a0) is reserved for Wasm4.

We modify the top.mbt correspondingly:

pub fn start() -> Unit {

}

pub fn update() -> Unit {

}

Now we can execute with:

moon build --target wasm
npx wasm4 run target/wasm/release/build/demo.wasm

Or you may execute with the debug mode if anything goes wrong and you'd like to see the stacktrace with function names:

moon build --target wasm -g
npx wasm4 run target/wasm/debug/build/demo.wasm

And the browser should open automatically with a game display. There's nothing moving now, so let's add something!

Example: Moving block​

Let's draw a block on the screen:

pub fn start() -> Unit {

}

pub fn update() -> Unit {
  @wasm4.set_draw_colors(index=1, 2)
  @wasm4.rect(0, 0, 80, 80)
}

And this is what you may see. Wasm4 has four palettes and four drawing colors. Depending on the specific API, the corresponding drawing color will be used. What happens here is that we set the draw color 1 to the color of the 2nd palette, and then we drew an 80 x 80 rectangle starting from position (0, 0). Remember that the origin of coordinate of display is at the top left corner and that y-axis points downward in the world of programming.

The moonbitlang/wasm4 provides a high level abstraction so that you can write at ease. To avoid confusion, the indexes of draw colors and palettes start with 1. It is also possible to set each of the 160 x 160 pixels. Checkout the Wasm4 document and the SDK API for more information.

We now have a block that sits still. But we are developing a game, so we'd like to make it move. The start function will be called once during initialization, and the update function will be called at 60Hz. So we can write like this to make it move

struct Position {
  mut x : Int
  mut y : Int
}

let pos : Position = { x: 0, y: 0 }

pub fn update() -> Unit {
  if pos.x + 80 <= 160 {
    pos.x += 1
  }
  @wasm4.set_draw_colors(index=1, 2)
  @wasm4.rect(pos.x, pos.y, 80, 80)
}

And it will become (though much faster than the screenshot):

Reacting to User Inputs​

A game will have to interact with the user somehow. And Wasm4 provides two buttons (X Z) in addition to four direction buttons. Let's try to move at your will!

pub fn update() -> Unit {
  if @wasm4.get_gamepad(index=1).button_right && pos.x + 80 < 160 {
    pos.x += 1
  } else if @wasm4.get_gamepad(index=1).button_down && pos.y + 80 < 160 {
    pos.y += 1
  } else if @wasm4.get_gamepad(index=1).button_left && pos.x >= 0 {
    pos.x -= 1
  } else if @wasm4.get_gamepad(index=1).button_up && pos.y >= 0 {
    pos.y -= 1
  }
  @wasm4.set_draw_colors(index=1, 2)
  @wasm4.rect(pos.x, pos.y, 80, 80)
}

More with development​

For debugging, you can use @wasm4.trace to write debug messages to the console. You can also press F8 to see the details of what is happening, as in the previous screenshot.

For publishing, you can execute npx wasm4 bundle --html game.html target/wasm/release/build/demo.wasm to generate a standalone HTML page. With a static file server, people will be able to enjoy the game you designed.

Notice that Wasm4 supports up to four players at the same time over network without extra configuration. This means you may be able to create your own snake duel of the Zenless Zone Zero and enjoy it with your friends! Check out the Wasm4 document and the SDK API for more information.

Conclusion​

What are you waiting for? Try to develop with our gallery which supports live reload. Enjoy!

Additional resources:

• Get started with MoonBit.
• Check out the MoonBit Docs.
• Learn MoonBit with the open course.
• Join our Discord community.
• Explore MoonBit programming projects in the MoonBit Gallery.

Moon: MoonBit Build System​

Moon, the build system for MoonBit, is now publicly available via the Moon GitHub repository under the AGPL License. Moon provides compilation, automated testing tools (including integrated expect test), coverage testing, package management, and more for MoonBit projects.

MoonBit integrates a comprehensive toolchain from the start, providing a streamlined coding experience with its compiler, cloud IDE, build system, package system, and AI assistant. As an essential component of the MoonBit language compilation toolchain, Moon integrates closely with the IDE, offering detailed project structure and dependency information for code analysis within the IDE.

Written in Rust, Moon benefits from Rust's memory safety, high performance, concurrency capabilities, and cross-platform support, ensuring a stable and fast build process.

Moon's parallel and incremental build capabilities are bolstered by the n2 project (both n2 and ninja were created by Evan Martin, with n2 being more lightweight and excelling in incremental builds). Modifications to n2 will maintain the original open-source license, see moonbitlang/n2.

Why Choose Moon​

1. Speed

MoonBit's compiler is incredibly fast due to a meticulously designed compilation process and optimization strategies. Acting as the bridge between the user and the compiler, Moon aims for a streamlined design, minimizing its own overhead to maximize compilation speed.

Additionally, Moon provides IDEs with comprehensive project structure and dependency details, crucial for latency-sensitive IDE environments. By optimizing the performance of core build steps, Moon ensures a smooth user experience ven in highly interactive development environments.

2. Parallel Incremental Builds

Bolstered by the n2 project, the parallel incremental build feature of Moon is key to its efficiency. By automatically analyzing and understanding dependencies among build tasks, Moon intelligently parallelizes independent tasks, fully leveraging the power of modern multi-core processors to significantly speed up builds. Importantly, Moon only rebuilds files that have changed since the last build or have updated dependencies, greatly enhancing build efficiency and making Moon capable of handling large projects that require frequent builds.

3. Integration and Testing Support

Closely integrated with automated testing tools, Moon can automatically execute unit tests, integration tests, and end-to-end tests during code submission and builds, ensuring every line of code is rigorously vetted.

For code quality assurance, MoonBit provides code formatting and static analysis tools that automatically check for consistent code styles and identify potential logical errors and security vulnerabilities. These features are especially crucial in CI/CD pipelines, allowing for early detection and reporting of code quality issues before code is merged into the main branch, ensuring the team can collaboratively develop high-quality code.

Benchmark Performance​

Build Matrix Performance Testing​

We tested moon against Rust's Cargo and Go's build system in compiling projects with complex dependencies. The test involves the generation of DR _ DC directories, the "directory matrix", and each directory contains MR _ MC modules, the "module matrix". The module in row r and column c of the module matrix depends on all modules in the previous row of the same directory. The first row of modules in a directory depends on all modules in the preceding row of directories.

The test setup also permits a lot of parallelism for actually executing the rules: the modules in the same row can be compiled in parallel, as well as the directories in the same row. For detailed testing criteria, see omake1, and the project generator code is available at moonbit-community/build-matrix.

In our tests, with DR, DC, MR, and MC all set to 6 and the main module, each project yields 1297 (6^4 + 1) packages. The test environment was a MacBook Pro Apple M3 Max with 128GB RAM running macOS 14.4.1. Results were as follows:

Debug Build: Moon performed excellently, taking 2.3 seconds. Go took 2.9 seconds, and Cargo was the slowest at 20.0 seconds.

Type Check: Moon was the fastest at 1.4 seconds. Cargo took 16.2 seconds. Go lacks a direct equivalent for type-checking commands like moon check and cargo check, so the result was -.

Release Build: Moon excelled again, taking only 1.6 seconds. Go took 3.1 seconds, and cargo build --release failed to complete due to memory exhaustion, resulting ∞.

Notably, Moon's release builds were faster than its debug builds.

For projects with DR, DC, MR, and MC all set to 8 (4097 modules total), moon build took 5.7 seconds, go build took 11.2 seconds, and cargo build took 1043 seconds. In this test, both Moon and Go completed in seconds, while Cargo could not finish the build within a reasonable timeframe.

Standard Library Performance Testing​

Currently, moonbitlang/core is the largest MoonBit project. As of 2024/07/03, it has 38,177 lines of code, 46 packages, 195 .mbt files, and 2576 tests. Type checking the project takes only 0.28 seconds, and running all tests takes just 1.27 seconds.

Try Moon Now​

Download the MoonBit toolchain via the MoonBit CLI tools installation script, or install the MoonBit plugin in VS Code and follow the prompts for one-click installation.

Usage: moon help

The build system and package manager for MoonBit.

Usage: moon [OPTIONS] <COMMAND>

Commands:
  new       Create a new moonbit package
  build     Build the current package
  check     Check the current package, but don't build object files
  run       Run WebAssembly module
  test      Test the current package
  clean     Clean the target directory
  fmt       Format moonbit source code
  doc       Generate documentation
  info      Generate public interface (`.mbti`) files for all packages in the module
  add       Add a dependency
  remove    Remove a dependency
  install   Install dependencies
  tree      Display the dependency tree
  login     Log in to your account
  register  Register an account at mooncakes.io
  publish   Publish the current package
  update    Update the package registry index
  coverage  Code coverage utilities
  bench     Generate build matrix for benchmarking (legacy feature)
  upgrade   Upgrade toolchains
  version   Print version info and exit
  help      Print this message or the help of the given subcommand(s)

Options:
      --source-dir <SOURCE_DIR>  The source code directory. Defaults to the current directory
      --target-dir <TARGET_DIR>  The target directory. Defaults to `source_dir/target`
  -q, --quiet                    Suppress output
  -v, --verbose                  Increase verbosity
      --trace                    Trace the execution of the program
      --dry-run                  Do not actually run the command
  -h, --help                     Print help

How to Contribute​

We welcome various forms of contributions from the community, such as documentation, testing, and issues. For detailed information, please refer to the contribution guide.

MoonBit Open Source Roadmap​

MoonBit was officially released last year, and we opened the standard library to the public on March 8th this year. Thanks to enthusiastic contributions from the community, a complete data structure library has been successfully implemented, further enriching the application scenarios of the MoonBit language and maturing its ecosystem. Following the open-source release of the build system, we will release the MoonBit Beta preview version on August 18th next month, marking a relatively mature stage for the MoonBit language, suitable for early users and developers to develop and test actual projects. By the end of this year, on November 22nd, the core part of the MoonBit compiler will be publically available.

Additional resources:

• Get started with MoonBit.
• Check out the MoonBit Docs.
• Learn MoonBit with the open course.
• Join our forum and Discord community.
• Explore MoonBit programming projects in the MoonBit Gallery.

In the realm of software development, testing is an essential step to ensure quality and reliability. Therefore, testing tools play a critical role in this. The simpler and more user-friendly testing tools are, the more developers are willing to write. While manual testing has its place, the deciding and typing task can be painful enough that it actually discourages developers from writing tests.

What is an efficient testing tool then? In the blog “What if writing tests was a joyful experience?”, James Somers introduces expect tests which make printing itself easy and save you from the daunting task of writing tests by hand.

To address this need, the MoonBit standard library introduced the inspect function, which we call expect tests, aiding in rapid writing tests. MoonBit's expect testing improves testing experience even better than that of OCaml and Rust, as it operates independently without the need for any external dependencies, enabling direct testing out of the box.