RustFest Barcelona – Niklas Reppel: Amp Up Web Audio Applications with Rust and WebAssembly

RustFest Barcelona – Niklas Reppel: Amp Up Web Audio Applications with Rust and WebAssembly


NIKLAS: Hello? Good morning. Thanks for having me here. I’m Niklas, working for the local research
and development centre of Catalonia, Eurecat, and my job is making audio software, nowadays
still in C++, and I’m fairly new to Rust but very well related to Claudia Saxer’s talk
yesterday. My journey was pretty similar in the first
few months. Today I’m going to talk about the web audio
and how Rust can be used for this but also a little bit why Rust might be a good option
for audio software in general. So why would you want to put complex web audio,
or complex audio applications on the web in the first place? I’m part of a community that does music with
code. I don’t know – who does not know live coding
here? Everybody knows it, I’m surprised. Small programming languages, with domain-specific
languages, and one of the problems that this community has is that workshops often end
up being installation workshops and after two or three hours people manage to their
things which is kind of sad because the stuff is fairly easy to use and it’s a lot of fun
to use and might be educational, a lot of people might get interested in the programming
but if you need two days just to install, I don’t know, whatever weird environment people
develop the languages on, it might discourage. So if you can put it on the web you just need
to load a website, you don’t need a lot of computer literacy and it may just lower the
boundary for people to use your stuff and might make it more fun for more people. I went on a journey to try to put my own stuff
to the web and I’m still fairly early in that process, but I was looking for the perfect
web audio stack, so what are the requirements for audio software in general? You need a deterministic way of working for
the audio thread itself. That means you don’t – you shouldn’t allocate
memory in the audio thread. You shouldn’t lock, you shouldn’t wait on
conditional variables and all those kind of things, and it is inherently multi-threaded
because you always have the audio, the realtime thread and some control thread at least to
either set parameters or schedule stuff, sequencing stuff, and so on, and so on. Out of personal preference what would be nice
to have not only for the within but for audio in general, memory safety is a thing because
you suddenly get random memory contents pumped directly into your ear, what at work we affectionately
call noise from hell, then memory safety becomes very important suddenly, just to be able to
keep on working and not being deaf. Portability is a thing because for me, if
at some point I would not be happy with the web performance anymore and want to have a
native programme, it would be nice if it would be easy to port. Small footprint. I don’t like bloaty software, and the easier
it is to use, the better. So options here, eked go with pure JavaScript
but it’s not that portable, I’m not an expert in JavaScript, but I think compiling it to
a native executable is something different. It’s a dynamic scripting language. That means it has probably something like
garbage collection which in some languages can easily stop the audio for a second, you
get glitches, and it gets easily bloaty. I don’t want to chime into JavaScript bashing
here, I like JavaScript, but I don’t think it’s the best for this kind of purpose. You could just compile C++ with emscripten
but that’s neither very convenient or very comfortable, I don’t know. So I came across Rust and WebAssembly and
that seemed to be a fairly good option. Why? Well, so first of all I looked around and
tried to find something that has already been done and I came across this little synthesiser
that’s not made by me but by some person from Japan and that was the proof of concept that
showed me: okay, it’s actually working. It’s like a little TV303 simulation, kind
of funny. [Music]
Even though it’s still lacking sample playback and precise timing and all things that need
to be done, but apart from that everything that’s written about it is in Japanese, which
is a beautiful language of which I have very little command unfortunately, so I need to
put some effort into that. So yes, why use Rust for audio applications
in the first place? And for web audio applications in particular? So the memory aspect, memory safety aspect
I already lost a few words about that, why you would want to have that in audio software. Rust boasts itself of fearless concurrency
so for multi-threading applications it might actually be pretty good. It’s a deterministic systems programming language. Gives you stable results. You don’t have any garbage collection to deal
with and don’t have to expect glitches. You can port it to the web or if you are not
happy with the web audio software anymore you could easily compile it to a native executable. The WebAssembly integration is very seamless,
compared to C++ or other languages where you have to struggle with emscripten or whatever. For Rust you just set a different target and
there you go. And last but not least the performance. I only did some very casual profiling in the
developer console but looking at the numbers the main audio processing method needed roughly
15, 20% less than the native JavaScript applications to test sample playback and to work around,
in general get used to working in Rust and WebAssembly. This usually needs headphones so it doesn’t
do a lot on a single speaker but it’s a very simple sample player and for profiling the
stuff I did the same thing in pure JavaScript and it seems to be more reliable with Rust
and WebAssembly. So what are the ingredients of an application
made with Rust? First of all you need a module compiled to
a WebAssembly module. Bindgen doesn’t really work that well in that
context, so it’s a raw library that you compile to WebAssembly. You need the Audio Worklet that is part of
the Web API that allows you to do custom processes that can use a WebAssembly model. You need some setup and control script, obviously
the main thread, the user interaction. If you want to do any scheduling, any time-based
stuff, you need some additional WebWorker thread for sequencing and these kind of things
and ideally some pure Rust DSP crates that easily comply to WebAssembly as opposed to
bindings, like if you want to do some advanced transform libraries like FFTW or whatever,
it’s probably harder, but with the pure Rust crate that works pretty well. What are the challenges of making a web audio
application with Rust? The Audio Worklet and WebWorker stuff is a
little bit tricky because you cannot load the models directly. You have to fetch them in the main thread
and then post them using the message system, but once you’ve got through that, once you’ve
got the setup done, it’s fairly easy. Setting up the shared memory between the WebAssembly
and the main JavaScript is a little bit tricky, and as I already said, it was the bindgen
for the worklet part itself, exactly because of the limitations of the Audio Worklet that
you cannot load the module directly in the work that you have to post it there in binary
data. What’s already there? Crates that support or that help you creating
audio software with Rust? There’s chfft, a pure Rust library if you
want to do convolutions. I use that followed by normalisation of this
part, like for the simulation of the 3D impression, and this if you want to check this out with
headphones and play around with it a bit, the current version is still time-based convolution
but the new version that I might upload soon is based on actual fast convolutions. There’s an alternative called Rust FFT but
I haven’t had a lot of luck with that. Basic.dsp crate which I wasn’t very happy
with, it’s a very vector-based way of thinking and it’s a little bit too detailed maybe but
anyway it exists so you might want to check it out if you are into dsp and want to do
some audio software with Rust. What’s New York working all that well yet? I haven’t managed to do any threading directly
from Rust, so in the sense that I want to spawn a thread to do scheduling tasks, so
scheduling music events, so you still have to rely on the WebWorkers I think, unless
somebody knows something more, and I’m willing to learn because as I said I’m still fairly
new to that in the first place. Debugging is a bit tricky, as we already learned
in the talk yesterday. You should test your dsp code well to make
sure it runs because to debug it directly in the WebAssembly part is fairly annoying. And unfortunately Audio Worklet is not yet
available in Firefox. So far it only works in Chrome and Opera,
which is a bit sad and I hope it will be implemented in a proper way in the future in Firefox as
well, but we are not there yet. So what is to be done? One of my future tasks in general is to do
some general benchmarking because I want to know how the Rust software on the web in particular
but also in general compares to my C++ stuff which I haven’t done yet, especially regarding
fast Fourier transforms which is a central thing in my daily work. To do past convolutions crate, to be able
to do re-works on the web and other things, and in general I had hoped maybe somebody
would be inspired to do audio software and we can sharpen Rust’s profile as an audio
language in general, so I can avoid in the future having random memory contents pumped
into my ear directly. As I already mentioned, that’s not a very
pleasant experience. So yes, some resources, there’s a Rust audio
forum, this discourse group. The Rust audio environment is actually bigger
than you might think. There is a VST crate that you can use to do
plugins for audio work stations, which I haven’t used so far but I know it exists, so if you
want to look for any resources, that’s where you would go. This is for some examples, if you want to
learn from that, especially with the WebAssembly part. There is this original seminal example that
I already presented. It’s all in Japanese, if you speak Japanese,
there’s stuff written about that. If not, I did the sample player. It’s well commented so it might be a nice
starting point for your own applications if you want to check that. And last but not least, a more complex example
that I made and as you say in my native language [German language] I might give you a little
demo of that, what life coding means and what you can do with Rust audio so put the microphone
away and I will give you a little impression of that. [Music]
[Applause] Thank you for listening. I don’t know if there’s time for questions,
or anything else? All of this is running in a web browser. It doesn’t need any server backend, it just
needs a web page and there you go.>>Hi, an amazing talk and demonstration,
thank you very much. Last time I checked for web audio in Firefox
I think there was a limitation of 256 milliseconds on latency. I don’t know how it is now but, depending
on which kind of application, latency is important. Also what is the difference between Audio
Worklet and web audio? NIKLAS: I would assume it’s getting less and
less. 250 milliseconds sounds quite a lot. I must admit I’m not – my own work is mostly
based just on output, not on input. I don’t do live processing where this would
be very important, but I’ve seen applications that do guitar effects on a web page and they
are fast enough so that you can play it with the recent advances. I don’t know if I could find it but there
are guitar effects that run on the browser that are at least fast enough to test them
and not completely bring you out of the rhythm when you bring guitar, so I assume that would
be better now.>>Hi, a great talk by the way. NIKLAS: Thank you.>>In the Ruffbox play-through there was a
sort of scripting language that you typed and it made the music. NIKLAS: Yes.>>What is it? NIKLAS: That is just a very, very simple pattern,
basically a sequencer that I made to test it. It doesn’t have big functionality. That’s my next project basically, to make
an actual more complex language for the synthesiser to use it more productively, or to port, I
don’t know, my existing stuff to these kind of things, and maybe other stuff. There’s a lot of nice live coding languages,
so that’s a future project.>>Good luck. NIKLAS: Thank you.>>Final question. Raise your hand.>>I’m curious if it would be technically
possible to do this kind of audio processing with the main web audio without Audio Worklet
and if you can explain why this needs Audio Worklet maybe it would help motivate implementation. NIKLAS: There’s always multiple ways to do
things, but yes, one of the reasons for me is that, for example, if the performance isn’t
enough, I can use it natively as well and in different contexts, which might not be
that great with pure JavaScript. Of course, everything of that could be done
in pure JavaScript or with different text apps but I think the Web API itself and the
nodes it currently provides, it’s not that flexible. For example, all the timing stuff you saw
here is done in the Rust part. The precise time, the scheduler, the pattern
language only gives out very coarse events and they are requeued and re-ordered in the
Rust part, and so on, and so on, so I think the complexity might just be easier to handle
if you have a systems programming language there instead of the basic web audio API facilities,
but of course that’s all personal preference, so there are of course different ways to do
it, and whatever suits your needs, you can use it.>>Thank you, Niklas. Give him a round of applause. [Applause]

Leave a Reply

Your email address will not be published. Required fields are marked *