The Weird Web

00:15 Okay. Let's do this. Hi, everyone. My name is Alex. I make fancy websites for an agency called EchoBind. You should hire us if you want us to help you build your fancy websites. And I love the web. And I want to talk how we are truly in a golden age of web APIs.

00:37 You know what makes the web so great? It is universal. It is worldwide. That is its name. And we've all felt how cool it is to code up a website and send a link to our friends and say, Hey, check this out. And the best part? They didn't even have to download or install anything special.

00:54 It is just there on the web. They can access it through their web browser. And the fact that you can do this is in large part because of the web advocates who work tirelessly to add APIs and maintain compatibility between all of the browsers that we use every day.

01:08 Because this is how the web has always been, right? This wonderful utopia of all the browsers getting along. Yeah. But there might have been some things you wanted to do in browsers that you just couldn't do. After all, they are basically remote code execution engines with cute logos. So, browsers need to take security and privacy very seriously.

01:29 That means that unlike desktop apps, web browsers can't just write to the file system or open an HTTP server or send UDP packets. There's probably a joke there, but you might not get it.

01:43 Yeah. I don't get it. Don't worry. That's the joke. So last year, I gave a talk about how I wanted to create a local first timer website that would send you push notifications when the timer was done without needing a server.

01:58 Service workers exist to allow you to send push notifications to web browsers, even if you aren't on the website. And it feels like something you should be able to do, but it is completely and totally impossible to do without a server. And that was really disappointing to find out. Bummer, dude. The web is limited.

02:17 We get around that by using desktop apps, wrappers, like Electron and Tauri, that combine a browser with a native back end. And that gives apps access to just about anything on the computer. And these are both great. I've used both of them. Love them both.

02:30 But they break the promise of the web, the promise that you can just send someone the URL, and they can go to your website, and they don't have to download anything special. People have been trying to add more capabilities to websites for a really long time. Has anyone, by raise of hands, who's heard of future Flash animator?

02:48 Anyone? Oh, you might know it by its new name. It's called Flash. Anybody heard of that one? Yeah. Yeah. That and Java applets were the OG browser plug ins for making websites do things they shouldn't really be able to do.

03:01 So, from the 1990s to 2017, Flash could access the camera, file system, do all kinds of other things that were impossible with web browsers. Really handy stuff. It's just too bad that it was slow and insecure and lacked privacy, controls. So, Flash is dead. Anybody remember Google Gears? Anybody? Yeah.

03:22 Back in 02/2007, Google released another plug in that basically sideloaded a whole bunch of new web APIs into browsers like local storage and JavaScript workers and geolocation. Wow. What great ideas. And it took a long time, but eventually browser engines realized there was so much that they could provide website authors without compromising on privacy and security.

03:43 And so, we're living in that golden age of web APIs where you can access media devices like the camera. Hi. And microphones. You can store entire L. L. M. S inside a browser's local storage and access them using web GPU. You can pinpoint your physical location. And I promise that is using the geolocation API.

04:01 I didn't just put it in Utah because I knew we'd be here. And you can render three d graphics. It's great. Golden age of web APIs. We are getting to the point where in web browsers you can just do things. And I'm serious here. These aren't the only web APIs that have been added to the platform.

04:19 There are some other weirder APIs that you've probably never heard of and I would bet you've never used. And I want to emphasize, these are all very real APIs. They exist in my browser. Hopefully, well, probably your browser, statistically speaking. But they are documented on NVM. And they have W three C working group specifications.

04:39 So, I'm not making these up. And we're going to start by talking about sound. This is a waveform. The sounds that we hear are waves that propagate through the air. And our ears pick up the frequency of the wave as pitch and the amplitude of the wave as loudness.

04:55 Computers encode this as a sequence of numbers representing amplitude at a given time. So, So microphones detect those tiny fluctuations in air pressure and stores them. You pump those numbers into a speaker at the right rate, and, hey, you've got audio.

05:07 To give you an idea of how much data this is, audio CDs use 44,100 amplitude values to store about one second of audio. That's pretty, pretty good fidelity. I mean, any audio files in here? I'm sorry. The rest of us don't have to worry about weird noises as we're listening to our music. We just enjoy it.

05:29 Sorry, that was that was mean. I shouldn't be mean. Yeah. Playing audio in the browser is easy. Anybody ever use the video or audio tag? Whew. It's so easy. Aren't we so glad to have those tags? Finally.

05:43 But what if we wanted to take those amplitude values and mess around with them a little bit before turning them back into sound? That is what the Web Audio API lets us do. It lets you take audio sources, like microphones, audio files, generated waveforms, it just makes the waveforms itself.

06:00 And then you could process them using filters or transformations, like adding reverb or changing the pitch. And then you can output them through speakers or headphones. So here's a little demo. This is a keyboard. And I can click here to play notes. If I can find my cursor.

06:15 I suppose I can click on the one that you're seeing.

06:20 Hey. Yeah. Fun fact. I'm actually learning to play the piano, and I was this close to bringing my keyboard, but it seemed a little extra. So, yeah. And the way that this works, there's a package on NPM, that has definitions for a whole bunch of different types of instruments.

06:36 So I can come here and change this to, the Clavi. What does that sound like? I'm sorry, everyone. I should have picked something different. We'll go back. Yeah. Yeah. So, it's able to generate the waveforms for each of these instruments using, by combining them together.

06:55 You literally just add the waveforms together and it creates new sounds with different timbres and stuff. And it though it does sound like bad MIDI from the eighties and nineties because that's where these different waveforms were defined. Speaking of MIDI, I'm just gonna pull this out.

07:20 I couldn't bring my keyboard, couldn't bring myself to do that, but this is just a little bit more portable. So this is a MIDI interface. It allows me to connect to a computer. In fact, that's what MIDI is all about. Musical instrument digital interface.

07:38 So it was invented in the eighties as a way to connect digital instruments like this one to each other or to computers. So if we go back here, I can do oh, let's try that again. I might need to refresh my browser. Oh, no. The demo gods are not looking down upon me with joy.

07:59 Give me a second. I'm gonna,

08:04 plug directly in.

08:10 Really missing that fourth USB C port. Okay. Let's try this again.

08:18 There we go.

08:33 I I have the the drums at the beginning too. Yeah. That's all you get. Sorry. I'm not good at the piano. I'm learning it. So yeah. That's MIDI. It lets computers talk to, musical instruments and vice versa. And, it's built into browsers. So this right here is what MIDI looks like in a browser.

08:57 You can see these messages as I push the buttons. It's got these note on message types that it's sending. It's just like pardon? Oh. I need to click on the right one. Thank you. There we go. Well,

09:14 hang on. Hang on. There.

09:19 Okay. So, you can see all of the different messages that come in. As I drag the slider it says pitch bend change. So the way that it works is, oh, man, these are finicky slides. That's perfect.

09:33 The way that it works is, just like a keyboard event emitter, you listen for events on a given MIDI device after you've connected it to your computer. And when you get the events, this is what it looks like, and then you're able to do stuff with it. So I've got control change like this. I've got the buttons.

09:47 Note on for when you push the key. This one doesn't support note off, unfortunately. Like, that seems kind of important, but whatever. Yeah. So I was using it earlier to trigger the Web Audio API, but you can control whatever you want, like just making it print to the screen like this.

10:01 Or this right here where I'm interpolating this, this, gradient using Okay Lab, which isn't the web great? We've got these beautiful gradient interpolation algorithms. And, also you might have noticed this thing that I brought up with me. This is a gamepad, so you can see that it also can control it.

10:22 This is using a different API called the Web Gamepad API. And it's actually kind of funny if you think about it. Did you know that this code, this is in Chromium, and it teaches it how an Xbox controller works?

10:37 Like, your web browser knows what an Xbox is, which is really weird if you think about it, but kind of necessary for making it so that Xbox controllers work with your web browser. And it's but you can connect the gamepad and do stuff with it. It works a little bit differently from the web MIDI API.

10:55 Instead of having an event emitter where it sends events to you, you actually have to pull the device in order to get its current state. You set up a loop and just keep asking it, what's your state? What's your state? What's your state? And then it tells you.

11:06 Which is not a huge deal for games because they usually have game loop anyway. But the cool thing is you can also send events to gamepad devices, like haptic events for making it rumble when, I don't know, something happens where it needs to rumble. So, yeah, that's in every single browser.

11:22 So remember, I can use this to control oops. That's not controlling the color of this box. I think this one does. Yeah, that one does. Okay. Because I, actually do have another demo for you. I have this little remote. And if I just point it there, I have just taken control of that light. Oh, no. Tragedy.

11:45 Oh, yeah. Because I need to push my special button. Special button. And oh, oh, we're gonna try one more time.

11:58 Nope. Okay. We'll do it on this browser.

12:02 There. This is the special button that you can't see. But well, now you can see it. Oh, no. Tragedy. Okay. Give me just half a second to try to make this work.

12:22 My goodness. This is why nobody likes software developers. We're all too literal.

12:34 I I like you all. Don't worry. I think you're lovely. Okay.

12:41 Oh, that's brutal. Is it just because my name is Alex? Yeah. Okay. Here we go. I can now control this light.

12:54 Yep. So here's the light. I'll make sure that everybody on the live stream can see it. Don't fall. Yeah. So I can control this light. There's red, there's a nice green, And there's some blue. And this one's really cool because it has ultraviolet. Oh. That's, yeah, it's working. Yeah. There you go.

13:17 Can't hurt you. Come on. This isn't Fyre Fest.

13:25 I have another light.

13:29 This one, I think, should just work. We'll see. Maybe. Yeah. It's it's got something. Let's see if it actually is controllable. No, not right now.

13:44 Maybe.

13:47 That's okay. One demo is good enough. I'm not going to push my luck. Okay. So we've got oh.

13:55 Yeah. Maybe I really shouldn't push my luck.

14:00 We've got this DMX light. That's that's the protocol that these lights use in order to communicate with each other. It's called DMX. Digital multiplex, to be clear. Let me just get this running one more time. Just to see. Yeah. There we go. Okay. We'll set it to a nice blue color like that. There we go. Okay.

14:22 So, this is, using a protocol called DMX five twelve.

14:26 It is basically a bus where these lights have channels for each of the different colors and the different functions, and each device gets a number, and then you just daisy chain the devices together, and you can have up to five twelve channels on a daisy chain, or in a universe is what they call it in the industry.

14:44 Hence, DMX five twelve. And, this is what this particular device's control setup looks like. So, it has seven channels for controlling all kinds of different colors of the light from zero to two fifty five. And, then a couple of other channels for special effects and whatnot.

15:03 And the cool thing about DMX is you can use it for all kinds of different things, not just lights with colors, but you can also use, smoke machines or motors or outlets. All of these lights that you see here around us are controlled with DMX. DMX is everywhere. It is in the room with us.

15:21 And so how do I get my browser to speak some arcane protocol like DMX? Well, with a fancy little API called Web USB. It's called Universal Serial Bus for a reason. And it works on the web. Asterisk. It gives me access to USB devices. You saw that.

15:37 And I can send commands and listen to messages from the devices I allow. So, I'm gonna guess, has anyone ever used WebUSB? Raise your hand high in the air. Oh, nice. You guys are cool. Everybody else is cool too. Just in your own different way. You can be cool for more than one reason.

15:55 So, let's dig in a bit just to give you an idea of what this is like. This is the web USB API. You call navigator.USB. Request device, and you're able to pass it filters that say which specific vendor and which specific product you want to be able to connect to.

16:11 That way you can't just, if you have other USB devices, which I do, you can't just randomly connect to them for your specific application. You're expecting a specific type of device. And then you open the connection and, claim an interface which says, I want to talk to you on this. It's like a port.

16:27 An interface is like a port. We know what ports are, right? Port zero. Then we have to this is what pops up after that. You saw that. I choose the device I want to click on. This is how security works with web USB. You just say this is the one I want to connect to.

16:42 And then you do a little bit of ceremony as per the USB spec to transfer control to the device and say, I want to control you. I'm going to use the interface protocol. Here's the request to set the control line state, whatever that means. And then we're ready to go. We can send messages.

17:00 This is how this particular device that I'm using, which it's taped down, I'm not going to show it to you. But it's a small little box that connects with USB and then outputs DMX commands, and I can send these different bytes to it to tell it what to do.

17:14 So, I say, we'll start with this start code, and here's the start of the message, and then here is the length of our universe, and then for every item in that, we're going to just tell you what value it should be. So, remember, it's from zero to two fifty five, one byte. Handy that, right?

17:31 One byte for every single channel that goes out. And then I can write a wrapper that makes it a little bit easier for me to send these commands by just saying, here is an array of five twelve items with what each of the channels should be, send it out to the DMX, and that is what happens.

17:46 It'll update to whatever values I set here. And I'm not the only one to put WebUSB to work. The Google Stadia team anybody have Google Stadia? You'll love it when your controller got bricked. When when Stadia shut down? Stadia shut down.

18:03 They had these controllers that you could use to play games through this service, but you couldn't use them anymore, so they issued a firmware update to let you use them as a regular Bluetooth controller. That's very nice of them.

18:12 And the way that it worked is you would go to your web go to their web site, plug in your controller, and click this update firmware button. And without having to download any kind of drivers or anything, it would just take care of it. It would upload the firmware to the device using web USB. Really nice.

18:29 Really convenient. Here's another use case I've seen. This is the new iFixit soldering iron. It has a USB C connection. And when you plug it into your computer and go to their configuration website, the site connects to the soldering iron using the even more low level web serial API.

18:46 So, you can configure the soldering iron right from your web browser. And again, the great thing is you don't have to download anything. No extra programs, no drivers, nothing. You just run it from your web browser. And back to MIDI and web audio. This is a digital audio workstation. It's called Signal.

19:03 It's actually open source, and it lets you hook up your MIDI instruments to compose music right in your web browser. Now, you might be wondering, why the heck am I building sites with web audio, web MIDI, web gamepad, and web USB controlled DMX drivers in a web app? That sounds kind of weird, right?

19:22 Well, I'm building a spaceship bridge simulator game called Thorium Nova that lets people play as, the different roles on a spaceship. So one person is the captain, the pilot, the weapons officer, and all of that. And players might be distributed all around the world.

19:39 And I wanted an easy way for them to all have a fully immersive experience. So web audio lets me play sounds with different effects, like a slight pitch shifting or reverb every time you fire your phasers or torpedoes. It sounds just a little bit different so it isn't boring.

19:53 And I can mix sound effects to different channels to provide surround sound so that if something happens behind your ship, you hear it in that channel. And yes, I know that sound doesn't happen in space. It's space magic. It's more fun that way. Just leave it alone. Okay?

20:09 Web MIDI and WebGamepad let players connect physical devices to make the game feel just a little bit more immersive and accessible. And then, Web DMX lets users control their own DMX lights so that their lighting can respond as the ship goes to red alert or is hit with weapons or jumps to warp speed.

20:26 It works over the internet. It requires no downloader drivers. It fulfills the promise of the web. But I do need to come clean. Web MIDI, Web USB, and Web Serial only really work on that browser and its children. And that's a bummer. But let's be real. It is awesome that you can do it at all.

20:44 And while showing cool demos is really cool and fun and awesome, the real point of my talk is that there are a bunch of great APIs that are available for us in every browser today, right now. And it's thanks to the folks at the interop team.

20:59 All of the browsers got together and said, Here are the tests that we're going to write for our browsers so that we can pass them by the end of the year and make sure all of our browsers are fulfilling the promise of the web for everyone.

21:09 So this year, we're including things like the navigation API, dialog elements, view transitions, which you saw in my browser demo. This is it's all in a browser. The has selector, CSS popovers, anchors, pointer events. Thanks to the tireless effort of these web advocates, we are in a golden age of building great user experiences using web platform primitives.

21:29 In conclusion, build weird stuff. The end.