686

October 30th, 2023 × #SvelteKit#Serverless#Prisma#PlanetScale

We Need Your Help With The Secret Sauce

Wes and Scott discuss the technology stack and services powering the new Syntax.fm website.

or
Topic 0 00:00

Transcript

Announcer

Monday. Monday. Monday.

Announcer

Open wide dev fans. Get ready to stuff your face with JavaScript, CSS, node modules, barbecue tips, get workflows, breakdancing, soft skill, web development, the hastiest, the craziest, the tastiest web development ed treats coming in hot. Here is Wes Barracuda Boss and Scott

Scott Tolinski

key. Welcome to Syntax. In this 686th episode of Syntax, We're gonna be talking about the new syntax website secret sauce.

Topic 1 00:40

New show discussing Syntax website secret sauce

Scott Tolinski

And this might be a new segment or a new type of of episode that we'd like to do, and and we're actually gonna Be needing your help on this one for future episodes because what the secret sauce is is we want to know What is the secret sauce behind your product? The stack. What types of tools are you using? What types of libraries. Where does it host? What what are the things that are practical about real world applications? And we're gonna be kicking it off with the new syntax website. We're gonna be showing you our secret sauce.

Wes Bos

We should let's tell how we got the name for it.

Topic 2 01:19

Got name Secret Sauce from whiteboard in Toronto

Wes Bos

When we met up in Toronto last week, we had I forgot who wrote it down. Somebody wrote secret sauce on the whiteboard and we're like, why don't have something called secret sauce. Like, that sounds great.

Topic 3 01:34

Want to know tech stacks of large companies

Wes Bos

So that was a name that didn't have any sort of Content behind it. The other day I was thinking I was like, you know, it's one thing I love to do is to talk to people who work at these companies Generally, like larger products like Netflix or how KAYAK or like any any one of these companies.

Wes Bos

I want to know what does it run on? What's their stack? What's? Yeah, what database do they use? You know, how does it all go together? What's what is specifically does Netflix used to deploy apps to their TV. You know, there's so much secret sauce there, and I want a single show Where we talked to somebody from we won't go in-depth on all the different tech, but I just want to know, like, what are the pieces that, put it together. You know? What's the secret sauce?

Scott Tolinski

What are you working with? Yeah. And I think one of the best episodes we did n. For our separate club interviews was when we were talking about, the rendering of Google Docs, like, how wild that was.

Topic 4 02:23

Google Docs rendering episode was great example

Scott Tolinski

I wanna hear more stuff like that, and I I'm excited to dive more into this concept of secret sauce. So if you're out there And you work on something that could be interesting for this. Hit us up. Yeah. Or if you have a suggestion, like, Like, what do you what do you think about? Suggestion.

Scott Tolinski

Yeah. Well well You can email me directly at scott@syntax.fm, or You could hit us up on any of the socials, hit our our supper club question form or our or what's it called? What's the, not the supper club question. Our pinxt. Fm forward slash potluck. Yeah. The potluck submission form. Either way, just get get in touch with us, or we will get in touch a. With you, our people will talk to your people, and we'll set something up. So let's dive into the syntax secret sauce. But before we do so, let's Talk about one key component to this syntax secret sauce, and that's how we took, track our errors and exceptions because syntax is presented by Sentry. And Sentry is a A tool that we use that actually we didn't have in this outline here, but Sentry is a key tool for us to be able to track errors and exceptions. Just Just yesterday, man, I was having a weird issue where the database wasn't quite Working where I our database to the markdown files.

Topic 5 02:51

Email or use forms to suggest future secret sauce topics

Scott Tolinski

It should have reimported all of them. The import was successful.

Scott Tolinski

M. You'd go to a show page, and it would give me an oopsie daisy page, which is our 500 level error.

Scott Tolinski

And I was getting really frustrated with this, and luckily, we have Sentry. So I popped open Sentry. And guess what? I saw that Part of the database sync wasn't running, and Prisma didn't run a DB push.

Scott Tolinski

So Prisma hadn't gotten the, schema changes, and that's what was causing all the trouble.

Scott Tolinski

And that error was difficult to find without century. So if you, have bugs, if you're shipping software, You wanna track those bugs? Sentry. Io. Alright. So let's start off.

Wes Bos

This is not exhaustive tech of the The new site. We did a we did a show on that, and we'll do another one when we actually launch it. But, again, this is the secret sauce. This is the things that we do. So the thing is built in JavaScript, full stack JavaScript. Well, it's all TypeScript, right? Soup to nuts TypeScript, Node. Js. On the Server side, we did consider doing an alternative runtime, maybe Deno, BUN, Cloudflare Workers, Fastify Fastly. I always forget to mix those 2 up.

Topic 6 05:11

Using SvelteKit frontend, Node.js backend

Wes Bos

Fastly Fastly is the platform. Fastify is The server.

Scott Tolinski

Yeah.

Topic 7 05:19

Serverless functions for API and data tasks

Wes Bos

Serverless functions for all of the back end work.

Wes Bos

We are And that is for things like the API, all of our logic, database saves, queries, importing all of the data, the cron jobs that does all the transcription And show note summarization.

Wes Bos

That's all in serverless functions.

Wes Bos

We do not have any edge functions, but I specifically have A use case for them is we're starting to use these like little short URLs syntax. Fmmeetup was one of them, and I threw that in Cloudflare Worker rule, but like you got to pay $5 a month once you have more than 3 rules on Cloudflare. And I was like, we could we could throw that in an Edge function So quick, just have a like a JSON file of redirects and we could we could do that right in and edge function instead of having to pay and and use some UI to do redirects.

Scott Tolinski

Yeah. And it's an interesting bit of things. I mean, you're able to get those thrown up in 2 seconds, and it just works. Yeah. For for the framework, we're using the SvelteKit with Svelte as the, the front end framework. SvelteKit has the full stack framework, so it's handling, the data loading, pulling from a database, Connecting, to any sort of server side, stop dealing with actions or how we send things to and from, the front end to the back end. It's all SvelteKit.

Topic 8 06:46

Using Mux's MediaChrome for audio player

Scott Tolinski

We're using MediaChrome, which is from Mux for the audio player on the website, and MediaChrome is great. It's a web component that you can use for video players or audio players or anything you want, and it is extremely full featured.

Scott Tolinski

And I'm I'm somebody there who's used every video player on the block. And I gotta say MediaChrome or in general, Mux video, the whole Mux packages there for The, any type of playback is they're just they're just lovely, and you don't have to use mux to use these. These work with any sort of video stream whether you're streaming or using audio or or anything. They're they're fantastic.

Topic 9 07:29

Fuzzy search allows quick finding episodes

Scott Tolinski

Website. In fact, we have, like, a fuzzy finder for every episode on the entire site.

Scott Tolinski

It's really, really slick. It helps you find things very quickly, a And we're doing so using something called flex search.

Scott Tolinski

Unlike, like a paid service or anything like that, there's no paid component. It's free. We're essentially loading up all that data, loading it in via a worker, and then it's available directly when you need it. In fact, most of the code of our implementation of FlexSearch, you know, came from the SvelteKit and Svelte.

Topic 10 08:01

FlexSearch powers fuzzy search via worker

Scott Tolinski

N. I think they call it site kit is what they they call it, but it it's what they use to build their own sites. It came directly from their code base, And there's a lot of nuggets in there. Obviously, we adapted it to our needs, but, I'm really, really impressed with FlexSearch. It's super fast. It works very well. And, again, you're not having to pay for an additional service there to get really nice and easy search going. Yeah. I'm I'm actually curious about that because I I haven't I don't think I've actually asked you questions about that.

Wes Bos

So when you use it, it searches through Show notes, show titles.

Wes Bos

Those those are the 2 things, I guess.

Scott Tolinski

Anything else? Yeah. Basic Basically, right now, it is just show notes or show titles, and it could be anything that we tell it to. So it it's really like The the way it works is you you're kind of like building a tree of data. Yeah. And that data is is coming directly just from a database call.

Scott Tolinski

And anything that I want to put into that tree can be used as a parameter on the search.

Wes Bos

So that that's really quite it. Do you have to build like an index of all of the shows to go into the search? Like, I'm just curious, how does FlexSearch know about All of the shows. And and is this done server side or client side? It's done server side,

Scott Tolinski

and the index is built by FlexSearch.

Topic 11 09:21

FlexSearch builds index from database call

Scott Tolinski

And so, basically, what I'm doing is like a database call, and I'm creating essentially an array of all the shows. And then I'm I'm passing that into FlexSearch, and and I'm creating an index there.

Scott Tolinski

And FlexSearch creates essentially f The tree, and then Mhmm.

Scott Tolinski

It creates the links and the breadcrumbs and everything and spits it out on the UI. So that way when I Start typing. It automatically and instantly

Wes Bos

finds everything from that giant list. Oh, okay. So it it just it just sticks everything in memory, and then it's able to have access to that pretty quickly.

Scott Tolinski

Yes. But it's doing it in a service worker. So Yeah. Or is it a service or just a worker, not a service worker? Sorry. Those 2 always confuse me. Yeah. Oh, okay. Yeah. I'll I'll we'll have to go deep into this Because it's actually, it's not that many files, and so you could really go through it pretty quickly in terms of, like, what exactly It is happening here, but, if you're out there and you wanna take a look in our implementation for this thing, m. You know, the the relevant stuff is there's a content dot JSON route, and that's essentially running just a normal Prisma client f. Find mini and then, creating it and using a a push to push it into the structure that we want it to have, And then that's sending it as a content dot JSON.

Scott Tolinski

Then we just have a worker that hits that content dot JSON with a fetch n. And then

Wes Bos

throws it into FlexSearch, basically, and then we massage a little bit. Isn't that, like, loading every single show note on the client side then?

Scott Tolinski

Is that huge? On the client side, it's been it's doing so in a worker, though, so it's not doing it on the main like, you're not nothing's being interrupted there. Okay. But

Wes Bos

I don't see it. I don't see a big request, but, like, this JSON file.

Wes Bos

Yeah. Does it have Every single show note ever? Yes.

Wes Bos

And that's being loaded client side? Yeah. But isn't that, like like, 8 megs of

Scott Tolinski

of data? I'll tell you exactly how many megs of data it is. Content dot JSON. It is, 4 megs. Okay.

Wes Bos

So it's this is similar to Our API route currently that has when we hit that API endpoint, we load every single Every single syntax ever on that API endpoint because that was not a big deal. Obviously, we need to paginate that, if we wanna Go any further. Actually, that's we get a message from Vercel every now and then that our we're going over our bandwidth because That specific file is is pretty large.

Scott Tolinski

Mhmm. But maybe that's a concern. Yeah. That's okay. Whatever.

Wes Bos

We styling. Let's talk about that next, which is scope styles.

Topic 12 12:29

Global styles, then scoped component styles

Wes Bos

One thing about Svelte is you simply just Put the styles in style tags in the same file that you are creating your components, and all of the styles are then scoped for that. There is a number of global styles as well for things like, text layout, you know, the fonts, the colors, the Sizes of everything. Global variables.

Wes Bos

Sometimes you just need a global right utilities. No, you need a good baseline before you get in.

Wes Bos

To worry about.

Topic 13 13:10

Some global utilities for common needs

Wes Bos

Also, we've used a class of highlighted somewhere else. You know, that's not a thing because they're all scoped to what you need. Some utility classes specifically around there's 1 for adding grit to something. There's a couple around font sizes that are just like little handy utility classes if you want to change the font size.

Wes Bos

And then there's lots are on layouts. Yeah, there's a whole, like, zones and layouts thing, which I think will save that For the light and dark mode

Topic 14 13:31

Layout utilities help component placement

Scott Tolinski

podcast. Yeah. There's a lot there.

Scott Tolinski

And it it it's interesting because, like, Pretty much anytime you need to place anything on the site because we did some really, really slick stuff with, like, a grid based layout system. So using, like, grid specifically for vertical and horizontal layouts.

Scott Tolinski

And then you can now basically place anything anywhere Anywhere you want at any given point, and you can do so with a class nice and easily, or you can do so with straight up, named grid columns in CSS. So a lot of lot of fun stuff there. The Svelte animation API

Topic 15 14:13

Using Svelte animation and page transitions

Wes Bos

was very cool. I'm getting a little in the weeds here, but I had a lot of fun learning the Svelte animation API. It's not something I had used. I took an example that Scott did with the pagination. So I built out the pagination for the show's page, and it's so fast That when you click the next button, it just goes to the next page. It's already preloaded, so it's instant and it's a little bit it was a little bit disorienting Because all of the numbers shifted to the left by 1.

Wes Bos

So Scott threw like what, 6 lines of JavaScript at it and like Added a little swoopy animation

Scott Tolinski

and it like made it. It gave so much more context as to what was happening and it's so good. Very happy with that. Yeah. You you essentially just drop in a, like, a flip, just like a single directive on a DIV, and then it handled The flip animation, since everything was a keyed each block, it kind of knows where everything is and is able to do those flip style animations super easy, n. Took no time, and then I just added an in and out transition as well. We also use the page transitions API, which, h. Has gotten really easy inside of SvelteKit. It's not quite the same as what Astro is doing. I think people get confused there when they hear that Astro added the The page transitions API because they're actually polyfilling it back to older browsers where the way that SvelteKit is doing it is basically just, there there's just a a Simply a function in or a, sorry. Not a yeah. There's a a, an on navigate a. Kind of life cycle method that just runs when the site is navigated.

Scott Tolinski

And then you can simply tap into that and then, therefore, take advantage of the new view transitions API using just straight CSS through the rest of the site. So we did that. It's progressively enhanced fully. Works really well. Love it. Some other cool stuff, tooling Inside of this, we use Prettier for our formatting, which, you know, did we didn't really change that much in the Prettier. You know, I haven't really touched it.

Topic 16 15:49

Page transitions via CSS and onNavigate

Scott Tolinski

Very, very specific stuff, prettier plug in Svelte.

Topic 17 16:18

Using Prettier, PostCSS, PNPM

Scott Tolinski

Trailing comma none. Did you add that?

Wes Bos

Added by Scott Talinsky. That's surprising. I love trailing comma. No. I didn't add that. I I thought we did have Yeah. There must have been a default. Trailing comma. But we did we did go semi colons.

Wes Bos

Yes.

Wes Bos

We need something to enforce Much to my dismay. We need an ESLint rule to enforce, snake case. Because I had written a bunch of the transcript stuff outside of the show, and I did it all in camel case.

Wes Bos

And then that, coupled with my default to just go camel case, is there's still quite a bit of camel case in there that needs to be, like, we need a rule, and then we need to be able to list out where all the camel case because refactoring with TypeScript is easy. You can rename a function pretty pretty quickly,

Scott Tolinski

But I would like all the time. Yeah. Snake case.

Scott Tolinski

Yeah. I like Snake case. It's easy for me to read. It helps. I know people. Yeah. S. So that's not the JavaScript way. I get it. It's just easier for me to read.

Topic 18 17:10

Enforcing snake_case with ESLint

Scott Tolinski

Post CSS, we're using post CSS with a preset ENV to basically s Give us access to newer CSS functionality where we can with largely not having to worry about it. We progressively enhanced some things, some things we're just waiting on. We're using PNPM as our, as our, I guess, you would say your your NPM replacement for fetching and storing your node modules For making sure we're all on the same version of Node. So that way we can always make sure that if you're firing up the site. You have the right version of node, running scripts, all that stuff with PNPM.

Scott Tolinski

I really like it. It works fast. It works well.

Scott Tolinski

And then because we're using SvelteKit, the whole thing is built on Vite. And one thing I really like about the SvelteKit version one changes that they made, they basically have have made Vite an actual front and center thing where you still have access to your Vite config. Right? They didn't obfuscate that. So If you wanna add Vite specific things or you wanna get into the Vite side of things, you could still do that with with no issues. I'm just, I'm on the website right now, And I realized

Wes Bos

one of the images is being base 64 encoded into the JavaScript, and it's like I think it's too big.

Scott Tolinski

Okay. We'll figure it out. Yeah. We got a fit we got fit and finish to worry about. And then Oh, yeah. For sure. We're just gonna launch it,

Wes Bos

And, you, the the listener, will help us find all these little bugs. Although maybe it's not. I think I think Vee inlines stuff when it knows that it's okay to inline it. So maybe it's not. We'll look into it.

Wes Bos

Let's talk about database.

Wes Bos

So we are using MySQL, With a Prisma adapter, so Prisma allows us to generate our schemasor we sorry, we define the schemas And we defined all the relationships. We defined, all right, this is the show. A show has a show number. It has a title, has a description, It has and then it has a relationship to guests and then also has a relationship to a transcript. All of that stuff, right? And then Prisma will then translate that over to a MySQL database for us, as well as give us the typed API That allows us to query all of the data, which is really cool because, like, we don't have a show type. We just have A type of whatever is returned from a specific query that we've wanted, right? Like I might want a show with just the number and the name, But you also might want to show with the number, the name, all of the guests, all of the transcripts.

Topic 19 19:00

Using MySQL database with Prisma

Wes Bos

So the way that the types Flow from the schema to Prisma to you have a query on the server and it's returned from the server And populates the types on the client side like it's unreal how the types just ripple all the way through the entire thing.

Topic 20 20:19

Prisma auto-generates types from schema

Wes Bos

And there's just a couple spots in the application where we've had to explicitly define data types.

Wes Bos

Yeah. That's awesome. Everything else is just Autumn, it's a it's a bit of a different, like, way to to get used to it, and sometimes it's annoying when you hover over top of a type and it's just just vomit. I would I would prefer that it was just a show type, but then you're like, well, the show might or might not have show notes, and you have to Do an if statement, and this way it knows

Scott Tolinski

if the data is there or not. Yeah. You know what? And and it's one of those things that when I was working a lot in Graph q l. I felt like I was defining my types in 4 different places, and now it's like, alright.

Scott Tolinski

You really just have 1 place, and it and It cascades the entire app using Yeah.

Scott Tolinski

The Svelte kit, the way that they handle types, but also how Prisma generates our types and stuff. It's all just very, a. Very nice to work in. Services.

Scott Tolinski

These are some things that I think you know a little bit more about considering you were doing a lot of this AI work. But we're using several things For our transcripts as well as, AI tools to identify who's talking when, a I OpenAI, Claude, and Deepgram. Do you want to dive into those? Yeah.

Wes Bos

So for the transcripts, we're using Deepgram to transcribe the shows. We've we've talked about that in the past as well, but They do a very good job at diatarization, which is detecting who the speaker is.

Topic 21 21:31

Using Deepgram, OpenAI, Anthropic for AI

Wes Bos

It doesn't tell you their names, and we can't train it on n I, which would be nice. But we fix that by tacking on a little hunk of audio that is known to the end of every episode When we transcribe it, so it just says Scott says, like, I'm a I'm not gonna say it because I don't wanna screw up the actual Yeah. Our whole plan here, but I'm a cool guy who loves cheese. And then Scott says specifically that weird sentence.

Wes Bos

And then I find out What speaker number that's associated with and that's how we know if it's Scott or I speaking.

Wes Bos

Then we also Have some AI show notes, which are I'm very happy with. It will do really good summarization as well as give you like markers For the different topics throughout the episode, so if you're like, oh, I want to I want to know what Wes was talking about, it'll give you a really good like subheadings.

Wes Bos

You can quickly scroll down to One of the subheadings is see what we were talking about. I think that's gonna be extremely helpful where you say, oh, what what did they just say? You know, what was that library they mentioned? I didn't catch it. Sometimes I'm on a podcast and I have to rewind 6 or 7 times, and I cannot understand what that One word that they're saying because I want to look it up. I think that the show notes will be very sorry. The AI notes will be really good for that.

Wes Bos

We are feeding those into OpenAI, and that has been pretty consistent. But then I also have been playing with Another one called Anthropic Claude, and they have been fantastic.

Wes Bos

So, I'm going to it's a little bit More wild and a little bit more hit and miss getting JSON from them, but I think it's worth it for the quality that we get. Yeah. Yeah. A lot of interesting stuff there.

Scott Tolinski

Testing. We use, I you could consider TypeScript as being part of your testing infrastructure. I might Because it can help save you from potentially finding bugs.

Topic 22 23:32

TypeScript provides compile-time checking

Scott Tolinski

But we're using for, you know, more more classical testing. We're using Playwright for end to end testing, and that's the only testing that we're doing. We're not using any mocking. We're not using any of that kind of stuff. Basically, just fire up the site with the developer DB. And And the things that we're not testing, I'm not testing the, I'm not testing a lot of database actions, really. I'm I'm using Playwright to jump through the site, Click the player. Make sure then the episode starts playing when you click the player.

Scott Tolinski

Every time you you visit the page, the The specific header that you're supposed to expect to be there is there. So you're making sure that all of your pages are loading as expected.

Scott Tolinski

The things you're expecting to be there are there, And the functionality that's important is there and works fine. And for the most part, I've been liking Playwright just fine. I really like how When you run it without the UI, you can run a lot of tests in parallel, and it's just really fast. Hosting.

Wes Bos

We are we've been like kind of devin' on it on Vercel, and been pretty happy with that. I'm Fighting a WASM issue right now because with WASM libraries they need to you don't explicitly, include the Wasm file, especially when I'm using like a library. So I'm trying to figure out how to like get that Wasm file included in the build For Vercel, we did consider going to fly as well because some of the transcript stuff Takes in some of the importing stuff. Oh, no. The importing stuff that you did is pretty fast. Right? Like, how long does it take you to import all 600 shows? Like, a couple seconds? Yeah. A couple seconds. If you're on if you're on local, it takes

Topic 23 24:37

Using Playwright for browser testing

Scott Tolinski

milliseconds. It's so fast on local.

Topic 24 25:27

Hosted on Vercel, may try Fly.io

Scott Tolinski

Obviously, in a production environment, it's a it's a couple seconds. That's impressive.

Wes Bos

So some of the stuff I'm doing, which is hitting APIs, takes 30, 40, 50 seconds.

Wes Bos

So I was just I had to make sure that that sort of longer running processes, would fit within the timeouts of a serverless Because as soon as you start getting into over, I think it's 300 seconds is the max you can extend a serverless function on Vercel to.

Wes Bos

Then you start having to look at, running a longer running process. And Fly is kind of Kind of neat because you can run like you could basically just spin up a long running node process For 2 minutes and then spin it down and you only pay for those 2 minutes. So it's kind of nice because you you get the benefit of Long running node processes.

Wes Bos

But then you also have the benefit of you're not paying for it from the time you start it to the time Like until you shut the thing down, you're only paying for as long as you need it for. So I don't know. We're We're not sold on Verstylog just yet, but it's been been working really well for us, especially now that they have Cron jobs as well. It's in beta. That's one thing we've needed for CSM. A lot of the transcript stuff. Word. Hosting planet scale, right? So it's MySQL, but we host on planet scale and that allows us To interact with it via serverless functions, it does connection pooling for us, so we just have to connect to PlanetScale and it handles all the Hard stuff for us. And then, we are also using Cloudflare for some of the redirects, although I talked Earlier, I would like to move those redirects into an Edge function so that we don't necessarily have to pay for that, as well as It's part of the the code. Right? Like Yeah. I would hate for someone to come along and be like, hey, where's this, like, your redirect happening? Oh, you have to log into the service and and whatnot. I would love for the short URLs to be in the code, not even in the database, like a file

Topic 25 26:22

Fly.io good for longer processes

Scott Tolinski

With all the redirects, it'd be kinda neat. Word. Well, that is our secret sauce. This new site is gonna be launching, hopefully, by the time, you hear this episode.

Topic 26 27:35

PlanetScale for MySQL database

Wes Bos

F. Oh, yeah. Yeah. So

Scott Tolinski

syntax.f m, check it out. Check out the repo. There's a ton of tasty stuff in the repo.

Scott Tolinski

And, you know, maybe there's Some Easter eggs throughout the site as well. So take take some time to check it out. I think, I'm really proud of the code base, proud of the the work that we've done here, and it's it's pretty

Wes Bos

That's it for today. Thank you so much for tuning in. We'll catch you later. Peace. Peace.

Scott Tolinski

Head on over to syntax.fm for a Full archive of all of our shows.

Scott Tolinski

And don't forget to subscribe in your podcast player or drop a review if you like this show.

Share