Let's read from the book bound in human flesh

01:53All right. Howdy. I'm Peter. This is Lobster's Office Hours, and I am showing lobsters. The... Where am I at? Okay. So yeah, let's throw the message in chat. pushcx Welcome to Lobsters office hours, ask questions anytime!
Welcome to Oyster's office. Ask questions anytime. It's not typo the name of the site, because that's funky. All right.

02:41So yeah, if anybody has questions about the site, the code base, anything, feel free to pop up. and we'll chat about it in the meantime i tend to work on maintaining the site which is a lot of pull requests reviewing bugs adding bugs just to start out with something fun i saw this post go by on blue sky from the guy who started what's now valky i think is the live fork but it was kind of funny because he said oh i'm going to make a hacker news replacement that i'll run myself and i'll guarantee personally no benefit to individuals clear rules, total transparency, and a karma system. And then you can see one here and like four people in here were like, how about lobsters? Yeah. Another one. I want to say there was one or two in replies or maybe they're gone, but there were a couple of mentions of lobsters. Yeah. We're not seeing everything because my reply is missing from here. Maybe because I'm logged out in this browser. And he said somewhere in these replies that he's going to build something on Valky. And he didn't say Valky. No, he said Redis. That's what it was when it started. So anyways, I popped up in there to say like, yeah, it's not the tech stack you want. And it really sounds like you want a broader focus, but good luck. So we'll see how that goes, just a little fun thing. Speaking of how stuff goes, I noticed just before the last stream, and I didn't get a chance to look at it, that there was on prod what looked like a bug in caching RSS and JSON. And so I wanted to start there, because that could be, twitchtd what lobsters office hours on a tuesday?
significant improvement for production performance because we're still just you know relentlessly getting hammered by scrapers so let's see the the full page cache lives in the file system yeah thomas i have a a recurring commitment on monday afternoons now so i've shoved it over to tuesday and

05:41yeah i just remembered to update the schedule so i'm gonna do tuesday and thursday which i don't love because especially for pr review it's nice to have them kind of evenly spaced through the week rather than two days apart and then five days but my friday mornings are also usually have something going on so we'll see how it goes And probably late summer. It's hard to say, but I will probably end up going down to just one three hour stream a week at some point this year. I'm not totally certain on that, but I figured I should mention as long as we're talking schedule. Oh, and if you just joined. Your PR is the second thing on my to-do list. I wanted to look at this RSS issue.

06:50Yeah. So when I look at... ChaelCodes Hello! 👋
I was poking around in the cache for something else, and I realized I was seeing a lot of RSS and JSON hits. in the rails log and i was a little surprised because i was like huh i'm seeing more oh hey chael and then yeah yeah both of these are wrong all right so how many do i have let's see if that's the right amount of escaping for this because it yeah so then that's ooh dot html if html is messed up that's like Every fucking page in the cache. That would certainly be contributing to site slowdown. All right, so. I think it's the pre. graefchen Heya Peter and Chat limesHi
Yeah, so the the full page caching says if you don't have a cookie. And we have that file. trying pass. So that should handle RSS and JSON. But if we're sticking an extra dot in there, two out of three of these are broken. twitchtd hey all
So where even is our config for action pack caching?

08:43So we have this cache page proc.

...52But I don't know where we config the file name that gets cached.

09:07Is caches page our code? no that's part of action pack so this is the plugin we've been using for no this is a fork what's on your remove deleting slash this is just a random fork of the project so the concerning thing about the full page cache is right there That hasn't been touched for five years and if we don't config that file name and it's somewhere in here. That it's broken.

10:02Yes, there's perform caching. There's page cache directory. Do we set that?

...22Yes, but we don't have anything that sets the file name. All right. Custom caching per controller, I want to say that I already had the hot patch something, but I don't remember what it was because we've barely touched this.

11:04that's closed recently no not for years also we had something we had we did we monkey patched this in which case it's going to be in like lib extras right where did we put those

...40There is a page caching monkey patch. That's what I'm thinking of. Either appenza.html. Okay, yeah, so this is definitely broken.

12:02Yeah, it was. Oh, this is not true. because we cache RSS in JSON. And I guess I wasn't thinking of it when I wrote this.

...22It doesn't end in HTML. I'll see why. So this extension or default extension must have the dot in it already right if my code messing up is adding that second dot let's go find i wrote where it came from right so let's say prepend write patch catch file let's go find the original

13:09Yeah, they're doing the same thing. Where did they get default extension from? Is it just an accessor? Yep.

...39look at you extension dot type symbol so they already have it yeah why did i add the dot in there i must have this is the thing with string typed programming is you never know what's going to be in the string

14:09How do I want to test this? I can enable the cache locally. And we'll see how likely I am to foot gun myself. So let's reproduce the issue locally first. That's prod. OK. Start the local server. Let's go load local host. And then newest.rss. Well, it helps if I spell it. Well, it's just slash RSS. Yeah, it downloaded it. Great. And then let's say a random story. And we're going to tack JSON on the end. I'm just trying to fill the cache. OK, so now. There's the bug, .html, ..rss, ..html, .json, that's... All right, so if I get rid of that, and I gotta bounce the server, and let's get rid of these cached files. I should have kept these open tabs so I could just reload them, but I wasn't thinking that far ahead. I just wanted to see it, you know? All right, so there's that. Cache is still enabled. So if I reload this, go back, reload this, go back. Let's do RSS, and let's do newest.rss. Yep.

16:14OK, so the JSON is still wrong, and the RSS is doubling it up.

...38File the next name.

...45Why? Why?

...57So this is forcing the extension on even when it has an extension. Is the original code correct and we're just wrong?

17:14We cache HTML, JSON, and RSS. Zero is a dot. Use the existing. It's mostly correct. Try a route like youtube.com. Things.com is an extension.

...50Yeah, we have a Stringly-typed bug on top of a Stringly-typed bug because if we had a well-constructed path object, if it's empty, let's say if

18:22Double check, xName.

...35Will it include the doc? Yes. OK. So here, HTML, JSON, RSS.

19:00But that logic also doesn't make a lot of sense. Because if we have those on the file name, why are we adding them? Yeah, so we're saying if it doesn't include that, tag.html on the end. OK. Bounce the server. Let's remove the cache.

...40So there's new. Let's put RSS. Let's pull newest.rss. Let's go into a, well, let's do the homepage too because that's an index.html. And then this and last. Now we should see the cache correctly. Yes. OK. So watch me remember to turn the cache off. Such a foot gun. I get myself so many times with that. And I have to bounce the server. Let's go look at that diff.

20:28Let's also look at the history here so I can figure out how long I've been shooting performance on the foot.

...40Since December. Not so great.

...50Turn. Cat was up on the filing cabinet earlier today, but he's not right now. He's laying so he can shove the mouse around.

21:06So we don't say those. Say special cases. recognizes them and leaves them in place tags everything else with dot html okay

22:07I wanna write a longer thing.

23:24Well, let's get that one deployed and then I can actually do my, my usual round through the pull requests and such. But honestly, if, if Thomas is here, it's more useful to pick up the SQLite work. yeah. Wait, before I push. Let's remember to run the build script for once. Especially if I'm touching fiddly monkey patching that I've already blown up once. Oh look, a failing spec.

24:22It's a spec of this that was wrong. What's the name of you? Spec caching. Yeah, so that was wrong. That was wrong.

...48That would have included this.

25:51OK. Do I need to see these tests fail? I would like to see these tests fail. So if I just lean on undo, that's where I started. And most or all of these specs should fail now. Good. That reproduced the bug even. That would have been easier than doing it manually, but I do things the easy way. We're developers. Let's push that up and get that one deployed, because that's going to be pretty valuable.

26:56So there's how many in the public cache right now? Oh, this is going to be... Oh, that's everything that has two characters in it, because FD does regular expressions. Come on. Okay, so there's 152, which doesn't sound like a lot, but we have, as always, a... Everything on social software is logged normally distributed. So the homepage, the primary RSS feeds, and the stories that are on the homepage right now get, I don't know, 70% of site traffic. I'm kind of pulling a number off the top of my head. That's not something I've researched, but just an enormous amount of it. So if we have this, yep. well let's make sure i'm in the current directory good so the reason i just cd to the directory i'm in is this tilde lobsters current is a symlink to the current release and if you replace the directory you're in which is what the deploy system does your shell will be open to the old directory, not the brand new one, because it tracks it by inode, not by name. And so if you effectively rename the directory you're in, you are not seeing exactly what you think you're seeing. Yeah, and we're not hitting those house server load. Tuesday afternoon is a pretty busy time, but that's still a little high. But it's certainly not, you know, sights on fire.

29:13Yeah, that's a lot better. Now we're not seeing a ton of RSS and JSON here in the Rails log. That's great. CN slash RSS. Oh, that's going to be a logged in user.

...38Okay, cool. That's a valuable performance fix. I remembered that one from my notes. So let's take a look at where pull requests are.

30:54What the...

31:32It is a little bit tempting to just PR in a linter for the widely available baseline standard and ask Federico to like merge in main. But that's a little passive aggressive to give somebody a failing build instead of just talking to them.

32:35Let's filter down to

33:17I think I reviewed this, right? Rebecca did note in the chat room that Maine was misbehaving the other day, but it looks That wasn't me breaking main, unusually. That's usually your safe bet is Peter forgot to run the build and broke the build. Okay, so return false if the comment is gone or the author is the user being passed

34:19Yeah, that's kind of a parallel to is editable by. graefchen I honestly like the `@supports` at-rule from css. And I only learned about it just recently. limesSit
And I guess I still have the same OO gripe that there's no good place to land something that uses two objects, unless you want to get real small talky about it and make a third object for this concept of is the author of. But that's just, that doesn't really fix the law of Demeter concerns. Graf Chen, what do you like about atSupports, which also is not to the standard, right? AtSupports. Oh, it is baseline widely available.

35:13Maybe that would be, can you have an else? Is there a good MDN doc?

...27it supports can i say if not yes i can say if not well my my hypothetical it's not really hypothetical my semi-implemented linter could look for this and say, are these things in an at support at rule? Yeah, I had to break out the at rule separately. And then it's mapped over the CSS file. So everything is considering a thing in isolation. And what I'm thinking is, We could require that every at support. Like grab this selector. Or I guess it's not a deck selector, it's. Like a predicate, a declaration. And affirm that there is a matching at support, not. That exact declaration. Because that would get me... You can use these things, but you have to have the fallback. You're forced to define it, forced to think about it, which is... Yeah. Now that I can see my way through the nice little engineering puzzle, the test matrix is kind of painful. graefchen Just say if I can use this css feature, then use it, else not. Used it for some small fun project so things get displayed in the new masonry-grid or how it is called. And on other browsers that do not support the feature use something different. limesSit
because I don't typically keep around old and weird browsers that would hit these, the other side of the predicate, basically.

37:47Yeah, I'm trying to think of ways to force a responsible, like,

...58Oh, yeah, this is getting complicated. Saying not that is...

38:10Proceeds an expression resulting in the negation of the expression. The not can be applied to any declaration. But it's only doing these for that first style, not like not font. transform or whatever this one was font format. Hmm. Okay.

...50I guess we could also define that rule as if you're going to have supports transform style, you have to have supports not transform style or. And then there could be like a CSS variable here. And then developers could define that to force seeing the alternates on all of these. Hmm.

39:23Yeah, this PNF isn't too bad.

...36Or instead of adding all of that complexity and expanding the test matrix, I could just continue the don't do it rule, which is a little tempting because it's less work.

40:00hmm this could make our unprincipled exception for colon has a little less of an exception hmm i gotta think about that let's let's grab this thank you grave chin that's a I had heard of this but it clearly didn't stick.

41:16Why does this have an A comma B?

...29How would you know what these arguments are? Because they're only passing one, not two. Is there something special about that A comma B that I didn't catch because I skimmed past some stuff? no what was this using future queries

42:19Jamamp_ Does SCSS/SASS have the ability to compile down to older targets, much like es6->es5 for js?
Jamamp_ And/or does Lobsters pre-process its css
Okay, no examples of that A, B. How odd.

...29JamAmp, yes. And that is, that ability to compile down to older targets is a big incentive in the creation of SCSS and SAS, as I've heard it. We used to use, I think sass but it's been a few years because honestly CSS advanced to the point where we weren't getting anything out of the feature set because we got stuff like being able to nest selectors and the complexity of pre-processing CSS and JavaScript was just and a thing that introduced a bug or two a year. And so I wanted to get rid of it. I've also become kind of militantly against using JavaScript in the build chain. graefchen I very much like modern CSS. I really love it. And for some reason I am currently having way to much fun with also printing webpages. limesSit
I've just had enough pain keeping tools written in JavaScript working, especially across updates that I kind of In it's one of the couple of places where I have a moderately strong technical opinion and I put my foot down and I said we're just not going to let javascript in the build chain. printing web pages yeah I do print a PDF myself and it's really apparent when. Some sites have an old at screen declaration around all of their CSS.

44:15Are there any other implementations of is flackable that need their signature updated? Probably not. Nope.

...40olexsmir it's better to avoid js when it's possible (intended as a sarcastic and at the same time very serious message)
Jamamp_ That's so fair. I'm glad at least that some tools for js aren't written in js. Like `swc` being written in rust. Makes avoiding pnpm better, a bit
It's funny looking at this set of nested conditionals through the lens of gleam, like we're doing a type check and then we're doing a guard. Yeah.

...58What's SWC? All right, let's not lose that PR. Expensible. i haven't heard of that we do have an open issue for i don't know what the project name is there's aux format and aux lint i don't know if the project is just named aux or if it's named one of the tools but i would like to integrate that especially if someone wants to package those up as a gem because then that would really solve the distribution issues of how do we get all the developers to have this tool? I've tried really hard to make sure that as a developer, you don't have to install six more tools just to work on the site. So I've been really reluctant to take on dependencies outside of Ruby and Rails itself. I think libvips is the only one I've introduced in years.

46:27olexsmir it's kind of funny that python and js ecosystems are getting more tooling written in rust, a completely different language than the ecosystem itself
Looks very Yeah, that's a good point. You also see some go. And I think it's just the languages with extreme late binding, where You have dynamic dispatch and not reflection, but the ability to do metaprogramming, which usually requires a very extreme form of late binding. You're just never going to get anywhere near I compiled a binary in a statically typed language level performance. Jamamp_ At work our monorepo is getting more traction from multiple languages. We're moving from nx to bazel for a language-agnostic orchestration of tooling. But, it doesn't remove the requirement of multiple package managers...
Like, Ruby has indeed gotten three times faster in the last couple of years, but that still leaves it, I don't know, what, two orders of magnitude behind something like Rust or Go? And I would expect good Rust to be faster than Go, just because you can manually manage memory. You pay for it. You really pay for it, and how expensive... how time-consuming those languages are to write in, but... Can't say I don't envy the performance.

47:52olexsmir go and rust have basically the same performance
Yeah, I see a lot of folks integrating Bazel because it's so cross-platform. Do they have basically the same performance? Huh. You know, I wouldn't olexsmir if you dont do very strange stuff in go it's pretty fast
davidofterra It depends a lot on what you're doing. Rust has vectorization and the LLVM backend.
As I think about it more I guess I wouldn't expect an order of magnitude difference, but I would expect rust to generally be like. 50% to 300% faster typically. I guess I don't I write so little in compiled languages that I can't say this is anything more than a hunch and kind of seeing stuff go by. in random blog posts and on Blue Sky.

49:10olexsmir if i'm not mistaken, it's usually ~3% difference between go and rust in most benchmarks
3%, wow, that's really cool. I could not have guessed that. I did not guess that.

...28Okay, this we don't need. But the rest of this, we can just go ahead and use her message directly. Let me check on these issues because they might not both have closed automatically.

...52This one did.

...59This one Did not.

50:20Great. That's lovely to get to merge a couple of bug fixes. Did I put that in the notes? olexsmir also go sometimes might be faster than rust, since rust implement it's own malloc() and go reuses one from libc
olexsmir *slightly faster*
yes 88 88 isn't that the name of the taylor swift album i think so sometimes go might be faster than what rust what wait rust implemented its own malloc huh Did they... Actually, let's just go right to... Someone must have written about this, right?

51:16Grepen about C++, no. I guess I could also just ask if you're familiar, do you know why Rust implemented its own malloc? olexsmir i'm not very familiar with rust
I've seen folks wrap it so they can do things like manage their own arenas or pre-allocate a little better, but all right. You're clearly more familiar with me than I am, or at least you know more trivia than I do.

...56davidofterra Rust uses the system malloc.
Yeah. Alex, C3 and Jesus arenas, yeah. It's usually arenas. Rust uses the system malloc.

52:22davidofterra They used to use jemalloc but moved away from it.
olexsmir ok i might be wrong, but i remember reading that it has it's own
Oh, you know what? seeing the dimension of endpoints i should fix that but there's or i should properly review that pr but i just remembered let's run that fine one last time yeah nobody's there good alexamer maybe somebody said that because there is a popular library that has their own mallet Oh, or used to use GemAlloc, which, oh man, get hype. Facebook picked back up, which is great news for the Ruby ecosystem because all of the deep Ruby system people that I follow were like kind of side-eyeing Facebook. They weren't side-eyeing Facebook over Facebook choosing to drop maintenance of it but they were kind of side eyeing the whole ecosystem of like we kind of need that to work we can coast for a while but it's gonna break and it's gonna need maintenance how we're gonna pick this up and manage it and then it felt like all of them let out a breath they were holding when facebook said that they would pick it back up

54:23This thing is broken.

...30I saw 404 submitting this form or the origin form. This is a thing because, yeah, this is going to be what's broken. Yeah, it was an origin.

55:21So this one's going to go to mod origin path. They have their own create and ban action, do they? No. They probably should. But that's a bigger thing than I want to do right now. Just to explain what I was thinking there. There is a, like, it's not here. Why is it not here?

56:15Oh, I've moved it over into, did I make an origins band controller? Is that what I should be looking at? So the complexity is that to ban a domain, the domain has to be in the database. For a long while, the only way to get a domain into the database was to submit a story with a URL of that domain. But occasionally we would see spam where it was like example you know github.com example ai and you know just a total garbage post and i would want to be able to say like okay they're obviously going to submit exampleai.com next if we let them and so i i had that flow so that you can create and ban domains that haven't even been seen yet it's not a common flow maybe a couple of times a year off the top of my head which would be why I haven't added it for origins. And origins are basically our way of handling shared domains like github.com slash somebody is a different site. You know, github.com slash Carol is a different site than github.com slash Dave. And we don't want to have to ban all of GitHub to ban Carol if she's doing bad self promo.

58:01on origin pathway you call it identifier not id oh yeah it's not it's origin dot identifier I could probably just pass origin, and the to param would pick it up. But I just don't super trust that.

...56hmm i don't think the mic is picking it up but we've got a midwest phenomena happening where we get these little isolated intense storm cells that come overhead and rain on you for 20 minutes and then are done and so you go from full sun to pouring down hard rain and lightning Back to full sun in like half an hour. And we're just getting to the peak of that. olexsmir sounds like a good time to take a walk :)
So on the off chance the house gets hit by lightning and the whole block goes out, the stream may end abruptly.

59:47Take a walk. Yeah, I don't think I'm going to go get soaked. Bye.

01:00:01Oh, because I need a description.

...21Didn't fetch. I just merged something, so now I get to rebase, which... Yeah, if you didn't see, if you're still on Git, they just released a new version with some history editing commands that are inspired by Jujutsu. What was that new one? Arrange? Yeah.

...54olexsmir i keep forgetting about jj arrange
I haven't looked at this before.

01:01:06Do I have to give it a refspec or a revset? Yeah. What's the...

...37olexsmir if i'm not mistaken it's just jj arrange and you use J K to move coomits around
So if I said JJ range, like really what I want is like where I'm at. currently yeah but so its default rev set is apparently just my last local commit and like main is in a confused state right now and i was going to use arrange to nudge this commit i just made onto olexsmir it only works on mutable commits, if i understand correctly
this branch from the merge I did earlier, but I can't immediately write the rev set off the top of my head that is everybody back here. Because what I would like is this commit, this commit, and this commit with a tree. Yeah, well, that's fine, but I still need to see this one to arrange onto it.

01:03:16Can I say this is a local alias I have? I guess not. Yeah, so I can't.

...38olexsmir --ignore-immutable ?
It's not what I want. Ignore immutable. I'm going to mess that one up. I know how to do it correctly here.

01:04:00olexsmir oh i should alias jj up to jj git push, it's less annoying to type
as well deploy that fix all right so where was i was going through the pull requests before i remembered that i had my own bug says something about how rarely origins get banned that that's been broken for a little while use the main story endpoint for saving and hiding and merged stories don't show oh yeah okay Remember this. I thought I reviewed this.

...37And I asked for a rename. So this is all just hanging out. And because I took so long to review it, Federico is not immediately turning on my response. So this is one of those examples of why I try and keep a regular schedule and why I'm not super happy about moving off of monday to tuesday for these streams is i've broken the rhythm of reviews for this committer and the longer reviews take to come in the more likely it is they get abandoned or delayed on their own it's understandable but it's something i try and avoid without the really tempting thing of let me do a half an hour of pull request review first thing every morning, which is kind of what I used to do a few years ago.

01:05:52Separate form tag. The avatar cache. All right, so this has gotten...

01:06:07And again, no movement since I last reviewed. All right, so then the two open PRs left are SQLite and Anubis that are both addressing 18.14.

...49OK, that's pretty small. Let's see the rest of the code. Our HatchBox Anubis chain, right. So then this slides Anubis onto port 9000.

01:07:15The difficulty for servrobots.txt. Let's go look at the Anubis config. Let's figure out where I just dragged that browser window. Come here.

...47twitchtd ah, finally giving into anubis i see :)
Yeah, we've been talking about this one for a long while. twitchtd I bet the bot spam is crazy to manage
I would like to be able to turn it on selectively and just start with slash search because we had a very badly behaved bot. Yeah, you're getting there ahead of me. That hit our search engine 10 or 20 times a second searching for URLs. I think it thought that it was spamming URLs, but...

01:08:23There's no search on this, is there?

...31So where's servrobots.txt? Where do we think that's going to be documented? Configuration?

...55Did I typo? No.

01:09:17This is just some random site.

...32OK, maybe it's not documented.

...44That does not match the name in the file, but I'm going to assume that that's what the setting is. OK.

01:10:06Thoughts. Aggressive Brazilian scrapers. Definitely seen those.

...21Keep internet working. That's an interesting name.

...32We adjust 10. OK. Store things in memory. All right, so. OK. Why?

01:11:08So this is gonna intercept all traffic. And I would like to start with just slash search. And I'm puzzled by up here, we have a difficulty, but then the config

01:12:38Amen.

01:13:08Is there a list of the challenges?

...20And a refresh has no JavaScript, sure. This will have a much higher false positive rate. Is that saying, is that from the perspective of the scraper or perspective of the site? Hmm.

01:14:10Okay.

...41worry about this if we link into the issue tracker we're going to get a bunch of support requests posted as individual comments which is kind of disruptive but i don't know a better thing to link to besides hey the whole site is crashing under the load of badly behaved bots

01:15:09What is the deny status code 200?

...20Ah, because there are badly behaved bots that are just assuming that any non-200 response is a site issue. Hmm. Okay. I don't love it, but I get it. Yeah, that's Anubis in a nutshell. Like, I don't love it, but I get it. This is how we mitigate worse harms.

01:16:40I wish this had the word search on it somewhere. I would really like to limit this to just some paths.

01:17:06espartapalma hi, hello
Yeah, this looks like it's totally replaced by that thresholds. Oh, hey, good to see you again, Esparpoma.

...30Expression-based rule matching. Here we go.

...47Single expressions.

01:18:11What are these going in? Like they're a list. Name, action, challenge. Yeah, this has name and action, but it goes in bots? That doesn't feel quite right. Does it go in thresholds? These also have name, action, challenge. Where does this go?

...45olexsmir gotta love yaml
This is a rule that allows you.

...56This isn't YAML, this is just docs. The docs were written by somebody who knows this feature inside out and the software inside out. And I barely know anything about the software and have only read 20% of the docs. So for them, it's obvious where this goes. And for me, it's not on this page, and so it doesn't exist.

01:19:27But yeah, YAML is not exactly my favorite language. Not my favorite syntax for a programming language, which is a big hassle I had with Ansible.

...53All block contains a list of expressions.

01:20:02OK, so it's called all, but it's effectively a logical and. Then this is or, sure, sure, depending whether you think about these things as lists or predicates.

...22I guess that's implying I could do it elsewhere.

...34Well, there's a typo if anybody wants it. If you wanna make a PR to Anubis, this should be the word max instead of, I think it's mix.

...53Yeah. Background building really exposed users. This is pretty tempting. best combined with the weight and threshold system so you can have Anubis dynamically respond to attacks.

01:21:27Not going to try and patch this whole thing in. So this adjusts the weight by 10. And weight is just a variable.

01:23:43I said twice the number of CPUs, but let me say we've got, yeah, we've got four CPUs and the system bogs down around four, not around eight.

01:25:01Let me actually look at my calendar before I say something foolish. Yeah.

...26Okay. That'll make a lot of people very happy because everybody loves JavaScript and Anubis and is totally calm about it.

...47Actually, if you look at the open issue we have about it, a repeated thing that is kind of getting to me is

01:26:03that we want a really conservative configuration. And then people say, but I don't use JavaScript. And then I say, yes, we literally designed around this. And then they say, but I don't use JavaScript. And then they say, but I don't use JavaScript. And then in chat, they say, I don't use JavaScript. And then in blue sky, they say, I don't use JavaScript. I don't know why. just keeps coming up. Maybe I should rewrite that issue. graefchen Anubis surely can be annoying. But IMO it is more sad that it even needs to exist. limesSmile
Or I could swap in that issue number and just unsubscribe from it. I'm not trying to mock the people who can't run JavaScript, but it's a little frustrating to hear it over and over. Yeah.

01:27:04The thing I never hear is, let me help with some performance work so that you don't need to install Anubis. None of the people, and there have been at least a dozen between GitHub and these various channels, maybe more, IRC, it's come up there too. None of them have ever volunteered to say like, let's help with performance stuff.

...59What is this? What are we supposed to do with that? I hate these. I need to configure the bot to yeet people who post these like any update on this one kind of comments. But it's right there. You are viewing the canonical source. Please don't bump issues. We are actually working here.

01:28:32So let me pull up my email off screen. Thomas, if you happen to be around, I am looking for that. Is it SQLite is in the subject, I think? We had an email thread about it. That's not the email thread I'm looking for.

01:29:32graefchen Performance is hard. And I assume the easy way is in that regard a faster and safer call. limesSit
Honestly, Anubis is not necessarily faster or safer. It's that We've picked a lot of low hanging fruit and there's just limits to how fast Rails goes. I do expect that SQLite is gonna be a performance boost, just saving the network overhead on that. saving the process overhead on that. But here we go. I found the thread. So I'm catching up on that.

01:30:38So Thomas and I had an email conversation with a contributor, and I don't remember if they want to be called out, which is why I'm not saying their name. There's nothing particularly secret here.

...55graefchen I mean in the way that it is set up in a way shorter time-frame and might do its job good enough. limesSit
Story page issue is fixed. What's this?

01:31:06Oh, yeah, Anubis.

...17Anubis is now the lowest hanging fruit, right? Where the amount of time put in times the likely result is by far the biggest, but it also has a significant false positive rate.

...38And they've done a lot of work to minimize that false positive rate. And I can see that our proposed config tries to avoid that with the meta refresh strategy or meta refresh challenge rather than the JavaScript one.

01:32:33ghost_user_1984 Should I close that pr on Anubis?
Okay, so I'm not actually, I thought this email conversation had stalled out waiting for something on me when I was traveling and it is not. Where did 1871 get left off? Did I leave any notes in there? No, please do not close the Anubis PR. I actually just reviewed and left a couple of questions in there. ghost_user_1984 ok
Well, one question and one small request for more features. So if you have a minute to look at that, maybe we can try and get that deployed this weekend.

01:33:56pushcx @twitchtd ping
ghost_user_1984 yeah sounds good
ghost_user_1984 should be able
ghost_user_1984 Tonight
Cool, let's you and I chat on IRC then.

01:34:23ghost_user_1984 I’ll ping you after the stream
Trying to ping Thomas, because I remember there was one query that was crushing us, especially. And it must have been the confidence order path. I didn't leave myself good notes, and now it's been like six or seven weeks.

01:35:11This is the whole PR from the top. Is there anything in model's comment?

...37No, this is the port of confidence order path.

...48So realistically, the next thing for this is ripping out this query.

01:36:15yeah and there are kind of two ways to pull it out i could try to rewrite it to something that

...45is a recursive cte that sqlite is happier with or i could throw it away and write the one plus n version that sqlite is famously pretty happy with or i could be a grown up engineer and wrap some performance testing around this because we know exactly what this hotspot is Yeah.

01:37:48So.

01:38:26twitchtd hey sorry I just stepped away and am back
Let's just put this here in the general notes. Oh, hey. No, you got nothing to apologize for. Not like on call or anything. I was just struggling through remembering what went wrong when we merged 1871. And I got there because I didn't actually explicitly write it down in any of the pull requests or issues, but it was the confidence order path recursive CTE, right? And so I was trying to think of what I want to do to test that because I think that thread that we had going with the contributor who. wrote has kind of fizzled out. So.

01:40:07twitchtd ya I was basically waiting for him to send a PR
twitchtd but it seems like it isn't going to happen
Yeah, I mean, they had a couple of... I've got their notes off screen here. Yeah, I don't think it's going to happen even if I ping. It's been too long. Just too much other stuff came up and that unfortunately is the theme of the day that if you... don't respond to contributors they go away because everybody has lives okay so yeah so one suggestion there was two three

01:41:27And then secondary issue was undersized caching, which, boy, do I want to talk to somebody who is a SQLite expert before we tinker with it. veqqio Hi friends
Because this suggestion was to just crank the map size up to three gigs But I don't think Prod has that. Oh, hey, Beck.

01:42:12Big baby, it's not playtime. veqqio FRACTIOUS
I played with you so much today so that you would not be fractious during streaming time. Oh, why do you need so much love? It's almost like you're a mammal. davidofterra Spirited works well.
yes this is the official stream adjective for slightly misbehaving cats a vet applied it to my mom's cat years ago and it became well a household meme you know a family meme my mom's cat is actually a sweetheart but not at the vet this big boy is the exact opposite he is a bit of a chauvinist if the vet tech is a lady he is incredibly sweet to them and lets them do anything spirited also another i love how like 40s 50s old-timey that is that's that's a charming word yeah so this one is a big beefcake at the vet and is kind and obeys. And then they're like, gosh, why does he have the red jacket that says he bites people? And it's like, well, because he bites us. He puts his, not like bites bites, but puts his teeth on to get attention, which is not a good behavior.

01:43:51graefchen Cats at the vet are weird. One of ours was chilled the last time and the other already planned everyones demise. limesNodders
is this begin concurrent is this like a real thing oh yeah we used to have a cat that was an absolute gremlin at the vet in that she once escaped and ran around the inside of the back of the vet when she was back there supposedly drugged out of her mind after surgery so she ran around trying to pop her stitches i guess and they were like we gave her not just cat valium but cat anesthetic she should have been unconscious but she was just an incredible gremlin

01:44:51veqqio Did the switch to sql happen or is this investigating it again?
okay new model oh yeah i've heard about i've heard about this downside it starts the transaction without requiring a write lock so and rails makes it real easy to start a transaction do some reads and then do your write which is technically what you should be doing but you really don't want to hold a write lock all the way until the end vec no it did not happen that we switched to sqlite and then some life happened for the last two months and now we're coming back to it because we have a pretty good idea of what the performance problems were caused by and how to fix them so this is me reading up on how to fix them and then i was thinking i could try to instrument a little so we're not just shooting in the dark on prod which prod is not dark prod is on fire veqqio ...shoot the fire!
shooting in the dark while on fire mix in the metaphors pretty badly on that one this optimistic locking like but i don't want to i don't want to have to have this locking concern be reflected up into my code, shoot the fire. Yeah, well, there's the tracer bullets pattern. Tracks visited modified set of versions.

01:47:02I'm not really too worried about write contention.

...15And I realized that's like a horror movie line that I just uttered, like, let's split up, or let's go down into the creepy basement of the haunted house, or let's read from the book bound in human flesh. I'm not too worried about right lock contention. He said, as, you know, Hans Zimmer steps on the organ.

...47But we actually have a pretty low queries per second rate for rights, because we are not right heavy. We are very, very read heavy.

01:48:22graefchen I think burning the fire would be way more effective. limesNodders
We didn't start the fire. Prada's always burning since the world was turning.

...41This is just all right side.

01:49:12You know, I was just griping about the, like, any update on this? I have to say, I don't know if this person is involved with SQLite, but... Great. The answer of, like, write up a thoughtful article or spend 80k a year. Those are actually a great couple of responses. Maybe I can copy and paste this for when people get mad about the site, including Anubis. Although I've been incredibly reluctant oh yeah here's a literal like any updates okay so stefan is indeed an actual sqlite employee or owner i know they're very small so yes this is indeed sequel light saying if you want it go find your wallet which is a lovely response we have a little bit of the opposite with lobsters because if you want it pr is welcome i try not to use that phrasing because it's such a rude brush off sometimes but don't want to take actual donations for lobsters because then you have to worry about have you created a second class of citizenship because people are going to automatically it's just the way people work they are going to assume that people who donated or subscribed do not have the rules applied to them in the same way which is a fair thing to suppose because I mean it's Yeah, I would rather not get too spicy, but like, we have seen examples of this happening. We have seen examples of this in the last year. Oh, look, it's Ruby code.

01:51:48All right.

...54graefchen You take only donations in work. limesNodders
ndbe Which wm is this? cwm?
graefchen should be awesome
so how do i want to test like uv i feel like this is isolated enough that we can do benchmarking directly from ruby because this is this this code that pulls out comment trees is not just our most popular query it is The single story view is a majority of site traffic. Over 50% of hits that make it through to Rails are trying to load individual story pages. So I have very high confidence that this is the problem. And yes, GraveChain is correct. pushcx https://push.cx/stream\
This is awesome WM. There's a FAQ up at, let me throw the link in. ndbe thanks!
push.cx slash stream. I think it's something like, what am I looking at? You'll have to take the backslash after that.

01:53:06Yeah. So what's the... We have the benchmark. Yeah, the benchmark perf gem is in here.

...18Okay.

...32Yeah, OK. So how do we want to perf test this? We can make a standalone script.

...55OK. keep picking like a couple of things there we go just want to be able to boot but i can't just performance test the query because then i can't check the 1 plus N version, because then it's not a single store anymore. It changes the flow of the program enough.

01:54:46Yeah. So I really want to actually test from Rails. So through the whole render stack.

01:55:11knew there wasn't going to be an example of it.

...48What do we want, IPS or CPU? They're inverses of each other, right? IPS has the slight benefit that number go up. That's always satisfying.

01:56:19I'm going to be not here.

...41Actually, I want to start here.

01:57:10Just put everything, see where that gets me.

...38Okay, not... Right, not super readable, but not so bad.

01:58:23yeah that's reasonable just trying to get some kind of summary out of it you know

...55let's find yeah rails is doesn't have a nice entry point for i just want to simulate a single request does it

01:59:39sheet but there's gonna be something here about benchmark oh so the search term i searched for does not appear in the page github github i wasn't saying github it's the other g i'm disappointed by

02:00:23Derailed benchmarks.

...32A live project two years ago. That's recent enough.

...54Can I check a specific action?

02:01:21Yeah, I don't want to do mini-profiler. Because mini-profiler is nice, and we have it installed. It's not showing up because I'm not logged in, I guess. Oh, because it's not local. But that only gives me performance results on one page load. And while that's enough for huge changes, I would like to be a little bit more rigorous.

02:02:07My app is slow. Well, aren't they all? Yup.

...18Hmm. How about an action?

...28No, how about a path?

...53Do it through the test, right?

02:03:05Oh no, these are.

...15We're creating performance tests. That's not quite what I want.

...36Okay, robot, tell me something useful.

...57Oh, here we go. Someone with my exact question.

02:04:08I was kind of hoping to be in process. I guess that works.

...47Jamamp_ Does ruby/rails have a unit testing framework that also supports benchmarking? A la Go/Swift/probably many others
Jam Amp, I went past, there's a project there called benchmark-rspec or rspec-benchmark that includes some performance testing. Jamamp_ I just joined again after some work meetings, so I probably lost some context
But what I really want is something that's a little bit more shaped like, let me run 1000 and see how it goes.

02:05:20Jamamp_ Ahh yea that'd make sense
pushcx https://github.com/piotrmurach/…
Ah, we're here. And that's like, that almost gets me there, but I kind of would rather not use a screwdriver as a hammer on this one. So let's pick somebody with a high number of comments. Oh, speaking of, in case anybody didn't happen to see it, Actually, I expect most people wouldn't happen to see it. pushcx https://bsky.app/profile/push.c…
I threw it up on Blue Sky, but I got curious about how many words get posted to lobsters. Here we go.

02:06:12so the notice this is a partial month and i did it right about what four days ago so right about the middle of the month so that's why this first row is half of the others but this is averaging i want to say like 600 yeah 618 000 words per month Jamamp_ We got some typers
pushcx https://lobste.rs/stats
just pretty big that's that's like 10 non-fiction books per month and that's just you know a very rough order of magnitude because some non-fiction books are short and some are just enormous but yeah if you're curious about that there's more at upster slash stats

02:07:14Do we have Apache Benchmark available?

...30Sure doesn't look like it.

...39So I guess. That's they did not read the question response.

02:08:00This is this is one. I don't want one.

...15You know, I could shove it into a test. That's right, because that's a... The Rails app would be booted. I could do the warmup stuff outside. I would have to have a single story view with a lot of comments instead of just manually creating three. How do I get that?

...56Jamamp_ That's what I was saying loadLaugh do it in the unit testing framework. And perhaps not let it run during all CI. You get the most control over it
Yeah, if I call this the Ruby benchmark tool from a test, I guess I leaned away from doing it because I don't want to save this code. I want to throw it away, but you know, I can throw tests away.

02:09:35I want this to be a quest spec.

...54Jamamp_ Comment it our or skip the test, come back to it later when needed loadShrug but I do agree it'd add to cruft
Yeah, we do have a slow spec, but yeah. yeah that's reasonable it runs a benchmark that's it

02:10:37Yeah let's do IPS.

02:11:17let's just say we'll have five parents and then 45 times do great comment on the story with parent comment equals going to be ugly i'm doing i know this is terrible and non-performant but i'm doing the easiest thing off the top of my head if someone has a more clever way to do this you're welcome to pipe up please but i know it's slow So let's say where we're on this story and order is ran first.

02:12:34That will probably even work.

02:13:04olexsmir i mean performance doesn't matter that much in tests
Jamamp_ especially in the setup, not the part that's being benched
but we're not really writing a test. We're just borrowing the harness.

...19Okay, so what are we looking at? The average is 48. The standard deviation is five. Jamamp_ units!!! where are the units loadCry
jerbot I haven't seen a stackoverflow page in ages.
let's let's tweak that though because for ips it said yeah it samples for two seconds let's do more like 20 seconds oh yeah ruby has some kind of serious performance jerbot hello pushcx!
or primitive obsession hey gerbot i haven't seen a stack overflow page in seconds but it does kind of date the ruby stuff now a little huh okay all right so here on the maria db branch we're getting an average of 49 with a standard deviation that's still 5. I honestly expected that to change.

02:14:46OK. So let's grab that, and we'll throw it just in here because I'm going to keep notes. That's the wrong clipboard. There we go.

02:15:08And I'm just grabbing the whole line to save myself some formatting time. All right, and I should have Thomas's branch, which is called what?

...39Okay, the branch is called revert, even though we already reverted. Oh, it's like a revert of a revert. That's fair.

...59jerbot revert revert!
jerbot classic
What is this called? Yeah, Gerbot, if you weren't here, what? two months ago, six months ago. We tried to deploy this and then prod kind of immediately melted down. And so we reverted it. And now we want to try again. So obviously we're reverting the revert, right? So how do I get a bookmark for this branch? This is a thing I have only done like once on jujitsu. What is the last commit?

02:17:05Why can I not? oh 86 2779 i must have gotten a new line or something okay all right so when i did bookmark list i did not see this i did it off screen but

...48Okay, so now we're over on SQLite and we have conflicts. Let's try and pull them in. Let's make a new commit that is based on like to and on main cool there's our conflict and then we will one of the conflict become named that

02:18:44So let's go put SY after this new merge, which is SR. Good. And let's go edit the merge and figure out these conflicts.

02:19:12Jamamp_ this is why I love git UIs
Now, these are... Yeah, I had to bump a bunch of branches. Or a bunch of bunch. I can't talk. Guess I've been streaming for a while. jerbot I hate git uis, but I've never tried jj
Bump many dependencies for CVEs that do not affect us.

...46olexsmir i usually use mergiraf to solve those conflicts
Why are these later?

02:20:02Jamamp_ Git Fork is my recent fave, after Git Kraken slowed down a ton. Fork is macos/windows native which helps a ton there. No Linux yet though loadSad
all right let's just bundle update i know it contains merge conflict that's why i'm making you do this

...53davidofterra I use SourceTree for git and am pretty happy with it.
jerbot I'm a guilty user of the AIs and have them resolve my conflicts and run git commands for me NotLikeThis My memorization of git cli commands has atrophied
Let's just throw the conflict away.

02:21:07And throw that conflict away. What's over in the other file? This one is pretty minor.

...32Why did you start this a second time? Come here. Grab this. Not both sides of this.

02:22:13Okay. Wow. We know from cryptocurrency that what we want is number go up. So these are acceptable. Yeah, this is the kind of conflict that Claude would have just chewed through in about 10 seconds.

...42What was that commit? NY. Jamamp_ but doing it yourself kept it more salient in your head, and you're more sure that the changes were done correctly since they're by you
Let's go back up to the perf test and let's see if we can run it. Actually, I touched the gem file. Let's run the whole suite.

02:23:07config databases out of date, because now I'm on a branch where we're using SQLite. Make sure that I haven't left a password in there. I have not.

...32So this one. We'll do that. How was I toggling these before? I think I was going that. Nope.

02:24:03So now primary would become, yeah, that sounds plausible. and we're not going to run in production mode okay so i'm expecting a green test suite and then i'm going to run actually it's probably going to run the perf test in the middle of this so this is just going to stall for 20 seconds at some point Oh no, I didn't name it, so it ended with spec. Ah, how clever I was. spec perf.

...57And then this needs a different syntax in SQLite, so let's...

02:25:21Welcome back to Stack Overflow. How are we doing? Guys, I've seen your traffic graph. You're in trouble. What about random limit one?

...49jerbot you're migrating from maria db to sqlite?
olexsmir they are not doing as great as any ai nowadays
Okay, that's actually running the spec because it's taken the 20 seconds.

02:26:03Yeah, there's a bit of eating seed corn there to that one where the LLMs are useful because they're trained on all the Stack Overflow data and can regurgitate it. Jamamp_ @jerbot https://lobste.rs/s/oz7ebk/lobs… some context. they tried a bit ago but the site slowed to a crawl
in a nicer and more convenient fashion than Google searching against Stack Overflow. But we have badly disincentivized actually writing all these interesting questions and responses. Oh, thanks for digging that up, Jam Amp.

...49So that's not what I expected.

02:27:05It's not what I expected because SQLite is faster than MariaDB.

...20but it had a bigger standard deviation. Jamamp_ and lobsters is querying sqlite from in-process right, not over a socket?
I should probably have run this with like, instead of 20, I should have run it with 200 just to even things out. Because these two are within a standard deviation of each other. So what's my p-value here, like 0.5?

...57Well, let's see if I can still check if this ID. Oh, no, I know what it is because the error was. Yeah, I got to redo the MariaDB side. The problem was. SQLite was doing a full table scan of the entire comments table. And this test. does not have any comments that are not in that table.

02:28:39Lobsters, yes, the goal is to query it in process.

...49So you know, actually jam out. Grabbing this,

02:29:18What if I wrote?

...44Why am I always fighting my clipboard?

...56What if I just grabbed that story? Because it's got 60 comments. which is about the number we want. And then I ran against the development database, which has nothing in it right now. Let's call it perf. So I'll have to run the migration script that Thomas wrote. And then copy the database to a perf database, because otherwise I'm going to end up accidentally blowing it up. What's the script named? Well, it's over in that gist, isn't it?

02:31:02Dumpdb, yeah. So I gotta put back the database.

...25It's gonna take a second.

...38I don't remember how long this took to run. It could take a couple of minutes. So my general plan here is I'm going to dump this. So the way Thomas figured out to migrate from MariaDB was by using Ruby. So it grabs the whole production database, dumps it to a YAML file, and then I switch the config database back to SQLite, and I load that database dump. We can pull that up while it's running. Where was it? Maybe it went in lib tasks. Yeah. So it just says, give me all of the models because Braille thinks tables are models. And then let's dump everything out in batches. And then load is exactly that in reverse.

02:32:50Yeah, and it's ticking along. I want to say this got to be a couple of gigs. Do I have a dump hanging around on my home directory? Yes. No, it just found the dump.yaml that is this one we're exporting right now. But once I have that loaded into the development database, the SQLite development database, I can copy that SQLite file to another one just called perf, just because I wanna make a second copy of it Number one, so it's standardized between test runs. But number two, I don't want to accidentally run all of the SQLite tests and have them clean the database and throw all of that data away and I have to dump and reload again. Because the test runner does try to clean out tables.

02:34:04I don't remember how big this got. A couple of gigs.

...17Ha, I have it on prod. It was... 1, 2, 3...

...28It was 4.7 gigs, which is... That's... What is it? Gibibytes? Not gigabytes? So we'll see this number be... 5 million... and a little bit when it's done. Jamamp_ i pronounce gibibytes like gibby haha
So we're on the home stretch.

02:35:00Gibi? Gibibytes or Gibibytes? Yeah, I don't know. Now that you have me think about how I'm pronouncing it, I don't know what I just said 30 seconds ago.

...20So we're going to do that. And then we're going to say that.

...33olexsmir tech people are terrible at naming things
And I'll copy it to perf.

...54Yeah, I don't think anybody saw the full SI prefixes thing coming for computing at the time they defined it.

02:36:10Here we go. Of course, there are two more months of comments than there were. Ooh, so it went to 5.1. The production one didn't do that. All right, there we go. We're done.

...34All right, so now we can run loaddb, because I already swapped the database file over. And we already have stuff in the development database, don't we? yep so i probably could have just found another story and not done this but honestly it doesn't hurt to exercise all of this

02:37:35Jamamp_ So, how would you characterize Ruby/Rails? I have never touched it. And when I was looking at the lobsters repo wanting to see if I could contribute, I was a little overwhelmed. I am so used to C#, then Go, then now Express/React as frameworks. It'd be fun to contribute here and earn an account. I need to probably sit down and focus on just getting the dev env running in dokcer at the very least.
Could not find right because create just creates the database it doesn't put the. schema in there.

...49How would I characterize Ruby rails. I think that's a. Pretty broad question. But it's great to have a. broad question while I'm killing a couple of minutes for all of this data to load, so thanks.

02:38:26So Ruby is dynamically typed. which is going to be the big difference between C sharp and Go. And I think of Ruby as having really wide objects where objects tend to have both a lot of data on them and a lot of methods. And if you look at like the Ruby enumerable module, which is the workhorse of Ruby data structures, It's got to be at like 80 methods or something. Jamamp_ Ruby gives me strong Python vibes. And I've done so much typescript this past year it's not so scary thankfully
Because Ruby doesn't really have function composition.

02:39:18Yeah, Ruby and Python are kind of close cousins. They're not actually related in a family tree sense, but they... Jamamp_ And we like Rails?
they have a heck of a lot of similarities. If you use VS code, we recently merged a pull request from Chao to use dev containers, which apparently makes it very easy to get set up with a developer environment running. It's very much like a one-click kind of thing. Jamamp_ that'd be a good opportunity to play with devcontainers for the first time too
But getting the Docker thing that also has a lot of her work in it really shouldn't be more than an extra command or two. As for we like Rails,

02:40:28So I've been working with Rails for about 20 years. And so a lot of it's, lately I've been feeling a lot of its limitations. And overall it is nice and it solves a bunch of very real problems, but Its development stalled pretty bad in, I don't know, kind of ballpark 2019 until 2023. And I'm going to just gloss over why that happened and what happened after, but

02:41:27I don't know the the imaginary web framework that i've been writing in my head fixes a bunch of long standing issues with rails and i've been thinking fairly seriously about that, so I kind of only see that when you ask what my opinion on rails is. there's a. kind of a three layer cake where on the bottom there's Ruby the language. And at the top layer, there is Rails, the toolkit. Jamamp_ For having never touched Ruby before, I did randomly attend 2-3 local Ruby meetups (San Diego) and I got some of the high level drama lol
And the middle layer is the Ruby ecosystem. And Ruby the language is doing great. Ruby the ecosystem is not. And Rails is coming out of a long slump, but

02:42:31is the source of all the drama anyway yeah the if you're in california i know san diego is down at the bottom of california if you're ever up in san francisco They have the best Ruby meetup. It is basically like a small conference that they put on. I am incredibly impressed by their work. The stuff Evil Martians has been doing in the last year has been just hands down the best community work happening in the Rails ecosystem.

02:43:34espartapalma been there in SF with frequence... I was not there when you were
What was that file, like three gigs or something? Yeah, so we're only about halfway.

...51I have visited San Francisco a few times, including earlier this year, but I've actually never gone to an SF Ruby meetup. Either I didn't overlap or I had other stuff going on.

02:44:14espartapalma ohh... I though you were there
espartapalma I haven't been this year because family affairs
Jamamp_ As indicated bt the TWO chiacog tags on this stream loadLaugh
Jamamp_ duplicate!
hmm no no i'm i'm chicago that's part of why i keep sneaking chicago themed easter eggs into the site yeah two chicago tags i have two chicago tags what are they Oh, weird. I do have two. That's kind of special, Twitch. Why can I add the same tag twice? Well, I can delete one and save. All right. Well, thanks, Jamamp. I just fixed it. I don't know if it gets updated now or if it'll be updated for my next stream, but thank you either way. Jamamp_ I refreshed and it's one now
theyagich i didn't know chicago was the sole twin city
guess we are required there is kind of a chicago vibe where you're required to like hype the city up philly has that same level of chip on its shoulder nobody understands how much my city punches above its weight vibe so i've visited philly a couple of times and i always kind of like it because it you know is mentally similar to chicago

02:45:40yeah the i haven't spent much time in minneapolis the actual twin cities but that's not so bad at least it's midwestern oh you know as long as we're we're killing time for a second here if anybody hasn't seen it and i'm guessing most people are younger than it

02:46:09pushcx https://www.youtube.com/watch?v…
so i've probably got 15 20 minutes left on the stream so you know i'm throwing this out there to watch after the stream yeah here we go where's this the full thing yeah I'm not gonna play this on stream, but this is an incredible bit of Midwestern culture. It is a serious cultural touchstone. We don't use the individual phrases in Chicago, but a lot of the vibe is similar. And this thing, I wanna say it's gotta be about 40 years old now, this video. It's about a half hour long. And if you are in Minnesota, I think roughly every year somebody does a stage production of it. And that's a pretty good show.

02:47:10Could be worse, you might say. Getting there. So once that's loaded, I will copy perf to perf And then I can run this again. Jamamp_ I hope to visit chiacgo for the first time later this summer for Lollapalooza. If the waitlist opens up.
And I'll have to go back and run it again on the MariaDB side because now these are with synthetic data and then these will be with raw data and this will be maria db then we'll have sqlite base and then we'll have sqlite with story id and we'll have sqlite one plus n is pretty easy sqlite new rcte which i'm not sure i want to write because it's possible that like So what I'm roughly joking. Oh, Lollapalooza. There's a big show. Yeah. theyagich oh are you prepping for the sqlite transition?
What I'm roughly expecting is that this one's going to be about the same. So like, I'm going to guess ballpark about 50. And then the base here is going to be worse. So like, I want to say we saw significantly less. So I'm going to guess like 20. Low confidence on that one. With story ID...

02:48:59If it works... If it works, I'm really hopeful. And that might jump us up to more like 60. And then 1 plus N... It's really only gonna have to look up the thread routes by story ID and then everything after that should be PK of another comment. Jamamp_ thank you for "1+n". the "n+1" saying people use is the wrong order, and it irks me haha
So unless we hit like a Ruby side thing where there's a cost of going into active record over and over instead of just grabbing a whole list of comments, this might be even better. So I'm gonna guess like 70. and then like write it again I don't know maybe that does even better yeah you know I'm out here this is this is one of those like handful of pet peeve tech terms the other one is Ruby has this Jamamp_ ?. but similar, yea
operator so if i have a method or if i have an object like story and i say i don't know story.title you've been writing javascript so you'll get this one javascript has i think i think they do it as dot question mark where if you're not sure if story will be null or undefined you can do this and they they generally get called the null coalescing operator question mark dot okay yeah so in ruby the syntax is this exact same concept but this one i call this one the nil propagation operator because what it does is it just spreads those you know nil is null in ruby same concept but all it does is like who equals like well now you're taking this nil And instead of dealing with it properly or setting a placeholder, you were just propagating nil over and over and over. And nobody uses that phrase except me, but by God, I'll die on that hill. espartapalma the lonely operator?
Jamamp_ the tiny hill operator
All right, let's see if we can run this perf test. Lonely operator, yeah, there are a lot of cute names. The best operator in Ruby is the flip flopperator. And I definitely don't have time on stream to get into this one, so I'll let you look it up. Tiny Hill, I haven't heard that one. That's a great name. And you know, if you come for Lollapalooza, you will see no hills.

02:51:57Jamamp_ cause you're willing to die on your tiny hill for the nil propagator loadLaugh
Couldn't find the story with short ID, question mark. Oh, it indented. I just... Oh yeah, die on my tiny hill. Yeah. All right, what's going on here? Where's my story? This really looks like it's searching by nothing.

02:52:44Okay, it's there. Let's just not use my little...

02:53:03theyagich the first result on google for "ruby flip floperator" happens to be a lobsters post by someone named "pushcx" hmm 🤔 🤔
Still not finding it today. Changed you to perv.

...16We're getting around here.

...37Jamamp_ it's all a conspiracy
Yeah, there's the story.

...48You know, it didn't look in the, yeah, no, it closed out the wall. I was thinking like, is it possible that that one story somehow was sitting off in the wall and didn't get copied over, but no.

02:54:14oh yeah okay it it zeroed the database no it zeroed the table i'm seeing like a i got a vacuum don't die yeah this this wiped everything

...44Jamamp_ ripperonis
now it's tiny again okay so that's why i copied it because i saw that one coming you know as long as i'm doing this let's just take a quick backup out of the way So now we're seeing the limitation of trying to use the test system as a harness because the test system wants to start up and clear the database so that there's nothing from a previous run polluting it and making flaky tests.

02:55:47And Rails is so particular about owning the request response cycle that there is not a nice way to just load up a controller in context and say, fire this action. Is there? Let's try it from the shell.

02:56:45yeah and it it's so clever about injecting stuff that there isn't just a method signature so jam amp you mentioned python before the big difference between the two languages is cultural ruby is implicit and has clever quasi-magical things and python is like no everything should be explicit and right there on the page so day-to-day python is a little bit tedious where rails feels Jamamp_ you would _hate_ Go LUL it's so explicit
kind of effortless once you learn these conventions and hookups. And then as soon as you want to do something that's a little bit odd and unanticipated, it's straightforward in Python and impossible in Ruby and Rails.

02:57:38I get what you're saying. And yes, Go is definitely on one end of that spectrum. pushcx https://gleam.run
theyagich lobsters gleam rewrite?
But I have to disagree with the what would I hate because I've been really getting into a programming language called gleam that is also way over on the explicit side of the spectrum.

02:58:15I would have to write the web framework. They don't really have one. Which is kind of tempting. Jamamp_ swift is also pretty expressive CaitThinking
But it's like, is that where I wanna spend several thousand hours of working time? Jamamp_ lobsters go rewrite would be fun but not worth it for myself lol
You know, if I'm doing six hours of stream a week, That's what, six times 50, 300. So if I start my office hours writing my own web framework, I'll be done in only about three years and then I can start porting. That's, yeah. Yeah, Jamamp, you were talking about AI coding. That might be a fun thing to throw at Claude. We have a couple of places where the code base is not super well organized. Jamamp_ twasn't me. boo claude loadLaugh
but kind of funny.

02:59:29Oh, I'm sorry. It must've been the other new person mentioned something about just kind of hacking away at stuff with AI.

03:00:04Jamamp_ gleam looks very interesting though!
theyagich i need to hack away at lobsters too at some point
Yeah, so this is... One of the things that make Rails a framework and not a library is Rails wants to own the thread of execution rather than it's a set of tools that you can call into. Can you just call app.get, can I? Is it that simple and I've been griping about it? This must have been added sometime since the last time I lost a ton of time to this.

...46What was that short ID?

...55Okay, so I got a 403.

03:01:03Jamamp_ where's the Rosy the Riveter style lobsters open source contribution propaganda when you need it
This feels like development configuration.

...22Just grip for.

...49theyagich "We Can Ruby It"? hmm
theyagich we can workshop it
I like the idea of Ruby the Riveter.

03:02:40theyagich ruby the riveter is nice
theyagich because ruby is nice
theyagich im not looking forward to setting up the dev env though, thats always annoying with ruby
Jamamp_ Lobby the Riveter: too close to lobbying. Luby the Riveter: well, you know. Lobsty the Riveter!
The edge if you use whoa there we go if you use the container the dev container it's pretty straightforward. But honestly i've dropped enough of the external dependencies that it's not so bad outside of a container either.

03:03:08So if we're going to say that, can I just say grab this?

...42Yeah, see?

03:04:08Jamamp_ ah yay, docs https://github.com/lobsters/lob…
Now it's fighting me because it's doing something clever in the console. It's not available from a script. Rails is so clever, I can't get shit done.

...38pushcx https://twitch.tv/ChaelCodes
yeah that's all chill you came in much later than she spoke but here i can make it it's if you want somebody to watch on twitch who has much higher production values she's a streamer too

03:05:09Alright, so whatever this convenience app object is that's available in the developer console, it's not available in this script.

...26Action dispatch integration session. Actually knew one of those up or is that like also a thing that has a bunch of implicit argument yeah, of course.

...47And rails being a framework makes. there's just always these kinds of. Rabbit holes to fall down where you want to do something that just slightly is different than the mainstream. I just want to call a fucking action. I'm getting exasperated where

03:07:03How do I pass you an app object?

...11Like where? Is that rails.application? But I don't even have, I guess I have lobsters application.

...39Is that the hoop I have to jump through?

...51I'm not optimistic. Whoa, whoa, it's actually working. Whoa. Working? Who does that? Okay, let's get the... Oh, and I can't cancel. Let's get the object out of the loop because we're testing more than we need to.

03:08:35And it would be nice if I could turn off the log so that wasn't confounding but i'm not going to hold my breath on that one yeah watching these durations go by that are up in the 270 millisecond range is a little painful average is three three so maybe when i said 20 I was a little optimistic. So let's let's grab this.

03:09:36So that gets me 20. Let's go jump back to MariaDB just so I can pick up that.

03:10:00Wow, these durations are in the same neighborhood. That's painful.

...10Let's close this. That's a confounder.

...20You also are saying an average of three. IPS is not going to be IPS is not going to be as useful as CPU because if the number is three, we're just not going to see a meaningful improvement.

...56Oh, why are you mad at me? Oh, because we don't pass time for the CPU tests, we pass.

03:11:13Warm up and I'm going to tell you how many.

...31theyagich is the switch to sqlite going to be particularly painful in any parts do you think? besides perf
repeat so let's do i really hope not i wouldn't have picked it if i thought it was going to be painful let's do well if we're seeing

03:12:01270 milliseconds and i would really like to do a thousand then we're talking 270 seconds which is four and a half minutes all right let's do 200 see where that gets us

...35why are these taking a full second what did i just change that took us oh because it's taking no those are individual hits are now taking a thousand milliseconds instead of that like which is wretched instead of that awful 270.

03:13:09What are we doing wrong? Because changing the test harness around this run should not have changed these individual requests.

...39What did I do?

...52What made it take three times longer?

03:14:09It's failing. That's what's happening. It's failing because I'm running it on the wrong branch. So all of this code is using Thomas's SQLite code, but I changed database config back to MariaDB. So it's taking so long because it's rendering the stock Rails error page. Do you have a status in you? Yeah, you're saying 200 because you're in debug mode, but that's 500. Okay, let's get back over to main.

03:15:18278 milliseconds was slow for anything. I typically in dev see like 80 or 100. And now I'm seeing nothing. What are we stalled for?

...46I want to talk to this like it's the cat. Sir, sir, what are you doing?

03:16:20Okay, so it was running, and the average was 592. All right, so I said 10. Are you going to take, call it 25 seconds? My prompt has the number of seconds that a command took to run. Okay. So you're saying 595 milliseconds. I don't know how to put the response body. Is that a real variable? Can I just print this? Make sure I'm seeing something like, yeah, something like I expected.

03:17:22Why did running it against SQLite print those log lines, but running it against MariaDB loses them? I'm very suspicious. All right, let's do 100, just so I have a round number.

...45Cost about $7.

...51So I usually stream for about three hours and we're at three hours, 15, but I'm, I feel like I'm actually close to getting what I want out of this. So I'm going to go an extra minute, although I'm not going to go into, like I can fill in the MariaDB. I can take the SQLite base and I can get the with story ID. Those are real straightforward.

03:18:30benchilling1 hi
the hassle is since i'm doing all this while streaming which you know pins a cpu and choose up some ram as soon as i end the stream i'll have to redo them just so i have a reasonable baseline that's not so bad hey ben chilling welcome benchilling1 what is the origin of your username pushcx
So same basic time.

03:19:08Jamamp_ maybe it simulates all the other stuff running on the server. good stress test!
So what do we have to change to go over to SQLite? We have to swap the database. I've been chilling. It's x86 assembly. Let's do this, this.

...30benchilling1 what is your history with x86 asm
And then we got to rebase back over. pushcx https://push.cx/pushcx
There you go, Ben. I wrote a blog post about it. In a second.

...57Let's. You know, I could actually leave this test here on the SQLite branch.

03:20:10Or here on top of main, because then if I jump back over to my merge, it exists. I don't need to keep shoving this commit around. Or maybe I do. Please note YAML. I've messed up the YAML.

...41What are we mad about here?

...49When I ran that script, was it running in test or development? It was running in development.

03:21:05Jamamp_ `test` primary is commented
I wasn't testing the database I thought I was. Yeah, yeah, because when I ran that, because we stopped messing around with that test harness, this wasn't a, this was a SQLite test. This was the SQLite base test. No, it can't be, because we were on main. This was rendering another million error pages. That's what that was. Peter. All right. Let's put all this away. Let's jump back onto main.

03:22:09Let's load the server locally.

...23This is that other fix breaking things.

...31May have to put that back. We'll see.

...40line two, column one. What are you mad about?

...50You're mad about that? Did the tab slip in? No.

03:23:07Jamamp_ in the yaml, `test.primary` is malformed
Cursing a block mapping at line two, column one. That must be relative to the block mapping. Test primary is malformed. Aha, it is. Yeah, there it goes. Okay. Thank you, JamMap. Good eye.

...46Now let's see if we get a reasonable number back. The interesting question is, how big was the standard deviation between the two runs, where I did 100 versus when I did 10? I have it over in the Vim history, don't I? So here, standard deviation was 0.06. And this was also 0.06. Yeah. I'm going to turn down this repeat. I don't want to sit and stare at the prompt forever. So if we swap 100 for 10, we should have about the same average and maybe a slightly wider standard deviation. Slightly smaller.

03:24:58Still plausible.

03:25:09So if it's taking 350 milliseconds

...16Really you know duration is 754 for the homepage if I click a random story on the homepage. 186 okay the random story I clicked on has. 21 comments, rather than 60 so it's in the right neighborhood all right.

...47So let's jump back over to SQLite, hopefully without mangling anything.

03:26:05And now we got to jump back to that merge, which is SR.

...19Two-sided conflict, because I've been tweaking the file.

...34Let's just fix this merge conflict by throwing away the merge conflict.

...45All right, so let's make sure that I can load the dev server. Yes. Rendered the same story. I'm going to just reload off screens because I want to see that duration number. About twice the MariaDB. That is the kind of painful performance hit we saw in prod when we actually deployed. So I feel like I'm seeing a real-ish number. All right. Come on, SQLite. Make me sad. Here we go. You gave me basically the same number.

03:27:30Maybe that first one was warm up.

...51Here's my query, store threads on comments ID. benchilling1 what frustrates you about x86 ? (i know x86)
And we want it to shove down. I don't think anything frustrates me about x86. I don't know that I've written any for, Over a decade. Where did that note go?

03:29:01benchilling1 ok, i asked because of "the frozen lake of frustration I have in my heart for x86 assembly"
Ben Chilling, maybe check the data on that blog post. Offhand, I would guess that I was griping about how much work you have to do to get anything done. benchilling1 i see
I am very happy in very high-level languages for the past little while.

...31Are you working in x86 or it's for funsies?

...57Yeah, this is the single story view, so we'll say, where storyId is story.id.

03:30:11Accept. That's a class method.

...26So I can't actually do that because it's not a proper query object.

03:31:02benchilling1 for fun
Jamamp_ 3h30m time check loadLurk
Oh, right on. Yeah, I got into x86 assembly because it was, well, a whole millisecond faster.

...22I feel like that's well within the bounds of noise. Yeah, yeah, I'm running way long. Let's just slap that in there to see if we get anything different.

...45350, no, that's the same number.

03:32:05benchilling1 i would say even +3% performance is enough justification to use asm instead of C or rust
You know been chilling it depends a lot on your resources.

...15So for us developer time is our most strongly limited resource.

...28403 blocked hosts example.com we're not testing it we're testing the rendering the error page which means that's probably what the maria dv side tested as well that's why the number is the same benchmarking is a pain in the ass oh look

...57Yeah, but are you actually benchmarking anything? Yeah, no, this is the Rails error page again. Because you don't like this? Okay.

03:33:24benchilling1 there's enough software, it's mostly lacking in perf imo. if we completely ignore the driver aspect
Now you like that. And this is the sort of queries we should see in production.

...42So this is, now I gotta go redo the MariaDB side of the equation again, because I really was probably just rendering the error page.

03:34:12davidofterra @benchilling1 You can probably get another 3% with profile feedback, LTO, and BOLT.
Wait, I don't understand this number. The duration on all these pages is low nine hundreds, but the average is. 1060. Oh. benchilling1 whats BOLT
Because there's more stuff happening around the loop of building up and tearing down the request. OK.

...52So if I took the story ID out, would I get similar numbers? No, I get the same, slightly higher numbers. So that adds 100 milliseconds of latency. So that's, what, 12% higher? So, okay, the story ID thing is a performance win.

03:35:34We're not quite doing apples to apples. Why can I not put this in here? So because this is a recursive CTE and you don't like me being clever, you know, this doesn't have to be a join. This one also can be a constant.

03:36:36davidofterra @benchilling1 LLVM's postlink optimizer. It reorders basic blocks and functions based on a perf profile.
But I do need this join because we have to go fetch merged stories. Yeah.

...52That's pretty cool. To reorder blocks and functions based on perf.

03:37:07Let's see if we can run this at all.

...19It's not story underscore. It's story dot.

...30You're mad about syntax here. Why? no such column comments dot story id what oh it's because it's c because out here the table is aliased that's what the error is about okay so that's a full render so now we can run the script

03:38:16Wow, adding, pushing those constraints down into the query in more places tripled the execution time. That's gonna have an incredible execution plan.

...40That's obnoxious. The only thing that should be able to do is hint the query planner better. And it absolutely demolished performance. Was it this side of it? Because I made like two, three changes there. Well, that's that's half of it. And then let's just lean on undo for a second put this one back and delete that okay we're back down into the 900s that's very unexpected to me

03:39:43Okay, well, fun with counterintuitive performance tuning. I will go tune up my intuition.

...58Yeah, my guess is all way the hell off.

03:40:07We never did get a good MariaDB because I didn't stop and think to run it in the command line.

...21I feel like I can salvage some dignity by saying, you know, the relative sizes of my guesses were roughly correct. Except I can't, because I did think the story ID was going to improve things, and instead it shot it in the butt. And then I pushed the condition down more, which I thought might improve it more, and it made it horrifically worse.

...58Man, computers. theyagich computers ResidentSleeper
All right, that's a good place as any to end the stream. Off to come back and get the MariaDB number just for a ballpark. Computers, how do they work? I hear there's a series of tubes involved.

03:41:42theyagich go back to vacuum tubes
Jamamp_ thanks for the stream!
theyagich thanks for the stream! (2)
I'll go back to analog computing. Well, thanks for hanging out with me. And JMAP especially, thanks for catching that YAML error. I appreciate it. So the next scheduled stream is going to be Thursday morning, Chicago time, 9 a.m. Oh, I didn't update the Twitch schedule. ChaelCodes Thanks for the stream!
There's something for my to-do list.

03:42:14I'll get that on. Ah, chill. Nice to see you again. Jamamp_ i blame yaml loadErp
I was talking up your channel a few minutes ago. ChaelCodes Schedule is weird, I think you have to access it from the profile
All right, so yeah. That's a surprising set of results, but that's why we performance test.

...41ChaelCodes Nice to hear that. <3
Yeah, I'm going to play with that some more. All right, folks. Yeah. ChaelCodes I've been in and out all stream - sick kid today
All right. Well, I hope to see you all on Thursday morning, whatever time that is for you in your time zone. And until then, take care. Ah, bummer. Best wishes to the kiddo.