Heap dump pair

01:04 pushcxI've got the stream started and I think we're both muted. [SPEAKER_01]: Oh, no, I'm obviously not.

03:35nogweii yo! sunsYo
pushcx Hello, welcome back! Fingers crossed that audio behaves today.
hejihyuuga o7
[SPEAKER_01]: Hello, folks. [SPEAKER_01]: Let's get started. [SPEAKER_01]: Oh, hey again, Hedgie. [SPEAKER_01]: And of course, hello, Nogwe.

...47[SPEAKER_01]: I don't think I need this anymore. hejihyuuga hi! hope you're doing well pushcx
Don't need to delete all of that. So today on stream, I had a fun article I liked a lot that I wanted to talk about for a second. And then... We are going to have Beirut, who has dropped by the chat a couple of times and is all over the issue where we've had a memory leak, which has been a long-term hassle for lobsters in production the last couple of months. And yeah, I'm doing well. I'm a little scratchy, a little phlegmy, because I went to a... What was it? dababyy1121 hey
Yeah, I went to a party on Saturday night and it ran long, so there was lots and lots of talking. dababyy1121 do you ai for coding?
I'm trying to find a voice coach to protect my voice, but so far they have flaked on me. It just seemed like the kind of thing that would be worth an hour or two. Hi, welcome baby. Do I use AI for coding? I have experimented a little bit, but we are not gonna be doing that on stream today because today is mostly investigating stuff. So the, article I really liked was this wrap your dependencies one. I thought it was a little bit thin, because I really had hoped that it would have a rule of thumb for when to wrap your dependencies. I know some people will say, and this kind of implies it's 100% of the time. But I don't think 100% of the time is totally worth it. Like all things in in programming is there's a return on investment happening, where There is a certain size where it's useful. dababyy1121 its my second year in college can i use ai ?
dababyy1121 in cs
At the beginning where it is very, very thin or it's in a small project or the project isn't going to run for a long time, which I know is very, very hard to predict, I don't want to do it because it's just a cost. But then as soon as a project runs for a year or two, wrapping dependencies can become very useful. pushcx https://lobste.rs/s/9nnrjw/wrap…
I wanted to talk about it. I talked a little bit here. I'll throw this link in the chat. I know it's on the scratch file in your second year at college. Can you use AI? I don't know. hejihyuuga @dababyy1121 read your syllabus
hejihyuuga unless pushcx is professor
I mean, you should probably check in with your professors to make sure they don't consider it cheating, but otherwise you can do what you want with your life. I'm not a dad. hejihyuuga which i doubt
So I liked that, the. You know, Hedgie, it's funny you say read your syllabus. Some of this stuff is moving so fast that the syllabi are not caught up. nogweii Professor Push! LUL
I've seen this with friends who have kids who are in high school where the tools are really outpacing the responsiveness of the academic institutions where it's really hard to come up with rules because the tools become like markedly better over the course of the school year. dababyy1121 as someone who wants to pursue career in software engineering is it okay to use ai in real world?
even over the summer the sonnet that everybody likes for coding is i think that came out in what may so i don't know it's i think we're in for a couple of years of big upheaval with school where they want to give people assignments and then grade those assignments to make sure that they practice the skills, but now there are ways that don't involve practicing the skills, and so you have a real split between means and ends. Is it okay to use it in the real world if you're pursuing a career in software engineering? I think probably. There are concerns about the copyright training, and will it own the copyright? hejihyuuga @dababyy1121 depends where you work. Some employers have policies about which ones you can use, because of the liability concerns
the end like that's an active court case i believe it's the new york times and some other media outlets suing about the output of llms i don't know what that is there's big discussions around energy use for llms for both for training and for use there was a good threat on lobsters about that just recently actually if we jump over to

08:13pushcx https://lobste.rs/s/zv2gvd/intr…
[SPEAKER_01]: Yeah, here we go. There was a sub-thread in the recent ZAI announcement about energy. I just left a comment there this morning. hejihyuuga I also don't particularly enjoy reviewing PRs that are primarily AI generated
So I was thinking about wrapping dependencies and how I've gotten a lot of use out of it. I've seen it at employers be very useful. I've seen it useful at clients as a consultant. And Lobsters has even gotten some benefit out of it. One of the places we do it in the lobsters codebase, let me make a new tab, is we have this class called markdowner in the extras directory. And right now it just wraps common marker and it does some setup of, okay, so take this markdown in and spit out HTML, which is a pretty simple interface. It is a much smaller and narrower interface. than all of Common Marker. And it does our custom stuff, like it walks the text node so that it can allow users to be linked in comments. So if you just at mention, if you say at push CX in a comment, it does all of this stuff to make links, which is super handy. But the interesting thing about it is if you dig back through the history, let's do that.

09:42hejihyuuga Everytime I wish we'd wrapped dependencies are times where we expose something from a dependency in our public interface, which makes refactoring challenging if ever we decide to move away from that dependency
There was a replacement seven years ago where previously this class wrapped an entirely different Markdown processing library called rdiscount. And rdiscount was a pure Ruby Markdown implementation. And so all of this stuff below the highlighted line was an entirely different library. And all of the stuff above, I mean, above it's like half style tweaks and fairly minor stuff. But below there's lots of like configuration and this made it very, very easy to replace our discount. I mean, I can't speak for the person who did this Yuki Izumi. I didn't talk to them seven years ago, but having done similar projects, I am sure it was much less painful than it would have been if there were a whole bunch of places calling Common Mark in the code, because we have comments, we have story descriptions, we have user bios. have the mod notes although those didn't exist seven years ago and having a consistent interface for what lobsters thinks markdown is and what kind of features it offers has been immediately useful and then having this maintenance cost reduced has been pretty good we even might be seeing it again the common mark dependency I haven't been able to update it. nogweii I'm of the mindset that if it is used more than a handful of times (where the limit is variable/team-dependent) it should be wrapped
They did some kind of big API change maybe a year or two ago. This dependency is pretty far out of date now, actually. And we have an open issue for updating CommonMark. Let's see if we can talk Nogwe into painting another fence. [SPEAKER_01]: Let me pull this up. pushcx https://github.com/lobsters/lob…
so here in the issues it is it is not here common marker yeah i called it update common mark er that is our wrapper or i don't remember if the gem name is common marker or common mark singular let's pull this over to the the chat there for anybody who's curious but that's what that nogweii hah it seems easy enough, with that wrapped file
wrapper is especially useful for is it's been easy for us to keep the api in one place and wrong keystroke oh bad typo too so as is usual for lobsters there's folks in the comments who disagree entirely with the premise of the article and then there's people who agree very nogweii Not going to promise anything this time @pushcx :P
positively with the premise of the article, self included. And I like that folks have those different experiences. Well, thank you Nagwe for considering the issue. If not, I will get to it at some point. So I just wanted to chat a little about that because I like this article so much. And then The main thing I wanted to talk about for code is here with Byroot. So let's find out as a moment of truth, Byroot, if you want to unmute and see if we have all the audio set up, can you say something? nogweii he said something!
something that works. nogweii hey byroot
hejihyuuga Something!
Okay. Chat. Did you hear that? [SPEAKER_01]: The voice that is not mine. He said something great. [SPEAKER_01]: Wow. [SPEAKER_01]: So everything worked. Not bad. OK, so Jean, I don't know if you're set up for, can you actually see the Discord chat, or are you just looking at the? You mean the Twitch? Yeah, I have it on the side. nogweii out of curiosity, what are you using to handle the voice call?
Yeah, the Twitch chat, I should have said. Sorry, there are now lots of moving parts happening. So we've had a memory leak since we switched over to YJIT. Gosh, must be. at least six months ago. And I am going to skip recapping all of the history, aside from it started with me buttonholing tenderlove at RailsConf. And then the most recent thing that's happened has been Byroot helped us get a working heap dump. So for setup Nagwe, we are using Discord, and I'm using it for to bring Jean's audio in, and I was able to set up the OBS capture pretty easily. And then also, I am sending a second stream of just the one terminal to him so that he can see stuff live as opposed to on the 8 to 20 second delay that Twitch is on.

14:45 byrootAll right, so. The delay is not that bad on Twitch, to be honest.

...48 pushcxIs it not? The last time I looked at it, it was long i don't know i assume there are different levels of it you know if you have i've even seen a setting in somewhere in the twitch setup where you can add a delay and i think that's especially useful for people who play like first-person shooters and they don't want their stream viewers to see where they are right they're sneaking around anyway so i had the the two heap dumps and then i had your comment saying here's like five different tools that you can run on the heap dumps and i like played with the i think i made the the the flame graph and then i looked at it and i went well i don't i don't know what this means

15:37 byrootSo let's go into it. A very first step I like to do is just to look at the size of the tweet dumps. So you can use ls-h or just count the number of lines, just to see if there's really a big difference between the two. So it's these two here. There's the boot one. Yeah, double the size. That's promising. That's very promising.

16:04 pushcxYeah, you had said that because you wrote, and I'm kind of talking through stuff you already know because I want to make sure that the folks are able to follow along. It was in config Puma, you wrote this script for dumping. And you had said, I think, half a million on this. And I, in production, slowly turned this down to... I think I had to turn it down to 250,000 to get a dump, or even just 200,000. That would make sense with the... If it was 200,000 and then 200 megs, does that seem... No, it's just how many objects as opposed to bytes.

...45 byrootYeah, exactly. And each object has, in the JSON representation of an object, has a variable size. So it doesn't matter. Okay. Actually, it could be interesting for the audience if you could maybe do a less on one of the heap and show a few objects. So this is a format. It's very simple, actually.

17:06 pushcxBefore we do that, Didn't you have a script for making sure that I don't actually put strings up? I can send it back again to you if you want. I think I can pull it up. It was like one line, right?

...25 byrootOh, a bit more than that, but... I have your email. It's in my gist. Yeah. [SPEAKER_01]: I sent it to you on Discord. [SPEAKER_01]: It's in a gist. [SPEAKER_01]: Oh, I see. [SPEAKER_01]: Yeah, that's one second. [SPEAKER_01]: We're going to jump over to the hide my desktop for a moment.

18:12[SPEAKER_01]: Let's jump back to the standard stream.

...14 pushcxAnd now, Jean, on Discord, I'm sharing you the one terminal. So unless you also have Twitch up, you can't see that I'm showing this. All right. [SPEAKER_01]: Yeah, so let's grab this. [SPEAKER_01]: And I will go ahead and... [SPEAKER_01]: So do I just want to cat through it? Is that what you had said?

...48 byrootYeah, it outputs the stripped things. So you just, no, no, you can cat or you can, yeah, cat works.

...55 pushcxYeah, I was going to say to pipe to dot slash strips heap and then just put that in.

19:07[SPEAKER_01]: Ah, we have a bug.

...08 byrootOh, no, no, sorry, you don't cat, you pass the path, sorry. Oh, I passed the file. I wrote that in a page.

...14 pushcxyeah yeah i saw in your email now that i think about it but so we will call this the heat json okay and then as long as i'm here let me just go ahead and make the other one because it just takes a few seconds and you know [SPEAKER_01]: I have them off in the temp directory. I'm just going to go ahead and remove these so that I don't make any mistakes. Yeah, they're off in another directory, so I don't have to. Anyways. [SPEAKER_01]: So let's look at make more sense to start with the boot one then.

20:00 byrootYeah, yeah, yeah. The beginning is going to be more or less the same. So at first it's very hard to read, but if you page through a few pages, because at first you have the root object, so it has tons of recurrences. Here you have the shape, so you're probably going to want to page until, because this is a bit of a special object. Or maybe search for just, I don't know, string or array, and it's going to show you an object. Sorry, old caps. Yeah, okay, perfect, yeah. All right. Here we go. Let's see. Just so everyone understands, the Ruby has slots, like fixed size slots, and every single object lives in one of those slots. And so what you're seeing, yes, slot size. So you can see that this is a slot of 160 byte size. And so for every single of those slots, the Ruby Ipdom is going to represent each slot by a JSON object on its own line. And you're going to get the address, the type, and a few, it's going to depend on the type, but a few information.

21:02 pushcxRight, so I see things like this bracket in here.

...05 byrootYeah, exactly.

...06 pushcxhere you have a module that's a module so you can see that like it's a module sprockets processing and you can see oh yeah i didn't realize it wasn't still on us a string yeah exactly so this is you know you were explaining things for for the viewers but also you were explaining things to me this is the first time i've looked at a heap dump so i do know what shapes are and i know you know i took a compilers class in college but that's been you know 20 plus years so

...32 byrootOh, yeah, and also this is specific to Ruby, right? It's not a native programming dump. And so the important part we care about here is that, like, taking the example of this module you highlighted, there's an array of references. So that's address of all the slots that you have just here. And then at the end, you have like the mem size, which is like, because each object is in slot, but if it needs more memory, it might just malloc some more memory on the general C heap. And so with that information, with the references and the size of each object, you can do a graph of all of your memory, right? That's what the flame graph is going to represent. Now, of course, like an object graph is a cyclic graph, right? You can have like back references and everything.

22:18 pushcxWhich is a great way to leak memory, yes.

...21 byrootYeah. And so that doesn't map very well to a FLAM graph, which is only for acyclic graphs, right? So the tool I recommended you cheats a bit. It just takes the first reference and consider that's the one retaining. So sometimes it can lie a bit to you. It's not always super clear. But for the common cases, it allows to show you like, oh, the Sprockets processing module is holding that many variables that have that many ashes. And so in total, it's like two megabytes or something like that. That's what you're going to see on the FLAM graph.

...52 pushcxOkay, so I did make the flame graphs. Let's see. Let's pull one up, and you're going to have to watch on stream here. Yep, no worries. Mark this as visible.

23:09 byrootSo this was the boot flame graph. Can you, could you actually open it in a, because I don't think that, like this is an SVG, it's best open in Firefox, because if you click on those sections, then it zooms. hejihyuuga oh wow i didn't realize that obs worked on linux

...25 pushcxYeah, I saw that actually, because when I first opened it, it came in. [SPEAKER_01]: So let's go to mobsters and then boot. [SPEAKER_01]: Here we go.

...39 byrootHey, nice. So here, it's a pretty obvious gigantic object right in the middle.

...45 pushcxYeah, this one thing, it really surprised me because I started looking through things. And there was a bunch of module names in here that seemed related to test libraries. Like RuboCop went by, and I don't know why RuboCop would load in production. And then, you know, it's a million things. And then VCR was over here. See, I did play with this a little, but I didn't totally know what I was looking for.

24:11 byrootThat's extremely weird and concerning.

...14 pushcxWell, after I saw that, this is a minute ago. Yeah. [SPEAKER_01]: Do I not have my, here we go. [SPEAKER_01]: Then I went to the gem file. [SPEAKER_01]: Oh, yeah, I got rid of sidekick because that was unused. [SPEAKER_01]: And then somewhere in here... Oh, I thought I committed it. I must have done it on a branch playing around or something. But I was trying to move the production gems off into a different group. Oh, no, it was... Here we go. It was over here. I... tweaked the bundle install command to not install the development groups. [SPEAKER_01]: I just know where the task is. [SPEAKER_01]: It's here. nogweii @HejiHyuuga obs is great and crossplatform - it even runs in Mac!

25:19 byrootBut the thing is, even if it's best not to install them at all in production, even if they're installed, they shouldn't be loaded. So there's something quite concerning that's worth looking into.

...33 pushcxYeah. I swear. Maybe I did do it on a branch. I swear right in here in the bundle install, I told it to skip the... development and testing.

...47 byrootYeah, I think normally you do dash dash weave out, I think, and you pass development and test. But I'm not familiar with that. I'm only familiar with Bundler, not with Ansible. Yeah.

26:01 pushcxSo this whole chunk in the middle I was kind of suspicious of. hejihyuuga @Nogweii I had a friend who used to stream from their mac, but i don't know why it never occurred to me that it would run on Linux.

...05 byrootYeah, it's extremely suspicious.

...09 pushcxYeah. Otherwise, I don't know what I'm really looking at. Do you want to also see the later one, the bloated?

...17 byrootYeah, yeah, yeah, for sure. But it's actually surprising you have such a huge object straight out of boot. OK.

...27 pushcxSo this is an interesting shape here on the left.

...31 byrootyeah can you put your cursor on the class like the big so when you put your cursor we have more information on the one at the top like the root okay itn config so this is like if you put your cursor back so your rtn this tells us your itn data like your translation file do you even have like translation file no we pulled it out years ago because

...55 pushcxknow maybe eight years ago nine right we briefly used the rails internationalization framework but running lobsters has been enough trouble that you know trying to run it as a software project that supported multiple languages was just too much work and so right we pulled it out but maybe there's still something left in This seems like a significant chunk of memory to be in use.

27:21 byrootYeah, it's probably the default translation that comes with Rails and that amounts to 15 megabytes plus change. And that would be 11% of that heap.

...31 pushcxWell, that's significant. And then there's this big object over here, RubyVM.

...37 byrootYeah, so that one is VM itself. So it's lots of caches and things in the VM. It's totally normal for it to be relatively big. 20 megabytes doesn't scare me more than that. It seems totally reasonable.

...51 pushcxOkay.

...52 byrootBecause in there, you have things like the symbols, tables, things like that. Actually, I would like to break it up in the VM in multiple objects so it looks better on flame graphs, but for now, it's like that. [SPEAKER_01]: You want me to change something here? [SPEAKER_01]: Sorry. Oh, no, no, no, no. I would like to see the brown one in the middle, the big brown one.

28:20 pushcxThis one?

...22 byrootYeah. The problem is the frame graph doesn't give us any information for whatever. Can I see the red module under it? Yeah, why would RuboCop should not even be in production? That's RuboCop. It makes no sense why it's here. Yeah, and then these are its, you know, 1,000 or something. So that's where, yeah, it's all the RuboCop linting. But this is not a leak because it was there on boot, right? Yes, it was. We saw this over here. What's intriguing is what's different. Yeah. So this is something we should fix, but that's a baseline that you have, like an extra baseline that we can reclaim. The question is like, what else grew to make it look smaller now?

29:16 pushcxOh, I follow because if it's the same size and it is. If it's visually smaller, we know other things have grown. And, you know, the obvious one is the Ruby VM has grown.

...27 byrootYeah, but that includes, again, BVM. It's not that bigger. Yeah, like 20 megs is reasonable. [SPEAKER_00]: So then there's some new big things back here. Yeah. So you have your running threads. So those threads are your ongoing requests, basically. Or like the Puma threads. And each thread has like a fairly large stack, so they're quite chunky. So that's totally expected, to be honest. I think the unexpected thing is the ITN. And I think the big difference is that you'll probably have IATN configured lazy loaded or something because it should be on boot.

30:06 pushcxYeah. So I did a quick search while you were talking for mentions of IATN and I don't see anything explicitly enabling it, but then I also don't see that we tried to ignore it. So, is it. No, it's config application where you can turn things off. Yeah, and we've uncommented some of the things we don't use.

...35 byrootYeah, but I think I would need to look, but I'm afraid it might be too baked in to fully remove. But I suspect that over time... Something is causing to look up in ITN and which right, which loads a lot of, YAML files and strings and stuff, but I just never freed again. And that's what looked like a big bloat, basically like a leak.

31:00 pushcxYeah. Well faker, but then faker shouldn't be loaded in again. It should be, but then it looks like there's just a couple of hits for it and in the gem file lock. So I figured maybe something took an explicit dependency on it.

...17 byrootBut I only see active support. So dependency is not so much a problem because you can depend on something and just only load it in some situation. Right. I just thought that would narrow down the possible culprits for what could be loading it in fraud.

...31 pushcxBut active support is a big thing.

...34 byrootI think at this stage, what I would do is just bundle open the gem, put some raise statement or breakpoints or whatever, and try to run the test suite and see whenever it's loaded.

...49 pushcxBundle open i18n? [SPEAKER_01]: Yep.

...52 byrootOK. And so yeah, so it's basically like a key value store. And you can see the back end is what loads things. And if you look at the back end directory, there must be like a YAML or something like that back end. Oh, no, maybe not. Maybe I think it's simple.erb. So see, it's not initialized by default, and the first time you access the translation, it just stores translation. So maybe we go back in the initial file, which is like lib slash itn, and we should try to find a good point where if you go lower, I'm sure there's something that triggers lazy loading. Config equal, no, it's not that.

32:52 pushcxBase config? I mean, it'll have to touch config early on. [SPEAKER_01]: Eager load, see? [SPEAKER_01]: Reload, eager load. [SPEAKER_01]: Yeah. Whatever this function is with an enormous translate.

33:10 byrootTranslate. Yeah, that's conspicuous. Yeah, so config.local. Here it accesses a backend translate key. It's been a while. I haven't contributed to that gem.

...23 pushcxI don't quite remember the... Just judging by the size of that comment, translate is the main workhorse method.

...33 byrootYeah, it is. But maybe simpler. Can you search in that gem for YAML dot something? Like a capital YAML? Like the constant? Yeah. Here we go. Yeah, the load file is what we're looking for. Yeah, exactly.

...54 pushcxThe unsafe or the regular?

...58 byrootUnsafe. Yeah, it's the same anyway. So basically here we can put a, oops, sorry. I click on something. Yeah, here you can put like a raise or a print or whatever.

34:13 pushcxWell, I want to do a raise because we want the backtrace, right?

...17 byrootRight. Yep. [SPEAKER_01]: Or you can do Petscoder, but yeah. [SPEAKER_01]: All right. [SPEAKER_01]: So I will run rspec.

...28[SPEAKER_01]: Looks like we got to... [SPEAKER_01]: I should just say rspec fail fat. [SPEAKER_01]: Wow.

...32 pushcxIt's already there. We only need one failure as opposed to... Yeah, exactly.

...39 byrootYou might need to run in backtrace mode. I need to run with backtrace, yeah. Yeah. That's one of my personal pet peeves, is that your dependencies are your code. And so that makes no sense to hide your code by default. [SPEAKER_01]: Is it trace or backtrace? [SPEAKER_01]: Backtrace. [SPEAKER_01]: Or dash b.

35:09[SPEAKER_01]: Here we go. [SPEAKER_01]: I think the whole option is

...12 pushcxIt's just there, because there's so many layers to RSpec, because it's so clever. The whole function would kind of just not be needed.

...20 byrootFRANCOIS CHOLLETTE- Yeah, I think RSpec makes sense. I have no problem with hiding RSpec. It's really with hiding all the raise frames. But I guess they're huge, but.

...30 pushcxCHET HAASE- So we have n plus 1.

...32 byrootOK, so this is the for domain method. FRANCOIS CHOLLETTE- But then you end up in a view rendering something. Build path 3GX, check the available locals. You're saying this? This is in Rails. Holy shit. Oh, yeah, I know why. Because when you render a template with Rails, you can have a local name in the path of the template. You can do user.fr.html. You mean like this? The locals directive at the top of a partial? No, I meant local as in ITN local. If you want to internationalize your application, you might want to have... a template for the French view and template for English view. And then you have two different HTML, basically. And one is .fr.html, the other is .en.html. And so that's why ActionView is checking in ITNN how many things you have. So here we can go back in ITNN and check available locals. Sorry, my accent is terrible. It's okay. It's better than my French. I hope so. And we can see there's a way to turn it off, basically, like to set it to an empty thing without loading everything. Is one of these files? That was in the main file, which was lib slash itn.

37:18 pushcxOkay. Do you want me to keep in our rays or just...?

...25 byrootYou can keep it for now. Yeah, leave it for now. Because actually we would like to get to a point where we don't hit the rays. [SPEAKER_01]: So now we need to search for available local.

...43[SPEAKER_01]: I think available is a better keyword to search for, I think. [SPEAKER_01]: Enforce available locales.

38:02Oh, here it is. It's here. So there's a bit of metaprogramming, and it called config.availableLocals. And so now we need to go see the config object, which is in config.av. And there must be another, the same method somewhere, right there. Yep. Which available? Yeah. So it's initialized to Neil. And when you actually see the force the first time it says a backend and accessing the backend load, all the stuff. So how was this line?

...39 pushcxI'm sorry to distract with the basic Ruby thing, but this or equals, yeah, it's a no up.

...46 byrootYeah. Shouldn't be there. Okay. So the question is, I wonder, can you just scan this method and just return empty array or something just to see?

39:00 pushcxShoot, now I'm getting all the... Yeah, I just wanted to check to see... Oh, we do still have a locale file in the Rails code base. This is probably dead code now, but maybe that's part of what's prompting this? Or it's...

...18 byrootOh, no, no, no. It's loaded way before that. I just want to see if that's the only thing you have. Because if you actually have things depending on it, it's...

...28 pushcxSo you want me to add another raise here?

...30 byrootYeah. [SPEAKER_01]: No, you just return empty array.

...40Just to see. I think we can configure it so it does that. And I just want to see if it still explodes everywhere. Okay. And then I'll run the specs again? [SPEAKER_01]: Yep. [SPEAKER_01]: Let's see. [SPEAKER_01]: Oh. [SPEAKER_01]: OK, so most of them ran. What do we have? And now the next step is a bit whack-a-mole, but OK, so you do have some code. OK, yeah, you do have some code to. Oh, we call numRootDelimiter. I can see why that would hit. It just raised. Rails have a number of helpers that can use translation so that you can configure them, basically. So yeah, it's still going to be loaded. And we cannot really afford not to load it, actually, because of those helpers. So I think what we need to do, but I don't understand, I mean, should already be the case. Sorry, I'm going to show you something in Rails. Wait, you're on Rails 7.1, sorry. I believe 7.1. Yeah, 7.1.3.2. [SPEAKER_01]: Oh, wait, wait, wait. Yeah, okay, I think I see the problem. Do you have Rails 89 in your gem file? I don't think you do. [SPEAKER_01]: No. Oh, i18n-rails? No, rails-i18n.

41:17 pushcxThere's nothing with i18n, period, in the gem file.

...21 byrootSo what's happening... I'm going to send you some URL in just a second. [SPEAKER_01]: Wait. Huh. nogweii this is fascinating stuff btw
[SPEAKER_01]: I need to find this code. So basically, when we looked at... At the main file of the gem at first, there was an eager loan method, which I added a few years back specifically for that issue, because you do want to load those translations. If really you need two of them, you're much better to load them during boot, but they're shared between all the processes. Rather than having them loaded in each different process, because I don't know if you're running like 20 Puma workers, then you're not loading 20 copies of them. Yeah. So it's a big way. So basically, if we cannot get rid of that thing entirely, the next best thing we can do is to get it to appear in the boot EAP dump to ensure it's loaded during boot. So you can just go, we're going to do it simpler. You can just go in your config.application.erb of your app. And there's something that's called config.eagerloadnamespaces. I don't know if it's already there. You might need to add it. Yeah, you need to... I'm going to send you load namespaces. I'm going to send you a link. Yeah, yeah, yeah. Makes sense. You say eager load namespaces? Yeah, eager underscore load underscore namespaces. Sending you the link to the doc. And that's supposed to already be an array, so you can shove something into it. pushcx ttps://guides.rubyonrails.org/configuring.html#config-eager-load-namespaces
And you can show the ITNN constant, basically.

43:22 pushcxOne moment. I'm going to share that link. pushcx https://guides.rubyonrails.org/…
I'm going to share that entire link with the Twitch chat so folks can follow along. And then...

...34 byrootthis back off screen so that little floater goes away so it expects a list of names so we just want to say yeah and you yeah it's a itin and constant not even a symbol like the actual constant oh so capital i it's yeah And that should be all. We can reset the modification we did in the gem. So you can do gem pristine itn in your command line.

44:08 pushcxOh, that's much easier than what I was about to do.

...11 byrootYeah. Don't worry. I do this for a living.

...16 pushcxWell, I am not unfamiliar with opening up a gem to insert debugging statements. But usually I just do uninstall and reinstall to make sure I'm back to normal. But there's a command for it. That's lovely.

...32 byrootAnd now we'll see if you'll add boot to my thing. Yeah, OK, you should be fine. So it at least runs. So this is something I would try in production. Might save a bit of memory for sure if what we observed is correct. Sure.

...56 pushcxI want to run even just the development server because I know This is one of those things where I got burned 15 years ago and I'm still nervous about it, but I have seen that the boot process for tests is very slightly different than the boot process for... Yeah, so actually, if you actually want to exercise this, you need to be in ego-load mode.

45:20 byrootOh, so I have to run it in there. Sure. Production mode, yeah, or you need to change your development configuration to be ego-loading.

...31 pushcxI'm going to change that because last time I tried to run it in the production mode, there were like four different code changes I had to make. I think I have a branch, but that's not the kind of thing I want to hash out. So let's try that and then run Rails server. OK, so it does actually boot. And if I reload here, local host.

46:01 byrootOh, but wait.

...05 pushcxDid you see something interesting in the Rails output?

...07 byrootNo, I just remembered something. It's like you don't run preload app, Puma preload app in production, right? You use a state deploy. So you cannot, Puma doesn't load the application before creating the worker. It first creates the worker, then loads the application. So there's no amount of ego loading we can do during boot. [SPEAKER_01]: Just remember that.

...41 pushcxYeah, I realize that lobsters is kind of unusual by serving a substantial amount of traffic on a single VPS instead of having multiple where you can...

...51 byrootIt's not unusual. it's just something to, you have to work around. So everything I said before, I mean, it's still better to load it on boot. So at least you immediately start with that memory that you eventually require might as well require during boot than during the first request that needs it. it just felt like what I said about copy and writes and sharing, it's not going to apply. [SPEAKER_01]: Right.

47:20so I think I would recommend shipping this regardless. and then I probably later on when I have a bit of time and see if I can like strip out more of that because you clearly don't need it or barely. [SPEAKER_01]: That's fine.

...53 pushcxWhere am I writing here? Because, see, I want to write so that it's shared among the workers, but then you also just said it's not going to be shared among the workers. Yeah, so... And I don't know how to write a short version of this.

48:04 byrootYeah, sorry. Eagle load 19n during boot rather than first request.

...11 pushcxAnd what's the benefit of that at this point if it's not going to happen...

...16 byrootYeah, well, at least rather than having your workers use more memory over time and look like a leak, they use the memories they're going to use anyway at some point straight out of the bat. So it's just like even that. And so in terms of instrumentation later on, it's better. But it's true that it's not actually saving you any memory. [SPEAKER_01]: Okay, yeah, I can see how that's...

...46Because if you had this in place when you had taken the dump, we would have seen it in the boot and we would have said like, oh, yeah, but just like memory use during boot. That's harmless, right? Yeah, exactly. All right. nogweii time to first byte, for the very first request. Optimize that crustacean!!!1!
hejihyuuga the guest on the call is great
dr3ig @Nogweii yup
Because we're searching for, we're playing seven difference and this is one of the difference, even though it's not legit. Yeah. So, you know, the next big question is all this free book up stuff. I'm not convinced it's a leak because it seems to be taking the same space in the boot and the bloated. [SPEAKER_01]: Yeah. [SPEAKER_01]: But it shouldn't be there. If I... So if you want, we can look at another tool that sometimes gives you a very different picture. It's a heap-profiler gem. Sure. Do you want to just add it to the gem file? No, no, you can just do gem install heap-profiler. You don't need it in the gem file. Sorry, heap-profiler. I'm not sure what profiler is. It's uninstalled whatever I just installed. [SPEAKER_01]: It's probably a very old profiler, Jim.

49:58[SPEAKER_01]: Hey, Drake. [SPEAKER_01]: Welcome back.

50:00 pushcxAnd Heji, yes, this is Beirut, who has dropped by a couple of times and is very active on the thread about the memory leak, if you want to follow his home page or anything.

...10 byrootSo what am I doing? No, it's very simple. There's a heap-profiler command and you pass it like a dump.

...21 pushcxOkay.

...24 byrootLet's see.

...25 pushcxDo you want to start with the boot or the bloated?

...28 byrootWe can start. We probably want both. The way I use it, I have both in different tabs and I switch from different tabs to see a bit of the difference. Sure. Start with the boot one. Start with the boot one and we're going to look a bit at it. So here we'll have the boot. So the input is very long. So you probably want to go back to the start. [SPEAKER_01]: Sure. [SPEAKER_02]: Here we go. [SPEAKER_01]: Oh, yeah. [SPEAKER_02]: I did play with this tool.

51:00 pushcxI didn't know what I was looking at.

...04 byrootso yeah, it, it basically, it can be more useful if you enable, object profiling. So it can give you where the object are from, but it is so much more memory than usually you cannot do it when you're breaking down the memory leak. and it basically gives you a breakdown of the memory used by each type. So you can see that like you have eight megabytes of string, six megabytes of classes, things like that. Right. The bytegems only works if you have tracing enabled, so it doesn't matter. So here you have nothing really particularly fancy, but it's normal for an app that just booted. And then when you compare with the bloated one, sometimes you see that a specific type just jumped out. So yeah. See, you have like 20 megabytes of strings we clearly didn't have, so you're like, That's probably UITNN strings in big part. The thread is somewhat normal. It's like if you have... How many threads have you running in Puma? Like five?

52:06 pushcxI would have to look at config Puma. [SPEAKER_01]: It's three or five. [SPEAKER_01]: Four. [SPEAKER_01]: It's four? [SPEAKER_01]: Okay.

...17 byrootI don't think we're writing this. That makes sense. Four to five megabytes. 4 to 5 megabytes per thread totally makes sense. You need the stack, the thread locals, and everything. So it's totally expected. And then iSeq is the bytecode. So 55 meg here, and on the other, many it is. So you have a teeny bit of code that's not here loaded, but nothing really major. [SPEAKER_01]: Yeah, there's nothing really crazy in there. Then it's like all pocket change. Right, yeah, it's down under a megabyte, so... And honestly, like, if you go up a bit, so there's a total on, like, one of the very first lines. So, yeah. So this is not real. I just want to be clear on something. This is not real memory. This is a total of memory reported by all the Ruby objects. But there's other, you know, like, there's things like if you load, I don't know, if one of your gems load a native library, like lib something,

53:26 pushcxI'm sure, I think Nokogiri has native parts, doesn't it?

...30 byrootYeah, exactly. That's not going to be reported, right? Because that's native code and everything. So this doesn't, like 143, that doesn't map with, I don't know how much actual RSS are reported on your workers in production, but maybe like 200, 250. So I just want to say it's totally decorated. And that also doesn't include, actually, does it include widget cut size? I don't even remember. So just to be clear, but here you still have like a growth on like purely is your Ruby object side. You have like a 50% growth, but we saw that part of that was the ITN is explained by the ITN thing. Right. Plus the Ruby VM taking 10 megabytes plus three threads. So I think everything is pretty much accounted for. So I don't really see a leak. Like in the Ruby sense, as in like you holding on tons of objects. But you definitely have some bloat in there that we can remove. That's for sure.

54:35 pushcxYeah. So before the call, to remind you, you had asked if we were using gem alec and i said i only learned about it recently and then i didn't want to change something about memory during this bug hunt and then you kind of slapped your forehead and said maybe that's the whole source of the problem do we want to touch on that yeah i'm quite certain of it so yeah it's

55:01 byrootSo as I said before, for those who weren't there, each object belongs to a slot that's pre-allocated. So when you do object.new, you're not actually allocating memory. You're just using a slot. But in many cases, objects may need some extra space. So they're called malloc, like a standard C malloc. And a problem Ruby users have, especially when they run Puma or Sidekick, is that when you do a lot of call to malloc and free, from multiple threads especially, you end up with memory fragmentation.

...42 pushcxSo you end up in situations where... You allocate many small objects and then you just free every fifth object. Exactly. And then...

...52 byrootYou have many roles in there, and so your allocator has many gaps, and you basically would want something like defrag, but you can't because it's live memory, so you cannot just move things around. Right. And this is particularly bad when you use multiple threads because glibc's value strategy uses different RNAs per thread to do different things, so it ends up wasting even more memory. and so that's why like the vast majority of the ruby community use jmanoc which is much better at avoiding fragmentation over time and most people complain about rubybloat it's just pure memory fragmentation so i mean if we're not showing any conspicuous 20 meg objects or you know 9 million strings sitting on the heap

56:45 pushcxI can see how that would point towards memory fragmentation as a possible culprit.

...49 byrootYeah, it could also be a native leak. So what we just did with the EVE dump is a good way to find a leak at the Ruby level. You know, if you have a native gem, something that, I don't know, Cogary, even though I don't think it's it, that calls into Sealand and that leaks pointers in there, like you won't see it on the EVE dump.

57:10 pushcxOn the other hand, it does feel kind of unlikely to me that Nokogiri at this level of maturity has a significant leak in it and that Lobsters is the only one hitting it. I mean, it does. It does.

...22 byrootBut Lobsters is not. I wasn't pointing at Nokogiri. I was just saying like any possible like native gem.

...30 pushcxAs just an example. But what I was trying to say is if we look at... The gem file, we actually have a fairly tight gem file. And almost all of these, like bcrypt, are very, very, very popular. And we're not using them in unusual ways. hejihyuuga This might be a dumb question, but does running ruby's garbage collector have a side effect of defragmenting allocated memory?
And so just my hunch is that none of these are likely to be.

...54 byrootYeah, yeah. That's why OcamRiser, I think it's just enable gem alloc and so it is. Especially since, like, based on the memory graph you published back on the issue, I think it totally makes sense. And I cannot believe I didn't think of that.

58:07 pushcxSo before we start adding GemAlloc, Hedji just asked a question that's probably better directed at you. Does Ruby's garbage collector have the side effect of defragmenting allocated memory? And my understanding was no from your earlier answer, because you implied there's no way.

...25 byrootYes and no. Yes and no. So that fragmentation issue, since I said there's two heap, there's a Ruby heap with fixed size slots, containing Ruby objects. And Ruby is capable as like a, it's not enabled by default, but it has compaction, so it's able to plug the gaps. And the only reason you want to plug the gaps is just because, it's not because those gaps couldn't be used. That's not a problem. It's just because it's faster to have a full empty pages, so you can just do a bump pointer allocation. That's why you want to compact stuff. But when you compact, sometimes it moves objects over, or it might just say like, oh, this object did a malloc on the CIF, but could have actually fit it in the object slot. So sometimes it's going to defragment a bit, but it's only marginal. hejihyuuga gotcha, thank you!
Once you're using the JLipC allocator and you've defragmented, the only thing that works is just restarting, basically. Yeah, it's exactly like that.

59:29 pushcxI saw this link go by on Hacker News a week or two ago, and it was so funny to me. All right, so I did a quick Google in the background, and it's very funny to see the second hit for Ruby gemalloc is, we solved our Rails memory leak. Do you know off the top of your head what steps we want to do to start using gemalloc? Because we could just implement that. If it's complicated, I can wait for a pull request or something, or I can do it myself offline. [SPEAKER_02]: But it sounded straightforward. I read one of these pages a little while ago.

01:00:08 byrootSo you can go very deep into autotune it, but the defaults are fine. And it's just a matter of like LD preloading the thing, or I think there's a gem that just does it for you, but I'm not sure.

...26 pushcxI mean, we can look at these experience reports. [SPEAKER_01]: I read, I think this one by Matt Heliski. [SPEAKER_01]: Yeah. [SPEAKER_01]: He talked about. Yes, this one I definitely read, but it's been a minute. This is probably the one where I learned about it. Yeah. hejihyuuga !lurk gotta dead to a meeting
[SPEAKER_01]: So when you use Docker, it's very easy. It had all these build steps that were not quite ours. hejihyuuga head to a*
[SPEAKER_02]: And then it said install the library and have this. hejihyuuga holy freudian slip

...58 byrootYeah, that's the thing. So my look is to be able to tweak it. nogweii @HejiHyuuga meetings are killer LUL
But honestly, even the default, I'll just make night and day difference. So just this first line you're saying.

01:01:12 pushcxYeah. Yeah. Okay. well then you're using poker or no, no, no, we use, Ansible for deployment and it's most of the lobsters just specific stuff is in this one file I had up earlier. [SPEAKER_01]: There's one more for, what is it? Come here. It was split out as a separate file. What packages are needed? Where was this required underscore packages defined?

...55[SPEAKER_01]: This is not something that gets touched often. [SPEAKER_01]: Ah, here we go. [SPEAKER_01]: So here you can just add [SPEAKER_01]: Is it called JEMalik? Yeah. This guy, no, he called it libJEMalik2.

01:02:14 byrootOh, yeah, you're right. nogweii ah you'll probably want to add the environment variable to the systemd file
My bad. Oh. Yeah, but it depends on distros, I guess. I suppose he's talking Ubuntu. Yeah, that might be it.

...24 pushcxHe is, which is what we run in prod. So real quick, and yes, Nagwe, you're jumping ahead, but that's probably where we're going to add the Enver. dr3ig dumb question, but what's the downside to using jemalloc instead of malloc? why is it not default
So off camera, I'm going to log into Prod real quick and I'm going to say apt-search gemalloc. And then I'm going to type it again, except I'm going to misspell gemalloc correctly. Because it works better. Yeah, so it calls it lib gemalloc2. And I'm just going to copy and paste it because I've already typoed it once. There we go. [SPEAKER_01]: So that'll get it installed. nogweii @dr3ig as I understand it, slightly higher memory usage on super-tiny scripts (e.g. your 5-line ruby files, etc) and also, it's Yet Another Dependency
[SPEAKER_01]: And then let's see, we have the Puma service file.

01:03:20[SPEAKER_01]: Right. [SPEAKER_01]: And you have your arms right there. [SPEAKER_01]: Right. nogweii but perhaps byroot has a better explanation
[SPEAKER_01]: So I'm just going to add this on the end effectively. [SPEAKER_01]: So you can see it without a wrap.

...40[SPEAKER_01]: Sounds about right. Okay. And it looks like I turned down the Puma workers at some point from that default of 20. Right. That was part of, oh, that was a really odd bit of debugging this memory leak. or a parent memory leak, I should qualify because it happened faster when there were fewer workers, which was very surprising to me. But then maybe that's very plausible. If it's fragmentation, then they would fragment faster because each worker serves more? Yeah, exactly. Oh, maybe we're on the trail or something. OK. All right. So then I guess the next thing to do would be to deploy. And then what, wait 12 hours to see if anything crashes?

01:04:29 byrootPretty much. nogweii @pushcx perhaps you could ask byroot @dr3ig 's question?
No, but I don't know if you have any telemetry, like if you can see memory usage over time or something like that. But yeah, as always, if you don't, then it's just wait to see how long it takes before the OEM or something just triggers.

...46 pushcxYeah, so let me show you what that looks like. It looks like I run, you know, watch-n30.

...52 byrootSorry, I'm going to answer, sorry, I missed it. dr3ig @Nogweii thanks; i'm sure they'll get to it
I'm going to answer the question of why, what's the downside of using jmaloc instead of maloc? Oh, yeah, please. Yeah, please, I missed the question. Like in the context of running a Ruby app in production, there is no downside. Many projects made it the default, like it's a default Redis allocator. Like if you're running Redis server, it doesn't care what the system allocator is, it's going to use gemalloc, it's embedded. There was multiple free tier requests to get it embedded in Ruby because it benefits a lot from it, especially the way modern Ruby is deployed with lots of threads. But The Ruby project is a bit resistant. Things like the la-cutter is a system concern, so that's why it's doing it. You can also, instead of running jmalloc, you can also configure the jlibc malloc to be a bit more conservative with memory. You might see some advice to use malloc n arenas equals two or something. I've seen that go by.

01:05:58 pushcxCould you say, what does conservative mean in this context?

01:06:02 byrootConservative as in, you mean like the Ruby project?

...06 pushcxNo, I mean, what does it mean to tune it to act conservatively? Does it make it larger?

...14 byrootSo it's always the same thing, right? Let's say you have a program. tumdum3 jemalloc is already in docker in rails 7.2
You need to allocate memory for five threads in a program. The easy way to do without having to lock for every single allocation, like to acquire a lock, is just to give them their own region from which to allocate their individual pointers. So you have thread one, it just starts at address zero, thread two starts at address one million, and so they just allocate each on their own, right? and so you give them each one arena but then like you you're gonna increase the fragmentation because if one thread one time need like you have a big request and that thread need to allocate 400 megs it's just gonna allocate those pages to give you the memory in there and then it's gonna release it but that's not gonna go back to the to the system because there's gonna be like some small amount of memory on each page Whereas if you tell malloc to say like, hey, you guys, instead of each having your own room, you're just going to share. And you might need to synchronize a bit. And you might need to whatever. But you're going to each reuse each other's garbage, basically. Like when the first thread is going to allocate 500 and then release it, and thread two needs to allocate 500, there will be space for it. It doesn't necessarily reduce fragmentation. It's more like it reduces the baseline that each thread, like baseline memory, each thread is going to keep. But the downside is that you're going to synchronize more, so your allocation is going to be slower, and allocation performance is a big part of Ruby performance.

01:08:00 pushcxI was going to say, doesn't Ruby do an allocation at every function boundary, basically, anytime you have... Local variables?

...08 byrootNo, not anymore. No? No. No, no, no. Not every function. But lots of APIs will allocate an object. And as long as it's an embedded object but entirely fit in its Ruby slot, it's fine. But many objects will need to malloc at some point. And that's going to be a problem. And so jmalloc is just much better implemented for that. It's just much better at allocating concurrently without using a much bigger baseline of memory. tumdum3 even if your page is completely free glibc malloc will not return it to OS if „later” pages are not completely free - sbrk vs mmap
And it's highly tunable, so you can basically tune the trade-off of like, oh, do I want faster allocation or do I want to use less memory? And you just can push the needle in any direction.

...52 pushcxI see. And if you're looking at Twitch chat, it sounds like Tom Dom knows quite a bit about the glibc malloc.

...59 byrootSo if you're using the docker file generated by RAID 7.2, yes, it uses jmaloc by default for that reason. And then even if your page is completely free, jmaloc will not return it to OS if later pages are not completely free. Yeah, honestly, I don't know the allocator intimately enough to unservice confidently. I was under the impression that most allocators are able to release fully empty pages. The problem is to have fully empty pages. It's very hard to ensure that pages are fully empty since you cannot compact.

01:09:40 pushcxYou know, the way you talked about upgrading to Rails 7.2, I had a moment where off-stream here, I pulled up my browser and I was like, oh man, how did we fall behind by a minor version? And then I see it came out only four days ago. So I don't feel so bad that like, yes, we missed it this weekend. nogweii Nose goes! Not me! @pushcx you can take that one on LUL

...58 byrootYeah, actually, I want the upgrade because, you know, I sent you up here to reduce the amount of queries and I think on 7.2, I can eliminate a couple more.

01:10:08 pushcxYeah. Actually, as long as I have you on the call, do you want to, I don't know how late it is for you, but if you want to take a minute, we can look at the upgrade to Rails 7 too. So hopefully someone will jump in on that. [SPEAKER_01]: But the, it's not a bug. [SPEAKER_01]: Did you want to talk through any of these if you have a minute?

...40 byrootYeah, sure. But whichever you prefer. I don't know which one you're interested in.

...45 pushcxWell, you just brought up this one, which was very neat. So for context, anybody, when a user writes a comment, which is a thing that happens about 200 times a day, somebody posts a new comment to Lobsters, there are a lot of active record callbacks that have been registered plus and some of them trigger more queries because comment is you know story is our god object but comment is our demigod object i guess if we're continuing that sort of structure so jean noticed that when comments are inserted they have to have a default score and The way I wrote this was to put in a place folder and call update score and recalculate, which is the method that kind of reifies and updates these cached values. However, Jean noticed that they are always the same because initial comments always go in with one single upvote from their submitter and nothing else. So, He opened this PR to just put in the default values. And I see what you're doing.

01:12:11 byrootAnd yeah, I had this problem of like a few tests fail because you have factories in testing to directly create a comment with more score.

...20 pushcxOh, is that what the test failure was? Yeah, I remember noticing there was one, but I didn't look at it because I thought either I would do it on stream or come back to it. Oh, I see you've touched the specs here. [SPEAKER_01]: Is that working now? [SPEAKER_01]: Yeah, it's passing now. I love the very nested expectations that nothing is changing. It's just kind of funny that that's how the RSpec syntax works. And I am totally happy with this very indented structure and just doing it all in one test instead of breaking out one expectation per test. I have never, I guess it's obvious from the spec file, but I don't really hold with the one assertion per test style.

01:13:04 byrootOh, yeah. One assertion per test is just a way to have extremely, especially if the setup is a bit slow, it's a way to Well, and everything is coupled to the database.

...12 pushcxAnd so it just makes things enormously expensive.

...17 byrootI think this patch is simple. I just don't know if my assumptions are correct. It's all based on my limited understanding of the app.

...27 pushcxNo, I think your assumption, your understanding is totally correct because these three bytes are... Let's see, one byte is the score, one byte is the number of flags, and one byte is the...

...46 byroothotness i think it's been a minute since i wrote this code and it was yeah this is i didn't compute this by end i just run the code see what it generated for every single comment and i just that it's magic value basically right

01:14:06 pushcxI think that's totally correct. And I'm surprised I missed it, especially on a big performance change. Yeah, I think the only thing I would change here in the code is this comment. This comment is now out of date because we don't immediately call update score and recalculate. Yeah, because I never read comments. Do you want to? And I think the other part of it is there's now a coupling between these magic values and these the spec down here and i'm totally fine with that but what i would like to see is a test that if we insert a comment yeah is this is this pr editable for me because we could just do these things I never know how to tell from where we are.

01:15:05 byrootYeah, you should be able to pull my branch and push into it. All right, let me do that.

...11 pushcxAll right, so I'm grabbing the URL off stream because it's over on the clipboard. So let's grab that, bring up my code. [SPEAKER_01]: Where am I here? [SPEAKER_01]: Wrong repo.

...36[SPEAKER_01]: I am going to move all of this stuff. [SPEAKER_01]: Let's just dump it in the git stash. [SPEAKER_01]: And nothing is.

...53[SPEAKER_01]: I will probably just drop that stash, but I don't want to think about it right now. [SPEAKER_01]: So let's grab your branch.

01:16:12do i keep getting this morning i've set some some git config on pull and now i get this warning all the time right if i look at your branch i have origin i have the thing i just wrote which is maybe what it's complaining about yeah it didn't update the branch pointer

...38 byrootYeah, you need to, you can do a re sorry.

...42 pushcxI, I miss, oh, this happened last stream. So I'm gonna have to debug my it config off screen at some point here. reset or let's go back to your commit. So now if I look at this, okay. I only have your one commit. So the two changes I wanted to make there were to update that test or that comment, which was... [SPEAKER_01]: So rather than saying... It's not a placeholder, I guess. Yeah. This is the initial value for confidence order because...

01:17:42[SPEAKER_01]: That's just the initial value. [SPEAKER_01]: And I still want to name check the function here.

...50 byrootYeah, to know where to look.

...52 pushcxYeah, and then the other thing, the thing I started saying that I want is, what was the name of that spec that you had to touch? Was it the byte packing spec? [SPEAKER_01]: Yeah, yeah, yeah, absolutely. [SPEAKER_01]: Bit packing. [SPEAKER_01]: I can't remember my own words. [SPEAKER_01]: you edited the these initial values the other thing i wanted to say was that you know based on the the name of the comment i wanted to say that when we load the comment out, that's that. But then also, if we explicitly call the update score and recalculate, we expect it not to change?

01:18:48 byrootYeah, it's the same test I implemented elsewhere before knowing this one existed.

...53 pushcxOh, did you already write this test and then just dropped it from the PR? Yeah, the nested expects, the test value. Oh, did one of them say that? I didn't see the confidence order was in there. So that was in the model's comment spec. [SPEAKER_01]: And then what line is that? [SPEAKER_01]: 163. [SPEAKER_01]: Confidence order, yeah, that's the last one. [SPEAKER_01]: But maybe it makes more sense in bit packing. [SPEAKER_01]: I don't know. You know, this code is so clever that I'm OK with testing it twice, because code that's this clever tends to be really brittle. So I'm going to just say that. Yeah, that makes sense. [SPEAKER_01]: Yeah, initial value from commenter's output. And what was the name of that method real quick that was assign initial?

01:19:57 byrootYeah. I don't remember. I have the PR off screen, so I'm cheating. All right. Assign initial confidence. You have it on the right. Yeah. I'm going to assign initial confidence. [SPEAKER_01]: That value is what?

01:20:25And I think you have to reload, after the update score and recalculate because it does direct like equal queries. Yeah, it does. That's true. So let's add that. [SPEAKER_01]: Okay. [SPEAKER_01]: And then that extras. [SPEAKER_01]: So if this is green, and your build was green, that's basically all I want.

01:21:07[SPEAKER_01]: Wrong number of arguments.

...08 pushcxOh, the update score in recalculate expects that it's getting the deltas. Yeah, 0, 0. Yeah, yeah. I should have spotted that because it's literally eight lines down. [SPEAKER_01]: That's OK. Oh. [SPEAKER_01]: Oh, we actually caught something. So the ASCII value of the tab is 7, I know off the top of my head. But we have one instance.

...39 byrootWhat you can do, if you go back to your spec to make this much easier, instead of doing to be bytes, you can do confidenceOther.bytes be equal. [SPEAKER_01]: And then you can use the derivation.

...59[SPEAKER_01]: You're saying do this instead? Yeah, to be, yeah, to equal and then an array of numbers. [SPEAKER_01]: I'm just going to, well, so that's going to be the, let's split this and then model comment.

01:22:20 pushcxWell, you can do the, yeah. So you're saying you expect that to be the 174, 82 one? Yeah, exactly. [SPEAKER_01]: Let's comment that up for a moment and drop it in here as well.

...45[SPEAKER_01]: I didn't run this spec, the comment spec. I'm going to see if that even runs for me real quick. I didn't touch it, so I expect it to be green. Whenever tests start failing unexpectedly, I'm like, OK, what do I actually trust? Because now I don't trust anything. And so I have to kind of take a step back. [SPEAKER_01]: All right, so.

01:23:13[SPEAKER_01]: Oh, tab is 9, not 7. Oh, my memory. So why did that one byte change?

...27So the third byte is the low byte of the comment ID. Oh, yeah. You can't pre-fill it. That's not fixed. And if this spec ran for you, you must have been running this spec in an isolated way. Or you ran the whole suite. So the tricky thing about this one, we can fix the test because this code is fine and you're [SPEAKER_01]: Wow, but is your still a... nogweii oh, the ID is explicitly passed in the `create` call @pushcx
So what's happening is the third byte of the confidence order is the low byte of the comment ID.

01:24:13 byrootOh, I see.

...14 pushcxSo, okay, so... Oh, yeah, and so like here in this spec, yeah, Nagwe just spotted it. Nagwe, who already has VIP for spotting bugs on stream. This byte is the low, you know, the first eight bits of the ID. The lowest eight bits, to be specific. And so it's just as a tiebreaker because otherwise, yeah. And so this spec, I really only, so I'm wondering. But wait. These two values are correct. Whoops. But this third one, it can't know until the comment is inserted what its ID is going to be? Mm-hmm.

01:25:07 byrootRight, so it's like a modulo 8 of the ID, right?

...11 pushcxRight, but until the insert query fires, it doesn't have anything, and we're trying to have

...20 byrootthis line of code self-confidence order run before the insert to save all those selects but we and a second update yeah so right we can do a second update without triggering the full might of the thing so i can but i can also cheat it so

...41 pushcxThe point of the third byte is to break ties for things with the same score and the same number of flags. If the third byte is the low byte of the confidence. So as a tiebreaker, it was trying to express that comments that were posted earlier should be ranked slightly higher in the tiebreaker. And so if we pre-fill the ID 255, We are just saying that, well, this brand new comment will be treated as last, and then that goes away as soon as somebody votes on it or flags it.

01:26:16 byrootRight. Because that's going to trigger the update. That works for me, but then that just breaks. It's not that it breaks the spec, but we'd have to update the spec to say like, oh, it doesn't change the first two bytes, not all the things.

...28 pushcxRight. So this becomes 255, and this is... [SPEAKER_01]: Yeah, or you can do first two. [SPEAKER_01]: I like both. [SPEAKER_01]: Can I?

...43 byrootYeah, dot first, and you pass two.

...46 pushcxOh, I didn't know that. Rails and Ruby have such just wide APIs. Although at this point, since I passed the nine up here, I almost just want to leave that off and specify that it's nine. Yeah, it's nine.

01:27:01 byrootYou know what? I would use c.id, because as a bystander, it doesn't know the form.

...05 pushcxOh, yeah, that's a good way to explain, because we had to stop and think about it, and Nagwe had to spot it. Do I do it down here? Yeah, I do it down here. [SPEAKER_01]: So if I said confidenceOrder.bites.last to equal that, and then this one must also change. [SPEAKER_01]: lights to equal yeah i'm changing the whole thing 174 82 255. all right let's run the whole spec green okay and then i can throw away the old code [SPEAKER_01]: And then let's make it clear what's happening.

01:28:25[SPEAKER_01]: And then.

...38nogweii would it make sense to change/include a new test
[SPEAKER_01]: see same first two values with the new with the comment id that feels like terrible grammar nogweii that has a large ID
[SPEAKER_01]: Don't bless me. nogweii so that you see the % 255
Would it make sense to change include a new test that has a large ID? nogweii er, % 8
Yeah, that's actually a totally reasonable thing to do to specify the fact that we are expecting it as the low order bytes. And so if I said 256 plus 9, I still get [SPEAKER_01]: I think you want to call 255. [SPEAKER_01]: Yes. [SPEAKER_01]: Yes, I do.

01:29:52[SPEAKER_01]: And are you not logged into Twitch? [SPEAKER_01]: I am. [SPEAKER_01]: Why? What's your username there?

01:30:04 byrootBYEBYE42. But it's kind of like a junk account. It's not really in.

...10 pushcxOh, well, you caught a bug on stream, so you get marked as a VIP. It doesn't really mean anything, but it is a nice way to celebrate that folks are catching my bugs over my shoulder. And you've certainly gotten it there. [SPEAKER_01]: So a little celebration.

...31[SPEAKER_01]: Let's run this whole file.

...38[SPEAKER_01]: Oh, maybe I was wrong. [SPEAKER_01]: Maybe it was actually 256.

...44 byrootYeah, I think you were right.

...45 pushcxOh, man, I'm going to have to evict you from being a VIP. nogweii probably want to change the "it" description for the spec
Wait, does that make sense? Was it wrong in the wrong direction? Yeah, no, I had it. Probably want to change the description for the spec. Yeah, I probably want to have something actually useful in there. [SPEAKER_01]: uses the low byte in the last. [SPEAKER_01]: Low byte of the ID in the last byte of confidence order. [SPEAKER_01]: That's reasonable. [SPEAKER_01]: All right, let's review. [SPEAKER_01]: What did I tweak? so here i changed the comment and then we got into this 255 as the tiebreaker so that's fine and then here why am i looking at the same thing again wrong file here's where we improved the spec with i really like this dot bytes to equal thank you that's such a nice Improvement. It's like, yeah, escape escapes. Yes. Like it's the difference between the test being correct and the test being readable. [SPEAKER_01]: So I'm going to say not that update comments backs.

01:32:28[SPEAKER_01]: test that id gets a place folder replaced good all right and what do i want to do here i want with that committed i want to jump over to master and then let's avoid squashing

01:33:00And let's push GitHub found a vulnerability, you know, like 90% of the time it's one of our HTML parsing gems and we don't use it on untrusted input. And so it's actually totally peaceful. I'm going to peek at that here in case it's like, oh yeah, we're not actually checking passwords. Yeah. It's another. rexml-denial-of-service. When it parses XML that has many deep accesses.

...34 byrootYeah, rexml. You're probably not vulnerable unless you're accepting XML in some way. I don't think we do. [SPEAKER_01]: But I'll go ahead and just bump that. [SPEAKER_01]: Yeah. [SPEAKER_01]: And where did that nice identifier go? [SPEAKER_01]: I always try and be fast about these things. [SPEAKER_01]: All right. [SPEAKER_01]: Shall we look at your other two PRs? [SPEAKER_01]: Yeah. [SPEAKER_01]: Yeah, one is very cosmetic. It's just like the second one, I guess, is simple. Well, let's look at that first then. I think you should see the results if you scroll a bit. Oh, wait. I'm not signed in. Oh, okay. Actually, you can see it on... Yeah, I understand. It's a good practice, actually. You can click on that teeny green checkmark, and you're going to see the difference, I suppose, even if not logged in. Sorry, next to the... Oh, I see.

01:34:50 pushcxSo it's splitting it out into four checks? Yeah. That's interesting. So they run in parallel?

...56 byrootYep. I did that because on my other peer, I got a failing thing on Breakman. So it's just not clear if it was my test failing or whatever. And it's just each one gets its own job and it's nicer on the peer and everything. [SPEAKER_01]: That is nice. Oh, yeah, I have Brakeman configured to fail if it's out of date. To cry if it's out of date, yeah.

01:35:22 pushcxWell, because otherwise, I'm going to forget it and leave it for two years. And then I'll be like, hey, wait a minute. Didn't we have something important there? Yeah, that's great. Thank you. [SPEAKER_01]: So off screen here, I'm just hitting the Merge button. nogweii @pushcx have you used any of the dependency update bots?
[SPEAKER_01]: So we'll pull that down. Good. And now if I reload this, all right, merged. Great. And what's your third one?

...50 byrootYeah, it's just a pet peeve of mine. It's just because I kept having like compilation issue with the MySQL 2 gem, which is not exactly abandoned, but not exactly... well maintained anymore. And like in the future, probably braids eight, we're going to make another adapter as a default for my SQL compatible databases. Is it going to be trilogy? Yeah.

01:36:14 pushcxI've heard about this. Is this, this bug from the read me.

...20 byrootYeah, and I'm actually removing it in my PR because it's not a problem. That was not the issue I had. It's just if you use 334, but that's the way in my Brickman PR, not my Brickman, but the CI PR, I also had a bump of MySQL 2. Yeah, I noticed that. Because if I go on master right now, And I'm on Ruby 3.3.4. And I try to bundle. It's just going to explode in my face. I try to compile MySQL 0.5.5. And it complains that MySQL close is not declared or something.

01:37:01 pushcxDoes Trilogy have anything that it needs installed as a package?

...09 byrootNo. That's the thing. That's why it's so good is that it doesn't depend only on MySQL. And so that solved so many issues.

...21 pushcxYeah. Is this? I thought I had even filed an issue about Trilogy once, because I remember hearing what you said, that it's going to become the default. [SPEAKER_01]: Yep.

...33 byrootOh, great. Oh, and I couldn't get rid of MySQL 2, because one of the gems you use for supporting views in the schema IRB doesn't work. Well, it was putting what was forcefully putting my sequel to, but I went to fix the project and they released it very quickly. So I was able to put it up here. I think I linked it in the description. [SPEAKER_01]: Oh, let's actually read your description.

01:38:01[SPEAKER_01]: Yeah. [SPEAKER_01]: Yeah. [SPEAKER_01]: And so you just sent this four days ago. [SPEAKER_01]: Wow. [SPEAKER_01]: Great.

...17[SPEAKER_01]: Yeah, there was a tiny issue. [SPEAKER_01]: I had to fix it on master. [SPEAKER_01]: Well, that's wonderful.

...25 pushcxI am going to merge this. And my only hesitation is I don't want to deploy it. Because it's such a fundamental thing, it raises the hair on my neck. And so I don't want to deploy it until I know I can sit at the computer for an hour, which I have to look at my personal calendar, make sure I'm not getting dinner or anything tonight.

...50 byrootIt's a small enough change.

...53 pushcxIt's fine. All right. So over on the personal browser. [SPEAKER_01]: Oh, I've by merging your. your other PR that touched the SQL. I've added a conflict, yeah. Can you get that real fast? Yeah, absolutely. Yeah. So Nagwe, I did see your question here. I just wanted to finish talking through the PR. Have I used any of the dependency update bots? nogweii @pushcx I've also added a comment to the trilogy MR
Yes, I've used Dependabot for, even before the GitHub acquisition, we used it for customers when I was a consultant. That or another very similar thing. And it's active on the Rails repo. That's what threw me the GitHub found one vulnerability, this message that you saw a minute ago. And I have it very strictly limited to just security issues right now. And I think it only opens these when I push rather than doing so proactively. I don't remember off the top of my head. I don't always check my GitHub inbox. Oh, so this is neat and I can show it. Yeah. Oh, I can't. Security setup. So this is neat. We are seeing by roots, new work, splitting out the job into four jobs. [SPEAKER_01]: That's very pleasant.

01:40:33 byrootYeah, because my pet peeve is just having like RuboCop complain about something and then you don't see the rest of the output. So I'd rather know everything up front.

...46 pushcxWell, you know, as long as we're taking a second, because I'm watching that dot and then I'll hit merge when it goes green. Do you know, you're going to say Docker, I bet, but do you know a way of making it so that Basically, do you know a way to run these in development, especially in parallel like this? Sorry, to add what? To run these checks in parallel in development. So locally, I run BRSpec. Right. I just have the equivalent of... but that's not especially useful. And it doesn't, I worry about it slipping out of sync, especially now that we have touched this. And I looked at this and I was like, well, yes, if I want to install Docker and effectively run all of this stuff.

01:41:46 byrootOh no, you don't need Docker for that. No, no, no, no, no, absolutely not. You can actually just use Rake. Not everybody knows, but Rake has a parallel run support. The problem with running in parallel is mostly just how do you match the output.

01:42:06 pushcxYeah, that would be a little chaotic. I guess also what I wanted was pulling these individual commands out of, which I guess I could write a little shim script that loads the YAML and pulls out the run command. nogweii have you heard of Just? https://just.systems/

...21 byrootRight. Yeah, I don't think it's really worth it. It's very little repetition, but I see where you're coming from.

...28 pushcxThat's fair. nogweii I love using it for command running
I just figured I'd ask because you definitely seem to know what you're talking about. Nogwe threw one comment on your Trilogy PR. So Nogwe had added a sample config database. So this changed in parallel with your

...47 byrootPR. Yeah, I thought I had it deleted, but apparently I didn't. OK. Yeah, I don't see it in the diff.

...55 pushcxThat would be great if you can do that real quick. Have I heard of Just Systems? No, I have not heard of Just Systems. Firefox, why are you showing me just the most garbage trash? [SPEAKER_01]: This is not. [SPEAKER_01]: So I made a separate Firefox profile for streaming. It's kind of still on. It's kind of obscene that Firefox includes ads.

01:43:24 byrootSo that's it. [SPEAKER_01]: It's updated. [SPEAKER_01]: Thank you, for catching this.

...39[SPEAKER_01]: Oh, it's an alternative to Make or Rake.

...46[SPEAKER_01]: Now that I look at this sample, yes, I have heard of this.

...49 pushcxI would almost certainly just write a rake file and on my blog, I have written a rake file because I don't want to learn another tool. nogweii that's very fair! LUL
Yeah. I presume there are, are benefits to this where it does something nicer than rake, but I try and be deliberate about dependencies and like a 5% improvement is not quite worth a, everybody has to learn this tool. [SPEAKER_01]: So I'm going to grab that and I'll leave that in the scratch. So I see that you just committed the. Yep. Improvement to the database YAML sample. And that's wonderful. [SPEAKER_01]: So as soon as that goes green again.

01:44:54 byrootBy the way, which variant of MySQL are you running? Is it MariaDB in production as well?

01:45:01 pushcxIt's MariaDB, yeah. Okay. Version 11 point something. There's a... Okay, but... Yeah, we have one database view replying comments. And MariaDB has a performance improvement where... let's see I'm trying to make sure I remember this correctly some of these filters will get correctly pushed down into a recursive join right and where is it I don't remember which joint it was it's been a couple of years now maybe three since we hit this but we tried to run MySQL in prod for a minute, and this view blew up, and everything went to hell because MariaDB had an optimization that MySQL didn't. [SPEAKER_01]: So I thought I said in the read me that it's got to be MariaDB.

01:46:08 byrootI'm running MySQL locally and the test read pass, but yeah, in prod, certainly not.

...13 pushcxYeah, the test suite will pass. I'm not sure you'll be able to... Did you start a local development server and click around?

...21 byrootNo, no, no.

...22 pushcxYeah, you would have to log in to see this. And then if you click around, you'll notice you get obscenely high page loads, like 30 seconds or two minutes.

...34 byrootMyADB also has support for insert returning. And in the case of comments... comment insertion in the 16 remaining queries, I believe there's two inserts followed by the select of what was just inserted. So I think I can squeeze two more queries, but the problem is to use it from Rails, you need to upgrade to 7.2.

01:47:01 pushcxWell, if you're bored, someone just filed an issue about that.

...05 byrootYeah. I'll see how much work it is, but no promises.

...13 pushcxI've tried to be aggressive about keeping up with the Rails minor releases just because so many things get easier and the maintenance burden is so much easier if we take it on a little at a time. When I first started maintaining the code base, we were a little behind and the first upgrade or two was painful.

...34 byrootYeah, for sure. I've heard so many old stories of people doing like... six years of upgrade at once.

...43 pushcxNo, I did one of those as a consultant once. And it was such a big upgrade that I spent the whole six-week project telling them, like, we need to make a plan for merging things. And the client did not pay a heck of a lot of attention on it. And then finally, the last two days, they were like, oh, we should talk about what it's going to take to merge and deploy this. And I was like, yes, we should have. three weeks ago. I don't know why you're only now hearing it. And it took them four or five months to finally deploy that PR. And it was kind of painful because it was just me racing their entire development team to get this updated. And so they just kept generating merge conflicts. And I imagine it only got worse after I left because then no one was chasing the merge conflicts. [SPEAKER_01]: Yeah, that's painful. [SPEAKER_00]: Yeah. Well, if you want to drop off, I mean, not only have we made some really big progress on that heap bug, which I can't thank you enough for, we got to merge all of your fun PR.

01:48:54 byrootSo if you want to roll out... Yeah, I'm going to go. It's getting late here. arh68 PopGhost yo this guy'sa pro btw
I'm going to update the issue tomorrow with what we learned tonight. And I guess, yeah, once you have Gemaloc running for a while and you have new measurements, please share them, see if it was it.

01:49:14 pushcxOh, that was kind of a joke I was telling, was you asked about measurements and we had some crosstalk, but my measurements are like, I log into production and I run DF-H with watch in front of it. We... are very unsophisticated about performance monitoring.

...34 byrootNo, that's okay. Just like we need, I guess we need to be able to tell if the problem is solved kind of thing. pushcx incredibly so
And the last thing we, but less important, but would be nice to figure out is those RebookUp stuff being loaded in your production processes, but I'll try to overlook. Yeah. when I have time. So yeah, thank you for the matches.

01:50:04 pushcxYeah, well, thank you for your contributions. dr3ig thank you @byby42
No, you're not allowed to thank me for helping out. I am thanking you for helping out.

...12 byrootNo worries. No, no, but when someone comes up and says, hey, I think there might be a leak in Ruby, or why should we We are really thankful when people give us what's needed to figure it out.

...26 pushcxYeah, well, it's been a lot of fun contributing and talking to... Oh, I'm so bad with names. I'm not going to try and list everyone's names from the issue. [SPEAKER_00]: Maxime and... Yes, Maxime and Aaron and... Just working on lobsters and then just working professionally in Rails, which is most of what I've done the last 20 years, I'm enormously thankful to you and everybody else who's contributed to Ruby on Rails because it's a huge chunk of my career right there. So having this bug in production, people occasionally were like, why don't you turn off YJIT and try and make this problem go away? And I was like, well, we're still gathering data. We're trying to make sure that we understand what's happening so we can give back because it If Lobster has a little bit of downtime, it doesn't really matter too much. Nobody's got an SLA where they're going to bang on my door and say, you had four minutes of downtime last month. You owe us $100,000. So it's very easy for us to buy that kind of thing.

01:51:25 byrootIt's really great to have an open source app because the reason Lobster ended up in the white sheet in the widget benchmark is like, it's not the, we wanted like a real Rails app because you had this like Rails bench thing for years and the debate is always like, okay, but it's synthetic. Yeah. And so taking a real Rails app in the Solva debate, but also COZ we considered where this course, which is much bigger beast to install. And yeah, I understand that has a lot of differences. Yeah, and also it's mostly an API, so it does lots of JSON, and after that it's like front-end JavaScript. And the other one was like Mastodon, but it's also like a bit more complicated than, noticeably more complicated than Lobster. So like when someone suggested Lobster, I just went through the code source and I say like, yeah, this is like a damn simple, like not simple, but... No, I'm not insulted.

01:52:26 pushcxNo, I try and keep things simple. And it was the original philosophy of the creator, JCS. He has a lot of experience in the open BSD world. And so this is my guessing, but I think that's where he came to the code style of liking things to be very straightforward and use the defaults as much as possible and have few dependencies. And so he wrote it in a wonderfully maintainable way where. He leaned on the Rails features for a lot of stuff. And where he didn't, it at least was clear to me as an experienced Rails guy, oh, he just doesn't know this API exists. And so he kind of re-implemented something. And so I've slowly replaced all those. And we've become more and more of a stock Rails app as time has gone by.

01:53:13 byrootFRANCESC CAMPOY FLORES- No, but that was perfect for us. So that's why I said thank you. Anyway, it's definitely cool.

...19 pushcxMARK MANDELMANN Well, I put it in our About page because I was so proud of getting to see it used that way. that's great anyways have a good night thank you very much yep have a good night just need to find his book yeah if you find the discord hang up button i'm gonna mute you now all right i'm gonna switch us over to the hidden scene so i can jump over to discord and make sure that it actually turned off so

...54[SPEAKER_01]: That was the first time I've had a guest on this. That was a lot of fun. And Drake, yes, you're right. He's just an incredible expert with Ruby and Rails. That was really pleasant for me to have someone very experienced to bounce stuff off of. So there's the pull requests we merged. [SPEAKER_01]: I'll put those in here, merging.

01:54:21[SPEAKER_01]: So with that, I had one more thing that was worth talking about streaming. We're not even at two hours. And this is quite short. So one of the things I said was for these streams, I wanted to be able to run queries for folks. Because in the same way the code base is big enough to be worth using, our data is big enough to be worth using. And I got, where are we here? We get email, main screen turn on. My God, there are probably people watching this who are younger than that meme. So somebody sent me this note and they said, they are gonna experiment with making a personal search engine for programming and technical content. And could they have a query with all the Lobster submissions that have [SPEAKER_01]: a score of two. [SPEAKER_01]: And I thought that sounded pretty good. So I wanted to kind of go through the process of that. So he gave me that draft query that I just pulled off screen there. And the thing I wanted to check was the definition of unavailable at. [SPEAKER_01]: Unavailable. Oh, I remember unavailable at is the indication that a story has been marked as no longer being served. So it's a thing that happens sometimes where we link to a story and then also a very popular site like Reddit or hacker news links to it and the site crashes. So we have a feature that moderators can toggle that say, Hey, the site is down and we serve a page out of the cache for a minute. And we can turn that back off once the site comes back up. So that's why he has the unavailable at his null. [SPEAKER_01]: I'm assuming he, I don't even know anything about this person's username.

01:56:36[SPEAKER_01]: All right. [SPEAKER_01]: So I'm going to jump up to a prod console and one thing I don't remember off the top of my head is how to run the query and dump it to a file. Does anybody remember for MariaDB how to do that off the top of your head?

01:57:04[SPEAKER_01]: Because it has a, oh, it's select into out file. binaryxvoid i think mariadb-dump right?
[SPEAKER_01]: There we go.

...16[SPEAKER_01]: Oh, but then it's that, MariaDB dump. MariaDB dump is for dumping the whole database as if I was going to back things up. nogweii @binaryxvoid not a whole database, just a single query
binaryxvoid Oh I see myb
And the hassle I always have with select into out file is that it does its own odd tab separated format, which doesn't play super well with if you're dumping strings. [SPEAKER_01]: But I guess that's acceptable. [SPEAKER_01]: So if I say...

01:58:06So I'm looking at the MariaDB man page for select into file, and it's the most SQL thing where it just goes right in the middle of the query. So we'll grab his query where isModerated is false. [SPEAKER_01]: And I'm wondering, does he have...

...34[SPEAKER_01]: So he doesn't... [SPEAKER_01]: I keep saying he. This query doesn't do quite what they want. because stories are marked as moderated if a mod has had to edit the title or tags. byby42 One last for the road https://github.com/lobsters/lob…
The is moderated flags mostly means that the moderator, I think they thought that is moderated means the moderator has removed it as opposed to is moderated just means a moderator has edited it. And we have this code so that if someone submits a story with the name of a blog on the end of it... Oh, ByeBye is jumping in. [SPEAKER_01]: One more PR for the road. [SPEAKER_01]: What is this? [SPEAKER_01]: Wrong clipboard.

01:59:37[SPEAKER_01]: Stop loading RuboCop in prediction workers. [SPEAKER_01]: Oh, John, this is wonderful. [SPEAKER_01]: prohibit form. [SPEAKER_01]: And is that what it is? [SPEAKER_01]: Is that what's been doing that? byby42 I don't have strong proof, but yeah, looks like it is
[SPEAKER_01]: Oh, it's my own little robo cop. [SPEAKER_01]: How funny. So robo cop is the linter. And I wrote one nogweii @byby42 nice!
to tell me to use form with because not knowing the different versions or Rails for like a couple of years there, Rails really changed the interface of the various form helpers. And I couldn't remember which one was the current one. So I wrote, yeah, I wrote this little worker to say, or this little RuboCop to try and nudge me to use the current one. And it just had a required RuboCop. Oh, man, I would never have found that. byby42 rg -F rubocop
How did you find that, Sean? [SPEAKER_01]: I don't know if they're still here. [SPEAKER_01]: RGDEV, RuboCop. [SPEAKER_01]: Well, that would certainly work. [SPEAKER_01]: Okay. [SPEAKER_01]: Well, you know, all checks have passed, so I am just going to pull that in.

02:01:19All right, and because this doesn't automatically load, I'm just immediately merging that in because I can see that that's going to be a safe bit of code. So if the test suite is green, I'm a happy camper. That's great. I will have to make some time in the next couple of days, maybe tomorrow to let's put that in my scratch because otherwise I forget.

...56[SPEAKER_01]: I will have to dump heaps again to look at them again. [SPEAKER_01]: All right. [SPEAKER_01]: So jumping back to this query. [SPEAKER_01]: The is moderated flag. What I want to do is update their query to say that we don't want it to be is deleted is true. [SPEAKER_01]: And is moderated equals true. So I am changing the semantics of their query. And I feel pretty safe in this one because this is one of those where you really have to know the story model to know what you're asking for. And so I don't think they asked for the right thing. [SPEAKER_01]: And let's just grab five here. [SPEAKER_01]: Oh, that's inverted. nogweii dang too slow typing
[SPEAKER_01]: We want to say is deleted is false. That was generating the exact opposite of give me five things moderators have removed. That's why I was like, wait a minute. Dang too slow. Yeah, we got to make a... I mean, you already are a VIP and more so for having submitted PRs to the repo, but... Maybe it has to be something like there has been an exception or I have expressed that there is a bug. On the other hand, it's super useful that people have caught them even before I've seen them. So what we want to say is let's filter out the ones that mods have deleted. [SPEAKER_01]: Actually, I think it's fine to just filter out the ones that users have deleted too.

02:03:55[SPEAKER_01]: Really, and the second one is there's an Occupy Wall Street that we didn't remove.

02:04:08[SPEAKER_01]: I'm curious about that one. [SPEAKER_01]: So its short ID is this. I wonder how this is. Oh, 2012. So moderation is definitely not a super exact science. Yeah, so it didn't get removed. [SPEAKER_01]: And that's fine. [SPEAKER_01]: So we'll grab this, and we will say grab all of these things into outfile slash tmp slash. [SPEAKER_01]: And then let's try.

...59[SPEAKER_01]: So this is a MariaDB setting or it's some kind of security setting. So you can't just use the MariaDB client. Yeah. I must not have set the file privilege to run this statement and MariaDB needs. It's odd that MariaDB needs as opposed to the user. And then there may be this secure file proof. So there's a lot of places that this could be getting prohibited and almost none of them do I want to check on stream because it's super fiddly. So let's think about what's the other way to run a MariaDB query and dump the results to file. nogweii `user` is the database user, and the 3rd sentence is about the `user` as a UNIX account
[SPEAKER_01]: I think I can just say here, take the query, [SPEAKER_01]: let's just try running it with the limit five and we got a line break across the word all right so that gave me some output so i am going to remove the limit 5 and just dump this to this is the

02:06:25[SPEAKER_01]: This is the I grabbed the wrong query. [SPEAKER_01]: That was the intermediate one.

...37Yeah, so this this boxes and lines format is kind of painful, but at least the. At least it's easy enough to pull into Excel or any other parser or just split on lines. So I will off stream. nogweii how big is that file?
I am going to just email them that query output. But that's the gist of running queries on prod is I take them, I look at them and make sure they're doing something roughly reasonable. How big is that file? Got to be. So I am going to, before I hit enter, I'm going to guess somewhere around nogweii also curious about disk space
somewhere around 95,000 lines, because I want to say we have 107 or 108,000 stories that just came up in the last stream or two. Yeah, 90,000. [SPEAKER_01]: And so like mentally, there's a bunch of stories that don't have URLs. [SPEAKER_01]: Curious about the disk space? [SPEAKER_01]: Sure.

02:07:49[SPEAKER_01]: Looks like five and a half megs. [SPEAKER_01]: Yeah, and it's just the URLs. [SPEAKER_01]: So it's not huge. [SPEAKER_01]: This is the kind of data I'm really happy to share. nogweii 5 megabytes of URLs is a pretty good seed for a search engine!
I hope they have fun making their own search engine. [SPEAKER_01]: So let's go ahead and put that on my [SPEAKER_01]: Five megs of URLs, pretty good seat for searching. [SPEAKER_01]: Yeah, yeah, hopefully. So I'm gonna also say, send user their query results. [SPEAKER_01]: That can be on my to-do list. All right. gtfrvz select distinct url
So we're just a little over two hours and we can go ahead and peek ahead. I was thinking of this for Thursday, but as long as we got through everything very briskly nogweii ooh, good idea
[SPEAKER_01]: Select distinct URL. [SPEAKER_01]: Oh, that's fair. [SPEAKER_01]: You know, that would be an improvement for them.

02:09:03[SPEAKER_01]: Because since they don't have any other metadata associated, like they weren't curious when things were submitted. [SPEAKER_01]: I'm going to grab this query. nogweii does that drastically change the amount?
[SPEAKER_01]: say I just put the, the query in the scratch notes. Does that drastically change the amount yet? No, honestly, the reason I immediately wanted to run it as I'm curious how much the changes. [SPEAKER_01]: So there are now almost a thousand fewer URLs.

...48[SPEAKER_01]: File size is not much change, 5.5 to 5.47. nogweii 1k fewer, huh. I thought there'd be more dupes
[SPEAKER_01]: Yeah, I'm not gonna run the, well, no, that's fine. gtfrvz a thousand reposts
[SPEAKER_01]: We don't get a ton of dupes and where was it?

02:10:22[SPEAKER_01]: Bigger browser. [SPEAKER_01]: A thousand reposts. There was also a change I made recently that's kind of related. And I'm trying to think of a word I would have used in the commit message or the comment about it. nogweii ah, hm, I'm thinking more about merged stories, which doesn't reflect as duplicate urls in the database
[SPEAKER_01]: I think I used the word discussion. [SPEAKER_01]: Yeah, here it is. So this is a fairly recent change. I did it March 2nd. But when someone is resubmitting a story, the meat of this change is to check if it's a resubmit. And if it is, it requires that they write a comment to start. [SPEAKER_01]: There's a few uses for this. [SPEAKER_01]: When people are new, David Price- They can't really submit from unseen domains, because unfortunately that's often spam and one of the easiest things someone who was a little bit malicious could do is just go pull a good story that was submitted a year or two ago. David Price- and repost it and that's not super helpful that doesn't add anything to us, and sometimes people resubmit stories, just because. They saw it and they kind of think it was neat but there's not a new discussion, the second time, so I saw a bunch of reposts where. Maybe we had a discussion, the first time around, but the second time we definitely didn't and it got zero comments or maybe it got one comment. And that just wasn't really a good use of the site's attention, and so I said hey if you're going to resubmit a story, you have to write a comment about what you think is interesting and. It's funny, this is like one sentence and it replaces two sentences, but I workshopped this wording forever. I really sweated it because I wanted to say, hey, you have to write something, like try and generate content. The other place this is valuable is with the outright malicious users where they're here to try and get traffic to their site. And they only submit links because they want to cover up the fact that they are just here to drive traffic to their site. For them, the most expensive thing would be if they have to actually participate in the comments like a normal user, because that's writing one-off responses to people. And so it's imposing that cost on the people who just want to resubmit links to try to cover for content marketing, which happens occasionally, where there are some users who, even though they understand the guideline of, hey, if you are interacting with everybody and you are a normal part of the site, we're great. We love sending you traffic. They just want to do the cheapest possible thing. They don't want to actually join the community and participate. So this was a speed bump for them, especially. So reposts were not hugely popular, as you can guess by the fact that they're what? What was that, 2%? No, not even, 0.4%? Because if it was about 90K and it dropped down about one, yeah, that's 0.8%, something in that neighborhood. nogweii speedbumps and pebbles are my favorite techniques in combating malicious users. since it's so damn effective
arh68 can you comment on like year-old stories ?
But you get enough of these nudges, and when somebody gets eight nudges, either they go away or they do the right thing. Or the other one is they are really conspicuously trying to break the rules. Can you comment on like your old stories? I want to say the limit is lower than that. We have the story model up. Yeah, the comment of old days is 90. [SPEAKER_01]: You can leave comments for about 90 days.

02:14:32nogweii so, no, you can't necrobump a story. hah
arh68 interesting. would fresh comments bring a story back to front page? or is that only scoredriven
That was also a driver for one of the other features that the slash active that I added to the top navigation here of this is a list of stories sorted by when they last got their most recent comment. Would fresh comments bring it back to the front page? No. arh68 ya I like active for that
Well, so probably not is the answer. And you ask if fresh comments would bring a story back to the front page, or is it only score-driven? So active is very simple. It's just, when was the last comment? [SPEAKER_01]: And... No, I didn't want... Where's the hotness function? But a comment could bring a story back to the homepage because... arh68 ah interesting
collemwest Oh neat, I didn't know lobse.rs was written in Rails. I learned Rails a few months ago for my job. Can get some practice solving the issues here
stories story comment scores give the story a boost to its hotness which is what determines the ranking on the home page so especially if something was right at the edge between the front page and the second page a good comment or two could bring that up it does ignore the submitter's own comments but otherwise it's pretty open and of anything i would be I have been thinking a little and this is very recently and very shallowly I have been thinking in like the last couple of days that because we touched this on screen. Maybe we want to increase this because a huge amount of the value of the site, the interesting about thing about the site is the comments much more than. Is this a neat link? Did people write really interesting things? So I started this stream two hours ago, highlighting this wrap your dependencies, because I thought it was a really interesting story. And I think I wrote the first comment on it just a couple hours ago. And when I first showed it on stream, there were, what, three, four comments? And now there are 19? This is the best part of the site. 19 times people have had something interesting to say based on that. Maybe we want to weight the comment score even more than the individual story upvotes. Because that's the big beating heart of the site. That's the good part. That's what I want to reward more. So I'm thinking a bunch about that. The downside there is... arh68 maybe like distinct commenters
when people have heated arguments, the people who agree with someone tend to upvote many of their comments. And so it's possible that that would promote more like flame wars to the homepage. Yeah. Maybe distinct commenters or like maybe there's a bunch of options there. nogweii maybe only top-level comments get higher weight?
[SPEAKER_01]: So here let's, let's go under here. nogweii to avoid flamewar threads
[SPEAKER_01]: And, pushcx https://github.com/lobsters/lob…
nogweii "everyone agrees" LUL
welcome is it column column west yes lobsters is written in rails and if you want to get practice solving rails issues everyone in the world agrees the best thing to do is to go to github and go to our issues i'm going to be shameless about this one and click on good first issue to limit down to which ones we say are great first issues please, please help fix bugs. We have plenty. mjiig Something h-index like? The largest n such that n distinct users have gotten n upvoted comments
We have 64 where we would love to have your contributions. Oh yeah. Everyone agrees. Yeah. I mean, we just had someone from, John is, I think he's on the Ruby core team. He might also be on rails core. I don't know. He knows rails very well. And he was fixing PR, so if you want to be cool and you want to be on roofie and rails core obviously a good place to start is by contributing to lobsters.

02:18:53[SPEAKER_01]: Maybe only top level comments yeah.

02:19:08So MJIG says, something hindex-like. Could you drop a link or explain what an hindex is in this comment? The largest N such that N distinct users have gotten N upvoted comments. [SPEAKER_01]: I'm not sure I can parse that one.

...41Top level comments is a really interesting one. Or another one could be count comment number votes on top comments by each author. So this is a little different than the number of distinct commenters. It's for each user, take whatever their top score comment is and count that directly in. And we can note that current [SPEAKER_01]: Calculation counts half of comment votes. [SPEAKER_01]: So there's a whole lot of, oh, and then mjig. [SPEAKER_01]: Why don't I just copy this whole text, because I don't totally parse it. mjiig h-indexes are a citation counting metric, a researchers h-index is n if they've had n papers each receive n citations
[SPEAKER_01]: So I want to think about that one later. H-index are a citation counting metric. mjiig On thinking about it for more than 30 seconds I don't actually think it works great for what you want here
Researchers' H-index is N if they've had N papers each receive N citations. Ah, so if Alice, the researcher, has written five papers, she might have an H-index of three if three papers have gotten three different citations. It's a little complicated, but I see where you're going there. Yeah, just something to kind of balance out and not totally overwhelmed. And this is all very, very, very speculative for anybody who wasn't listening five minutes ago when I said that this is a fairly new and shallow idea. A big chunk of what I do as a SOP is I look for these kinds of patterns and then I marinate on them for weeks and months and then I make very deliberate changes. I don't want to rush into things. It's very easy to have bad stuff. I mean, on that first stream we changed Story scoring to use fighting differently and I had been thinking about that one for months and. I posted it as a comment on the site and nobody had any concerns and it got I don't know 35 something up votes so that's a whole bunch of people who thought it seemed like a reasonable idea. And then it didn't work out in practice at all and to basically be reverted so. dr3ig so you want to encourage stories with high number of comments, while discouraging thread depth ?
These kinds of things I want to be very careful and deliberate on so it is fun to to noodle on them, but I don't want anybody to look at this and be like oh my gosh. story scores are going to change again don't want to encourage stories with a high number of comments will discouraging thread depth not really so. People often look at thread depth. Let me put it different. when discussing things like arguments and flame wars people have been really critical about thread depth and that is roughly correct that arguments or fights or just really unpleasant threads tend to be fairly deep where especially where like user a and user b are getting into it and like replying back and forth to each other and that feels like a really strong signal if you're looking at one of those and However, the arguments are such a low percentage of threads that they're very memorable because they get heated, but some of the very best comments have the exact same shape where they're like nine replies deep, but it's like Carol is saying, this thing should work like that or it's similar to this other problem and then dave is saying i'm sorry i don't understand can you tell me more and then carol says sure here's three more links and dave says oh that's neat does it relate to this kind of other library and you know if you if you cross your eyes which is you know what you're doing when you just look at thread depth and are there just two people going back and forth You see the exact same thing as you do in a really unpleasant thread and it's actually one of the very best threads and that is. dr3ig in that case boosting top-level comments is kinda equivalent to penalizing thread depth, no ?
I don't know 10 or 20 times more common than people getting into nasty back and forth so i've been really reluctant to say anything negative about discouraging thread depth like it's an easy thing to reach for but it's also the some of the very best comments yeah.

02:24:23So I'm going to write that down. So I want to stick this one in the bread summary. I mentioned it in the Lobster's chat room maybe two or three weeks ago, but it's maybe something I want to write up at some point because it's one of those ideas that keeps coming up.

...57So I have totally lost my train of thought because we rabbit-holed into talking about comment and story scores and then talking about how all the cool kids submit pull requests to lobsters to fix bugs.

02:25:18[SPEAKER_01]: What was I going to talk about? nogweii heh, I just hit the anti-dupe screen. effective, as I didn't have a comment other "this tool is neat!" :D
Oh, I was going to talk about boosting top-level comments is kind of equivalent to penalizing thread depth. yeah a little bit it's not actually a penalty it's just rewarding a different thing you hit the anti-dupe screen so oh yeah yeah sometimes when you have a like a me too kind of thing i've been kind of deliberately watching the URLs that get reposted and saying like, am I getting substantive comments out of people? Or are they just like, this was neat four years ago and I still think it's neat. And I'm trying to, to nudge people towards leaving substantive comments. [SPEAKER_01]: So that's fine. So one thing I had wanted to talk about and get into developing was pagination. in yeah so this was august of 2017 this first got reported so it's it's a well-aged bug it is very mature when this this url is slightly changed but when you were looking at a user's recent comments it's not paginated you get to see their last I think it's 10 or 20 threads that they've been active in. But there are lots of places that we do pagination and it's just oldest to newest. For example, slackjet slash active, which is a recent page, just has a page two. And as long as the pagination is counting up, this page is especially i mean one is unstable but two is also unstable and so like it resorts so it's very possible here that actually let's look so the top three here are fixing a bug irc js dates did any of them show up on the first page here fixing bug no so if there had been a bunch of comments between when i loaded page one of slash active and page two of slash active, things can get bumped down or skip over. And so that's not super useful. Where it gets real frustrating is, we talked a little about this on a stream, I think maybe two streams ago, is you can paginate very deeply into the site and people write very naive crawlers that very aggressively do. If you look at this, it even takes slightly longer to load these because it actually, the database has to load and sort all of the comments to be able to seek 400 pages down. This is called limit offset pagination because that's what the SQL query it's doing under the covers is called. pushcx https://github.com/lobsters/lob…
[SPEAKER_01]: and we have talked a little about it, that there is a performance issue. [SPEAKER_01]: I'll let people, I'll throw this link here in the chat and in the scratch file because that matters.

02:29:17[SPEAKER_01]: So this is kind of a fairly big [SPEAKER_01]: feature to implement. I took a shot at it when I first did some twitch streaming in the middle of the pandemic in what was that June or July of 2020. It just kind of feels satisfying to come back and actually solve this one. The Reason it didn't get done last time is there are a couple of absolutely enormous pagination plugins for Rails where, you know, if there was a plugin that was 100 lines of code and did roughly what we wanted, I would be pretty happy with it. But some of them are several thousand lines of code, and that's a really big dependency to take. [SPEAKER_01]: What we want to do is Reddit-style pagination. So if we look at our programming, just to pick one that's kind of similar to in content to lobsters, if you look at the next link, it has this cursor. And so it's saying after this story, this is the last one you saw on the previous page. That's its short ID. I'm not sure if they actually use this parameter. [SPEAKER_01]: I didn't think you can actually edit it. [SPEAKER_01]: Maybe I can. oh it's it's for their numbering so you see i got the same stories just with a different number down the side but i got the same amount and i thought this was going to be amount and it wasn't going to change anything but it's just this so don't mean to click so basically stealing that function functionality for pagination is what I would like to do, because there are many places where we paginate stories or users' comments, and I don't want to go wild with metaprogramming, but it would be very nice. And I may even have a dead branch around it, but the basic gist of this one, the outline of the work, which at least [SPEAKER_01]: can talk a little here, even though I don't think I can knock it out in like 10 minutes, is take one place on this site that paginates, like home page, update pagination to use a cursor instead of page number slash, which is limit offset paging.

02:32:28There are basically two kinds of objects. Well, there's really three. So that initial request was around, if we look at just a random user who's at top here, if we look at their, this pulls up their threads as opposed to their individual comments. And so we do want to paginate threads as well as comments. [SPEAKER_01]: So stories, comments, threads.

02:33:07[SPEAKER_01]: And that's kind of a rough outline of that one. It's a little bit of a puzzle because it's touching the controller and touching the model and touching the view. So it'll be all up and down the Rails stack on that one. And I think rather than dive into it and start coding when I'm already two and a half hours in, I'm going to, having sketched that out, say that this is going to be the rough plan for the next stream. dr3ig XD
Unless someone who is very cool and wants to gain three inches of height submits some pull requests and bug fixes for improving the site. Because that's obviously where I would start any stream is with very cool people like that. Yeah. [SPEAKER_01]: I mean, streams have to start with cool people like Column West and Nogwe, right? [SPEAKER_01]: Yeah, that's good. hejihyuuga i mean every stream starts with a cool person, pushcx B)
[SPEAKER_01]: All right. So with that kind of outlined and setting expectations for the next one, every stream starts with a cool person? No, no, I just sneak in. I'm not cool. [SPEAKER_01]: I will never be cool.

02:34:26hejihyuuga for the record i think you are cool
[SPEAKER_01]: On the other hand, it is not my job to be cool. [SPEAKER_01]: It doesn't benefit any. I actually... Yeah, maybe there's a good place to end. I am actually mildly in disagreement with the idea of being cool. I think a lot of the times where we've gotten is cool contrasts with things like being eager or enthusiastic or joyful. And I like those things. I like them a lot more than appearing cool. Like it is not cool to say, oh, I met a really impressive developer and he helped me with this heap dump and it's such an interesting problem. A cool person would be like, I don't care. Oh, I'm jaded. I've seen it all. Maybe I'm setting up a false dichotomy, but I'm not particularly interested in being cool. hejihyuuga I guess I think of being cool more as being someone worthy of respect/admimration
I'm more interested in being happy and effective and creative. hejihyuuga i might be bastardizing the word
There are sometimes even really negative versions of cool where it turns into kind of a crab bucket mentality where anyone who is earnest enough to start a project, well, that's not cool because no project starts out perfect and no project starts out polished. Therefore, it can't be cool. And I don't want to see that kind of anti-creative crab bucket. I'm really super opposed to that kind of thing. Lobsters is in a lot of ways a celebration of creativity and creativity requires mistakes. And oh yeah, just to remind the thread motto or the stream motto, if something is worth doing, it's worth doing badly. And I mean that very emphatically. As you can probably guess from running a website that has, what did we just see, 150 open issues? I try and get the pull requests quickly, but it's OK that we have bugs in production. The main thing, the commenting, the discussion, the generally happy, pleasant tone, the healthy community, that's the value. Perfectly cool, slick, well-implemented, well-factored code doesn't contribute as much to that as one might think. All right, so thank you all for hanging out with me for the last couple hours, or even just the last couple minutes if you're dropping in. hejihyuuga thank you for stream pushcx o7
I would say keep on not being cool. And I will see you all on Thursday morning. Thanks for dropping by, Hedgie. dr3ig bye @pushcx
And everybody else who has commented a bunch, like Drake and... nogweii see ya PrideWave
Nagwe and Ar... I don't know how to pronounce your name. ARH68. arh68 HahaCat see y'all later
I really appreciate having you regulars. Thank you. And for everybody else who's dropped in. This has been... Oh, oh, yeah. There's one more thing. That's going to happen on Thursday. On Thursday, either on the stream or after the stream. I don't remember if the Twitch dashboard is up to date. But at some point... Either Thursday morning or Thursday afternoon, I think I'm going to hit this stream on seven different days thing for the affiliate achievement. Still not super interested in being an affiliate because it comes with ads. It's just kind of a neat milestone. I just remember it. Thanks. [SPEAKER_01]: Anyways, take care, folks. [SPEAKER_01]: Have fun. hejihyuuga bye!
[SPEAKER_01]: Submit pull requests. [SPEAKER_01]: I'll talk to you later. [SPEAKER_01]: Bye.