I like you but not enough to leave you unsupervised on my machine

ChaelCodes Hello!

07:00 joeldrapperhumorously for once like linux audio worked better than mac audio so we got to see you know like an eclipse happen basically and then oh hey chael we are going to pair on experimenting with flex which is A library that Joel has created. And did you create it yourself? Or did Steven help? I don't know where your collaborations start and end. [SPEAKER_02]: Yeah. [SPEAKER_02]: Flex, I started by myself. [SPEAKER_02]: Okay. [SPEAKER_02]: But it's had quite a lot of different contributors now. [SPEAKER_02]: And I'm also have like, do you know Will Cosgrove? I know the name, but I don't think I've spoken to him. [SPEAKER_02]: So he's like an owner on RubyGems and on GitHub and does a lot of stuff on it now. [SPEAKER_02]: But honestly, there's not much maintenance required in Flex. Except when I accidentally convince you to add caching? fractaledm1nd Joel, your mic audio is relatively low
[SPEAKER_02]: Yeah, yeah. [SPEAKER_02]: I'm pretty close to doing that now. [SPEAKER_02]: I have a draft PR open. fractaledm1nd Sounds better
solnic_dev hi, yes Joel was super quiet
let me oh hey steven speak of the devil can you tell me if joel sounds better now because i just mixed him up a little hello do i sound better yeah you should be you're peaking in the right place now yeah your your previous mic setup you were much louder than me so i had you turned down a little so now that you adjusted we should be good cool awesome so Ah, well, thanks Solnik. Oh, nice to see you. pushcx https://beta.phlex.fun/
I don't think you've come by the stream before, but so flex is this, we will share this and it is a, it has all the, I love this little marketing tiles because like my very, the first time I saw this was I was like, oh, okay. So it's a alternative rendering library. Well, why would I want it? And then like the first one was cause it's Ruby and I was like, okay, but I've tried other engines and then it was like, oh, well it's fast enough. And I was like, well, that was the problem I had with those other engines. And then, oh yeah. Okay. It actually works with rails. which I think, I think you could add to this sentence is in addition to. [SPEAKER_02]: erb or something to make it clear that flex is not a you have to take over the world kind of thing which i think is very yeah smart design so it does say plays nicely with view component and action view but i wonder if people even think about their rails views as being action view or just as being like yeah i don't think they do i don't think they do yeah like i know it because i've dug into the gem and i assume any kind of rails contributors do so there was i wanted to find i found the thread on lobsters the other day ph oh yeah where i learned about flex a couple months ago where so i submitted this blog post i should throw this in the pushcx https://lobste.rs/s/ily9nh/you_…
ChaelCodes I don't think of it as ActionView....
chat in case anyone's curious so i submitted this blog post i saw where someone found a small optimization by if you if you put your logging messages where's the tldr if you put your logging messages in blocks they don't get evaluated unless they're actually run which is nice for performance And then there was a discussion down here where I had seen Joel's Quickdraw library, and I don't remember what linked me to that, but I was just like, oh, hey, I recognize that same technique of making your failure message a block, because at least with me, usually my tests pass, right? As opposed to failing all the time, that would never happen. [SPEAKER_02]: And so... Yeah, I think we may have talked about this at Madison. Yeah, that could be. Yeah, so Joel... And some coworkers gave me a ride up to Madison Ruby from Chicago. That one, oh, so disorganized. So Madison is only three hours from Chicago by car. And so I had talked to my spouse about, hey, do we want to go up and stay an extra day or two and have a little getaway weekend? And that never got planned or it never got totally ruled in or ruled out. And so we kind of waffled about it. and you know i had booked my my hotel because that part is easy that's going to be the same either way but then i didn't arrange transport because i didn't know if i was doing it for one or two and then i got this email from the hotel that was like do you want to check in early download our app and you can check in early and i was like that's ridiculous who checks in early like a week before their hotel stay haha what a silly email and then i was like like an hour or two later, it kind of percolated back up in my consciousness where I was like, it's that exact same instinct that's like, I'm being watched. There's a tiger behind the tree kind of thing. Because I was like, wait, what hotel would let people check in a week beforehand? That makes no sense. And so I like pulled up the tickets and I was like, oh, the conference is tomorrow, tomorrow. And so I hustled around to find a ride. And what I ended up doing was joining a bunch of Joel and his coworkers rented a van to drive up from Chicago. Cause so many of you are coming in and we talked about a bunch of Ruby stuff, obviously the whole way. So quick draw used that same block strategy. ChaelCodes LUL
And then this commenter mentioned the flex library and. So I'm not logged in and actually I already logged in, but when I came back to this thread yesterday, everything from here down had the little unread label on it. Cause I hadn't seen it, but I looked up flex and I was like, oh, this is neat. I like this. And I've played with it on a personal project. And what I didn't realize was by root who has been on the stream previously helping with the memory fragmentation bug, has some opinions about proper benchmarking and hash keys and cues. and in general performance as you can imagine for every conversation about performance it pretty much immediately goes into the weeds and a big chunk of that which we are not going to relitigate is how appropriate micro benchmarks are so since we were going to experiment with converting some of lobster's partials to flex, I would especially like to throw some kind of benchmarking code around it to see would it be faster for us. [SPEAKER_02]: Because it's actually realistic. Yeah. [SPEAKER_02]: And it's... [SPEAKER_02]: The micro benchmarks are not very helpful. Yeah. And there's a lot of questions around how closely do they actually resemble real world use? And I actually only care about my real world use. So that's sort of convenient. And speaking of performance, there was this other thing that's kicking around. pushcx https://github.com/Shopify/yjit…
So some folks know that the widget folks picked up lobsters and put it in a can to use as a performance benchmark of widget. And there is a big thread running because by route is updating since. So they took a snapshot of the code and they're going to come back and update to the current one. And he. right out of the gate, he was like, yeah, lobsters is like three times as slow in the benchmark. And he's digging into it now. There's like a very long and detailed conversation. I only follow some of it because I have not implemented Ruby itself. I just use it. We'll see how that goes. It's an interesting puzzle. So anyways, I guess the last piece of intro before we jump into it is Usually everybody gets to enjoy my Vim interface, but today we are going to use Zed because it has a nice file sharing thing and it handles the audio call. So we'll see how this goes. And I had suggested that we start with this subnav partial. So to show folks what this is going to look like, Flexes for making little components out of templates. And we have out, there's this sub nav here and it has a little bit of complexity because I got to fill the cache, man. Let's not click on the stats page because there isn't caching in local dev. And so those giant aggregation queries are going to run and take forever. I would click on the hats page instead. And this pattern of having a sub nav is repeated across the site. So I figured we could talk Joel into like nibbling this little one first, and then maybe we could make it a general thing. And I know you're itching Joel to like do the giant performance one of comment, but that one has so many moving parts. I need to see a simpler example first. So I will click follow on you and then here's the about sub nav. And then I think we're just following you. So I'm going to, how does this work with closing tabs? This doesn't mess up your view of the tabs, right? [SPEAKER_02]: I don't know. [SPEAKER_02]: I don't think so. [SPEAKER_02]: I think I can open my own tabs. Yeah, and we played with this. This is not totally YOLO, folks. We are... Ooh, it tattles on you. Not paying attention. We did play with this yesterday, and it follows along nicely. So if you see this nice pink border, I don't know why it's pink, but we are following Joel. [SPEAKER_02]: Yeah, I think the first thing... [SPEAKER_02]: If we want to start running this, the first thing we need to do is install Flex, unless you've done that already. No. Actually, how do we ask Zed for Git status? Let's make sure I'm not on a weird branch or anything. Yeah. No, I'm current on master, and I'm going to do a quick poll to make sure I don't think I merged anything. No. All right. So we are correctly. So if GitHub is up, anyone can see exactly where the Lobster's code base is now, because this is master. Mm-hmm. [SPEAKER_02]: I think you're going to have to run all of the commands. [SPEAKER_02]: Yeah, that's fine. [SPEAKER_02]: I've got a terminal here. The way Zed is basically file sharing this directory, it kind of raises the hair on my neck a little. If it gave you full terminal access, I would not have accepted it because this is still my personal machine. Let's stick this in the Rails section. We don't really have a front-end section. Yeah, I don't know. Because we put other stuff there. Frici your...call is going through zed? Interesting 🤔
Frici Also Greetings.
[SPEAKER_02]: So I think, just to make sure we're on the absolute latest, for the sake of this video, we should just pull straight from GitHub if we can. pushcx we are prefershanals
Sure, yeah. Because I know... So Flex is... The current version is 2.0.0 RC1, and so we want to use whatever that latest thing is that you just put in. [SPEAKER_02]: I haven't shipped the caching yet, but it also probably would work. [SPEAKER_02]: So because we're using... pushcx also welcome :D
[SPEAKER_02]: The version from GitHub, we do need to specify both flex and flex rails. [SPEAKER_02]: Sure. [SPEAKER_02]: Typically you would only do flex rails and it, it depends on flex. [SPEAKER_02]: I think if you bundle and install this, we should then be able to run the generators.

20:15okay. So I'm going to yoink up this and I will run bundle, templates now. Oh, this is my get set up. That's fine. okay. We bundled. Oh, I should bounce my Rails server real quick because we just bundled. So let's call the Rails server tab again. All right. So Rails will be up in two seconds with Flex installed. [SPEAKER_02]: Nice. [SPEAKER_02]: We should also run the installer. [SPEAKER_02]: Can you run, I'll just put it in as a comment. [SPEAKER_02]: Sure. Tabs are in order. [SPEAKER_02]: so we are running rails g flex install yeah this should fantastic so this is basically just giving us an initializer which sets up i should i should just jump into it so yeah let's take a look at the initializer first that's kind of a an outside in thing [SPEAKER_02]: Yeah, so this is basically the way that in Rails you can load one of your folders. [SPEAKER_02]: So here we want railsroot app views and railsroot app components to be loaded as a namespace. [SPEAKER_02]: where normally they're not loaded as a namespace. [SPEAKER_02]: They would be loaded as root. [SPEAKER_02]: So if you think of like app slash controllers, it doesn't give you a controllers namespace. [SPEAKER_02]: You put controllers on the end. Yeah, Rails has a very flat hierarchy like that. And then every team that gets to 10 or 20 engineers goes, oh God, we're dying. We have to introduce modules. [SPEAKER_02]: We have to change it. Yeah, yeah. Speaking of which, I thought of... My pet peeve is pluralization. I thought of one more reason to not pluralize these things. Because the trade off is having to mention the global namespace for models like you showed in props in literal on the last stream. And while I will agree that that is ugly, Every large size, every like non-trivial team has to do that anyways, to implement pack work or whatever their homegrown version of it is. But then also I know you have long-term plans and if you don't go singular now, you will live with plural forever. Yeah. Anyways, I won't harp on it. [SPEAKER_02]: We'll think about it. [SPEAKER_02]: I think for now, just cause the, the way the generators and stuff is set up, we should at least stick with this. [SPEAKER_02]: oh yeah. God, no, don't try and change it on the call. I'm just, yeah. I have a hard time letting the porous thing go. I've been annoyed at it for 20 years. [SPEAKER_02]: Yeah. [SPEAKER_02]: So because it's all just Ruby, like you could not use this setup at all. [SPEAKER_02]: You could have your own namespace. [SPEAKER_02]: You could put components wherever you want them. [SPEAKER_02]: Yeah. [SPEAKER_02]: the advantage of this is we have this components module and it extends flex kit. Do you want to take two sentences to tell us what a kit is, or should we wait till later? [SPEAKER_02]: Probably best to show it, but basically, this is the magic that lets you render other components without having to say, like, render foobar.new. [SPEAKER_02]: Instead, you can just say foobar, and it renders. [SPEAKER_02]: And it's because when you put a component class into this module, it's going to detect that and automatically define methods that render it. [SPEAKER_02]: So if we look at component space, we see it's including components, and this means that [SPEAKER_02]: it's going to have access to those methods that mean you can render other components. [SPEAKER_02]: Sure. [SPEAKER_02]: So, so that, that will come in handy later, but we're, we wanted to start with the sub nav. [SPEAKER_02]: So let's go back to that. [SPEAKER_02]: So this is what we're looking at in ERB. and if we want to be really old school about it, when you want the browser visible, how about you say slide? So there's a thing where Joel can drive the editor, but I have to bring in more windows and stuff. So you're going to hear a little of that plumbing as we go, dear reader. [SPEAKER_02]: Yeah, so for this, I would say we need to generate a component. [SPEAKER_02]: And the quickest way to do it is bin rails generate flex component and then call it something like sub now. Sure. So let me just bring that up on screen here. So that's... And do you want to start with a general thing like subnav or do we want to make one that's the about subnav and generalize? What do you think? Because I said we could start narrowly and then encapsulate the pattern. I'm trying to Without burning time, uselessly trying to encapsulate that flow. [SPEAKER_02]: Do we want that sub nav to be a reusable component that you can, we have like five sub nav, so it would be nice. [SPEAKER_02]: Cool. [SPEAKER_02]: Okay. so, okay. [SPEAKER_02]: We'll just say sub nav then I would say start abstract. [SPEAKER_02]: Then we may as well go, go into it. [SPEAKER_02]: Sure. [SPEAKER_02]: And make it flexible. All right. So we got a new file and a new directory. Great. Yes. Let me open . [SPEAKER_02]: So actually, maybe the first thing that we should do is render this. [SPEAKER_02]: This is just what the generator gives you and makes it easy to get going. [SPEAKER_02]: What is the view or partial or whatever where that is rendered? It's app views about, about, I think. About, about. So a Zed quirk here is if Joel is navigating to find a file, you don't see the menu. So we're just going to snap to files as he finds them. yeah yeah you found about 404 same directory but it's about.html so it's get ignored so maybe you're yeah that's why i couldn't find it yeah so that one is because people pick up and reuse the code base if you look at line four here there is a like specific I have seen a couple of sister sites that just grab our about page. And so like random crypto currency fan forum has an about page. That's like mobster is running. The admin is Peter bought Harkins. And I'm like, that's, I am not responsible for whatever this thing is. And so I try to, there's actually a second or third time bomb in here because I have had someone who just deleted this raise and then was like, great. Now it runs. it's still i'm still not in charge of your random website so that's what's happening here but here's here's our render so that's what you want to update to render the flex component yeah i i don't i didn't realize that you could render without this i thought you needed to put that in there but you don't need it well i mean apparently not because i can see it in the browser but i would have assumed so as well there must be rails magic happening [SPEAKER_02]: Right. [SPEAKER_02]: So since we're rendering from ERB. And I did a quick, so off screen, I did a quick grep of app views, and I think this is literally the only call that doesn't have the equals on it. So I have no idea how it works. Yeah, I'm assuming... Oh no, there's a couple other for these subnavs. [SPEAKER_02]: Weird. [SPEAKER_02]: I'm assuming it must just push the rendered thing to the output buffer when you call it, but it's okay if you do the equals because it just returns like an empty string or something. Sure. [SPEAKER_02]: I don't know. [SPEAKER_02]: Anyway, it seems that it works like this. [SPEAKER_02]: So let's... Basically, we can update this from referencing the partial to instead referencing our... [SPEAKER_02]: component but we're now going to be referencing like an abstract component rather than a concrete like about subnav so we might want to have an about subnav at some point but let's just see if we can render this one so we can just do subnav or components subnav.new and that should [SPEAKER_02]: render the component that just says, you know, find me in app slash components slash subnet. [SPEAKER_02]: We'll render this text. we got a typo.

30:00 pushcxDid you need to say, yeah. Did I type it wrong?

...08 joeldrapperIt says undefined and then like nil class, like some anonymous class components. Are we in a funny namespace inside of you? [SPEAKER_02]: shouldn't be and also did you mean to change it to percent equals because you didn't no i didn't so the issue is it's not even running if it can't find the components namespace then it's not even running our initializer right it was it was this initializer that defines the components namespace okay we need to restart the server Oh, you know, I probably haven't bounced it since we ran the generator. All right, it's coming up. [SPEAKER_02]: And initializers are not reloaded. [SPEAKER_02]: It's up. All right, so now we have lost our subnav. There's nothing visible, which makes sense because your component, I think everything was commented out. [SPEAKER_02]: No, it wasn't. [SPEAKER_02]: So this makes me think that when you're rendering flex, you do need to use that equals sign. Let's test. [SPEAKER_02]: Let's see if we can do that. [SPEAKER_02]: So it was this one. [SPEAKER_02]: No, it wasn't. [SPEAKER_02]: It was this one. [SPEAKER_02]: Let's change that to equals. [SPEAKER_02]: I expect that will fix it. Yeah, there you go. Find me an app component subnav. All right, so we learned something. [SPEAKER_02]: Yeah, so... pushcx Oh, the Zed theme is Brook Code Dark. I was stressed out by the angry fruit salad themes but Joel found me one that is almost as boring as my vim scheme.
[SPEAKER_02]: This is probably a good opportunity to explain flex kits. [SPEAKER_02]: So basically flex kits would mean you don't need this and you don't need this. [SPEAKER_02]: You could just do this. [SPEAKER_02]: And you have to disambiguate that this isn't a constant reference. [SPEAKER_02]: It's a method call. [SPEAKER_02]: But obviously if you're passing a block, that's fine. [SPEAKER_02]: You don't need it. [SPEAKER_02]: And if you're passing arguments, like hello. Yeah. So... I didn't know you had already found this terrible trick of Ruby methods, you can have a Ruby method with the same name as your class as a constant, because very subtly, they're separate namespaces in Ruby. Right. And so I didn't know you were already onto that trick. That's great. [SPEAKER_02]: Oh, yeah. [SPEAKER_02]: I use it heavily. [SPEAKER_02]: But the problem is, that only works if you're rendering a component from another component. [SPEAKER_02]: If we're rendering it from ERB, then we're going to use Rails interface, which is that we create a renderable, and we pass it to the render method. [SPEAKER_02]: So this is how that works. [SPEAKER_02]: So back in sub-dat nav, let's just copy over [SPEAKER_02]: what we were doing in here. [SPEAKER_02]: I'm just going to paste this straight in here. [SPEAKER_02]: Sure. [SPEAKER_02]: And then we can convert it. [SPEAKER_02]: So we're not going to need this content for. [SPEAKER_02]: That can go. [SPEAKER_02]: We're not going to need these tags. [SPEAKER_02]: They can go. [SPEAKER_02]: Or these. [SPEAKER_02]: And we probably don't have access to this link to different page. Yeah, so that's my helper for, as you saw as I clicked around the about pages, the one for the page I was on didn't get underlined. Right. So these are just regular A tags and they have a class if you're on the current page.

33:37 pushcxLet's see if we can find that. So that's here. [SPEAKER_02]: Cool. [SPEAKER_02]: so i would typically make this helper oh yeah i can see how it would be like just a line on the component yeah but since we already have a helper here that we can just write an adapter for it and we can continue using it for now just to speed us up so it does get used in the

34:15 joeldrapperthe main header as well, if it was in only enough sub navs, I would say, yeah, but we use it up here too. So if I click active, I can always click the slowest one. If we click comments, it has the little red underlined same as, so go ahead. Right. [SPEAKER_02]: Yeah. [SPEAKER_02]: So, for this, we can just register it as a helper. [SPEAKER_02]: So I'm going to register it as an output helper. [SPEAKER_02]: [SPEAKER_02]: and this and we can put this in component space for using this all over the place but we can let's leave it here for now because hopefully we can eventually get rid of it but this should now work except it's going to be obviously specifically the about now i think if we reload the page it will have our nav back let's see wrong button all right so it's there the styling is funny because we're not generating the exact same html but it's basically there what what did i miss that content for sub nav i think I don't remember where that's implemented. Oh, that's like grabbing something. Yeah, it's... There's like a... [SPEAKER_02]: I thought that that's how you provide content for a subnerf. So if you... Oh, I get it. So if you open AppView's layouts application... -huh. [SPEAKER_02]: Yeah, so we're going to have to use this again. [SPEAKER_02]: But this is a pattern that you wouldn't typically use in Flex. [SPEAKER_02]: But luckily, there is an adapter for it. [SPEAKER_02]: So we can include that helper. You get it, but we haven't shown it. Can you pull up the application layout to show what's happening here?

36:20 pushcxYeah. Yeah. Application. [SPEAKER_02]: What's happening is because of how ERB layouts work, what Rails does is it will render the whole page first. [SPEAKER_02]: And while it's doing that, it lets you put into like a buffer of content for. [SPEAKER_02]: these pieces of HTML or whatever values that you want to be put into the layout. [SPEAKER_02]: And then finally you render the layout, which has access to yield these content for things, which we can see here.

37:07 joeldrapperAnd then later on 121 is where we actually render the page. And so since these two things are separated. Yeah. [SPEAKER_02]: Yeah. [SPEAKER_02]: So this is... [SPEAKER_02]: me this is like quite a strange pattern doing the page first and then wrapping it in a layer that layout it prevents you from doing things like streaming and stuff like that sure and if if this was a completely flex app you probably have got rid of it but since it is [SPEAKER_02]: Like, obviously, no one's going to adopt Flex for everything overnight. Yeah, big bang refactors never work. I mean, Lobster's is small enough we could actually do it. Not, you know, in the two hours of the stream, but in like a day or two of coding. But it's maybe more useful if you want a demo to see how we... [SPEAKER_02]: fit into the existing pattern go piece by piece so you added a like include flex colon colon magic thing all right content four so very similar to how we've got this register output helper link to different page and that basically says look in rails is going to have this helper called link to different page yes and when we use it from inside flex we want to push it to the output buffer so it as if we had done [SPEAKER_02]: this with the equal sign because flex doesn't have that concept so we need to disambiguate output helpers and value helpers because the value helper is something like about path right where we we don't want to push the about path to the html output buffer no we want to pass it to this helper so that it can decide the class on the nav item yeah [SPEAKER_02]: So all of the built-in Rails helpers, Flex has defined an adapter for them and has disambiguated, like, is this an output or a value helper automatically? [SPEAKER_02]: And actually, if we didn't include this, we do get a very nice error message if we try to use it. [SPEAKER_02]: I like errors. [SPEAKER_02]: I make lots of those. [SPEAKER_02]: Let's put this back. [SPEAKER_02]: What is it? [SPEAKER_02]: ContentForSudner?

39:32 pushcxYep. Let's yank that. So I yanked it. Frici Knowing who Zed targets, I figured most of the pre-stream setup went to making it as out of your way as it allowed, flashy theme included LUL
So we can just get rid of these. [SPEAKER_02]: And oh, I'm used to having RuboCop automatically format this for me.

...47 joeldrapperYeah. So I got the plain theme, and I turned on Vim mode for myself. And then there was some kind of really incredibly irritating thing where it kept showing git blame in line and i figured out how to turn that off because it kept turning itself back on and then otherwise i haven't configured this at all all right so here's our error oh nice try including content four yeah so basically because you should this was a tricky design this could be maybe like add and then give me the whole line just give me the include space i don't need to i'm a programmer all i do is copy and paste things right so right yeah so so this is a decision to basically not include any this is a a stream character yeah you're right struggles with linux clipboards so this is a decision to not include any [SPEAKER_02]: helpers from rails by default apart from the routing helpers like about path and that kind of thing automatic included and the reason is about 95 of them you probably will never use if you really go all in with flex right because i could see replacing a lot of them [SPEAKER_02]: Exactly. [SPEAKER_02]: And they're, and they're so easy to replace. [SPEAKER_02]: Like even, if you're, if you're really in on Flex, you probably wouldn't even use link to. [SPEAKER_02]: You'd probably have your own helper that's maybe similar to link to, but it's, sorry, your own component that's similar to link to, but that's for, that's for you outputting links. [SPEAKER_02]: and so then everything is consistent. [SPEAKER_02]: Like if you want to do a link, you would do like, link to about. [SPEAKER_02]: about path and this is a component just like everything else right well and this could also be a like link to different page component do you actually do you make one line components like this or would you just make it a method okay yeah i sometimes do and because it is just a method right it would just be link to different page yeah and that's the name of your [SPEAKER_02]: component and then everything is just a component and there's less different concepts to think about like helpers and components anyway i think we should just leave this using this helper and using these helpers and this should now work i think is there this is a zed question is there an indicator somewhere on zed that the file is unsaved [SPEAKER_02]: Yeah, if I... Oh, it's a dot. There's a dot in the tab. Okay. Because I didn't know, and I keep trying to reload. All right, so that looks like it did before. [SPEAKER_00]: Yeah. And if I click over, yeah. Looking good. All right, so... [SPEAKER_02]: So what, so what's actually happening here is content for is basically saying, okay, I'm going to create a new output buffer just for this block. [SPEAKER_02]: And I'm going to render this block and I'm going to push it into the content force. [SPEAKER_02]: there's like a store for that. [SPEAKER_02]: So this, this component doesn't actually produce, and it doesn't render anything at the time you call it in the place you call it. [SPEAKER_02]: It puts something in your layout. [SPEAKER_02]: Frici I like the magenta indicator for joel's actions but boy is the magenta border on the whole buffer such a distraction...
[SPEAKER_02]: And there are techniques and things that you can do to avoid doing that. [SPEAKER_02]: And if you kind of go all in on Flex for layouts, then you get to render your whole view in order, which means you can actually stream it as well. pushcx Do you know how we can change it to a more chill color?
[SPEAKER_02]: That becomes an option. [SPEAKER_02]: Right. [SPEAKER_02]: We have to compromise here, I think, and keep this pragmatic. [SPEAKER_02]: So these two things, right, register output helper is just defining a method and include is just including a module that defines the method. Frici no idea, like you i'm a vim only person HahaSweat
[SPEAKER_02]: So we could, if we're using these all over the place, we can just copy these and put them in components base. [SPEAKER_02]: That's fine. [SPEAKER_02]: If this is the only place where we're using this, then I would say just leave it. [SPEAKER_02]: Yeah. Yeah. Let's leave it here. Cause we're just, we're tinkering along and yeah. so if you jump back to about.html the difference in the render call is very small where instead of naming the partial we're naming this constant okay so off screen i switched the repo i added a branch called flex and i switched us over to it so why don't you make a quick checkpoint commit that we've actually why don't i because yeah i can't do it zed have that no okay so it doesn't all right so let's look at this we have the gem file change the call and about where did the l and then there's also the new file so let's add everybody and we'll say add flex Move, oh, we can remove views about subnav. And then we'll say add flex rm r. What would you say? Absorb? Convert? [SPEAKER_02]: Yeah, convert, I guess. It's obvious from context. And I probably won't push this branch. I don't know. We'll see. So I just wanted to grab a quick checkpoint. So since this is such a minor change, let's absorb one of the other sub nav files so that we can kind of see. a little more flexibility. [SPEAKER_02]: We need to make it abstract, right? [SPEAKER_02]: Yes. [SPEAKER_02]: We've called it subnav, but it's not actually abstract. [SPEAKER_02]: It's really a concrete about us subnav. Yes, it is. So let's go ahead and make that change. [SPEAKER_02]: Let's do that. [SPEAKER_02]: So what we want in terms of an interface here is probably something like we pick up the nav. [SPEAKER_02]: What is it? [SPEAKER_02]: Something like this. [SPEAKER_02]: Does that valid ERB? [SPEAKER_02]: I already remember. [SPEAKER_02]: Yes. [SPEAKER_02]: yes. [SPEAKER_02]: And then we want to do something like, nav dot, item about, in fact, let me just copy from, I was going to say, we've got, from here. Oh, I was going to copy from the handbook, because this is pretty much the exact first example of... Yeah. [SPEAKER_02]: I'll just copy what you've got from here, but this time we're going to change these to nav.item. Oh, multi-cursor. I'm like a caveman seeing fire about that one. [SPEAKER_02]: And they're going to need to be in ERV tags as well. That's... You can't get rid of this closing tag at the top here and then stay in Ruby and end? [SPEAKER_02]: You could do, but it would be... [SPEAKER_02]: In my view, it would look inconsistent with... [SPEAKER_02]: with basically how all the rest of ERB looks. [SPEAKER_02]: And so I wanted to make this feel very comfortable and familiar in ERB. I want to make it uncomfortable because I really actually don't love the way ERB looks. [SPEAKER_02]: I don't know if you can, because I think... [SPEAKER_02]: So this is the way that you would do it from Flex, but I think this won't work. Let's find out. yeah well it won't work anyway because we haven't implemented an item method oh right well you didn't actually change but we didn't get a syntax error i would like to leave it like this because i just find it one of the reasons i got curious about flex is i find erb very very noisy and yeah it really just constantly hits me in the face that what i'm doing is like this souped up turbo version of having of having a string and being like hi like yeah yeah right like and so it feels like i've just added another syntax where it's like oh well okay instead of saying this i'm gonna say this and for all of the reasons that's wildly dangerous so is the erb So I would like to hang out with this unless you think it's going to break something. Let's go make subnav. [SPEAKER_02]: I think it won't work specifically because of the way that Flex has to adapt to be ERB compatible. [SPEAKER_02]: I don't think this will work. [SPEAKER_02]: But just as an example, what this would look like if we were looking at another Flex component here is you would just delete this, delete this, delete this. [SPEAKER_02]: And this is valid Flex. [SPEAKER_02]: see that's i like that a lot yeah so that's that's what we'll get to but i think for now we're gonna have to go back but we'll try it like this yes please what happens when you render and you pass a block is by default flex is going to yield we can't see it on your screen because it's black and white it's not very clear in the stream but this is now oh yeah this is a very goth color scheme [SPEAKER_02]: Right. [SPEAKER_02]: So it's going to yield itself. [SPEAKER_02]: So we're going to get here an instance of this component while you're rendering it, right? [SPEAKER_02]: We've rendered the opening part of the template, then we're yielding the content, and then we'll render the closing part of the template afterwards. [SPEAKER_02]: Sure. [SPEAKER_02]: So when we call nav.item here, all we're doing is calling the method item on subnav. [SPEAKER_02]: So we can just go and implement that. [SPEAKER_02]: all right let's jump back to subnav and we'll implement item to take text and path oh you know if then so we'll just copy this yeah oops chat you can express an opinion but i see now the file name is always at the top i was keeping this tray open to show where we were But since it's at the top here, I'm going to close the tray and have an extra 10 columns of code.

51:10 pushcxBecause we can only stream in 1080p. Yeah. [SPEAKER_02]: So now what we're doing is we're yielding our component inside the subnav. [SPEAKER_02]: And then we're just letting you call this item method several times, which is doing this. [SPEAKER_02]: And then we get to the end and we close out our content for a tag, which is… this represents the subnav wrapper. [SPEAKER_02]: And in fact, instead of this, there's a shortcut that you can do, which is to pick up… this is just a style preference. Right. You can pick up the block. and just pass it in like this. Which does the same thing. fractaledm1nd You need to replace the hardcoded string and path and use the item method arguments

52:10 joeldrapperThere's... Well, can you update line 13 to do the... Oh, and Fractal Mind got it at the same time I did. You're not using the passed in... You're right.

...28 pushcxYeah.

...29 joeldrapperYeah. So... I can see this code two ways. It's like the vase or the old woman. Or what is it? Face or two faces. And then it's young women or old. Where on the one hand, I'm like, oh, we just kind of did a bunch of gyrations to write code that just plops out a very simple string. And then the flip side of it is, oh, also, This is just Ruby. And so, especially if we had any kind of variable or anything, well, like we could actually drive a test through link to different page. I don't think we have one now because it's such a simple function and I don't want to add one. Just I realized like, oh, wait a minute. I have the real power of Ruby happening here. Okay. Yeah. All right, let's keep going and try another subnav. [SPEAKER_02]: Because of the way that you can just pull other components, right? [SPEAKER_02]: If you wanted to extract this to be its own component, I don't see why you would. No. [SPEAKER_02]: But you could just pick up like dot, dot, dot here and do item or like nav item dot, dot, dot. Wait. is that triple dot valid ruby or you're just saying you would name your args no that's valid ruby so that picks that's like when did that get introduced star star and percent right it's like the equivalent argument forwarding when did this get added a new shorthand syntax okay so that's why i don't know it yeah [SPEAKER_02]: But yeah, so this should work now. [SPEAKER_02]: I don't think it's going to work because of what we did to the ERB. [SPEAKER_02]: But let's see if it works. [SPEAKER_02]: It might do. [SPEAKER_02]: I'm pretty sure it won't. We only see hats. Yeah. If I click on hats, we get... Whoa. An explosion. Oh, so these other pages... we're still rendering the partial. So we didn't get a complete test. When I was clicking around before, only about was rendering your component. The others were rendering the old subnav, which I just deleted. So can we poke through the other? Can I collapse that? Here we go. Models, views. So let's update. [SPEAKER_02]: Well, before we do that, we might want to... Actually, you know what? [SPEAKER_02]: This will save us the bug that I thought we were going to run into, which is it looks like we need to use this component in a number of different places. Yes, that's its point. [SPEAKER_02]: Right. No, but what I mean is... Yeah, so we don't want to put this here because then we'd be repeating it in the others. Right. [SPEAKER_02]: Yeah, I think we actually want a, you know, about subnav component or something like that. Yeah. Shall we make, so I guess style issue. Do we want to make a subnav module and have like a base and have about in a subnav submodule? You know, as a longtime Rails programmer, I'm used to just kind of barfing all of my constants into the global scope. But now I have to think about whether things belong in modules. And just for reference, there's like four or five subnab files. [SPEAKER_02]: yeah so where i usually start five is just putting everything directly under components because it's simpler with the way that kits work i'm thinking about making it so that you can that kits kind of like spread out so if you put another module under kits that that module itself will become a kit but right now that's not part of flex so that the simplest thing is for us to just create a new component you know i'll just write it here because i can't So let's go and say, and do you want me to, as long as I'm like hitting enter on this, do you want me to make one for, let's say stories? Cause that's the next one I would want to do. Okay.

57:22[SPEAKER_02]: All right. [SPEAKER_02]: So we can copy all of this into about subnerve and just paste it here. [SPEAKER_02]: Get rid of the ERB stuff. we can get rid of this and get rid of, oh yeah, this was that nice little syntax you were showing off. [SPEAKER_02]: And now about subnav is just composing our abstract subnav. [SPEAKER_02]: If we go back to the ERB, we can render about subnav and get rid of... Throw away this hole. Right. And do you need the .new or does it... No, because the render is going to be a Rails call. [SPEAKER_02]: You don't actually. [SPEAKER_02]: No? [SPEAKER_02]: Yeah, you don't. [SPEAKER_02]: No, even the class itself responds to, I believe, the class itself responds to Rails interface, which is that it needs to respond to render in. [SPEAKER_02]: I think this will work even without the .new. [SPEAKER_02]: As long as you don't need to pass it any arguments. And we don't. Well, we do. We need to... Or no, link to different page actually introspects the controller. So we're using a kind of global state that way. So I can see how we get away with this. All right, let's leave it like this. I really like that. Yeah. All right, let's jump back to... So about subnav has that block with the different items. And... And then nav item is defined up in the sub nav component, which. Yep, I don't even know how to find. Okay. Oh, yep. I deleted that. I'll just jump to it. Don't close. Yeah, let me get back to following you. And then so that's where our item so. It's funny. It's very, it feels very cheap to define a DSL this way, you know, not cheap in a bad way. Just, oh yeah. A method is an item and we're, it's actually kind of pleasantly ambiguous whether item is like a whole component or just calling this helper or putting a string in here in the same way I've grown to like the way Ruby syntax is vague about whether you have a variable or a method all right so let's yeah let's grab in the stories one two [SPEAKER_02]: Instead of us having to, like, if we go back to here as an example, right? [SPEAKER_02]: The other options we could have had, like, sub nav item. [SPEAKER_02]: Yeah, yeah. [SPEAKER_02]: That's a lot. [SPEAKER_02]: About path. [SPEAKER_02]: But the problem with that is it doesn't ever make sense for us to render a sub nav item outside of a sub nav. [SPEAKER_02]: So I prefer to have sub nav encapsulate that and expose the interface to item. [SPEAKER_02]: Sure. [SPEAKER_02]: That's just my preference. So if we look at recent, the stories sub nav has newest and top. Let's consume this or actually wait. Let's finish the about because we have to put it on chat privacy and the hats index, which is over in another view folder. [SPEAKER_02]: Yeah, so let's just copy this and I'm just going to jump to... [SPEAKER_02]: I can't search for these. [SPEAKER_02]: Here we go, here's chat. So here's privacy. [SPEAKER_02]: It's not on privacy. [SPEAKER_02]: It's not on 404 either. Probably should be. No, it's not. There's a lot of organization happening, but if you look on the right, because I think you have the stream visible, we have hats index uses this. Let's do that one. And chat uses this, which I think you just did when I was looking down. [SPEAKER_02]: I did. Yeah. Okay. And then we already did about, and then stats index uses it.

01:01:57 pushcxThere you go. All right. I think that's it.

01:02:06 joeldrapperWrong one. I'm going to rotate up this. So now if I go back here to this and I go to chat, maybe we should have added a visual styling to make it clear that the subnav is itself. But honestly, these were broken when I clicked over because I deleted that file. Let me throw this in as... [SPEAKER_02]: There's one more thing I'd like to do, but I don't think we can do it until we've moved all of the subnavs to use this. [SPEAKER_02]: And that is to take the wrapping div or whatever it is that's in application layout. This whole content for thing? [SPEAKER_02]: no the header with the id and oh and move that yes and yeah it kind of so we would just we would just get rid of this line line 105 and line 107 right and put it inside the sub nav component yeah that would make a lot of sense but [SPEAKER_02]: I just, I don't think we can do that until we update all the rest of the subnav partials. No, we would end up double or they would end up without because yeah. Yeah. That's how the code is reused. All right. So let's jump ahead to the stories sub nav now. So it's app view story, sub nav, HTML, ERB. Right. [SPEAKER_02]: Yeah. [SPEAKER_02]: stories i got it yeah so very familiar pattern here yeah so we can we could use the subnav component in here or we can replace this yeah i want to replace and just totally delete the stories subnav partial yeah so i'll just copy these two lines and delete this yeah [SPEAKER_02]: We have a story subnav component. [SPEAKER_02]: Not for long. [SPEAKER_02]: And this is just going to do subnav do nav item. Actually paste these in. Did you delete that file as fast as? I didn't even see it happen. [SPEAKER_02]: I did. All right. Way ahead of me. Great. It's so strange to have someone on my file system. [SPEAKER_02]: Yeah, it's weird. [SPEAKER_02]: I just need to change my... [SPEAKER_02]: I had hardtab set to false. [SPEAKER_02]: I didn't want to do that. [SPEAKER_02]: Sorry, yeah, I had hardtab set to true, which is giving me a tau character here, which messed up the indentation. Oh, yeah. I didn't attempt to... Oh, we stopped following. I didn't attempt to set up standard rb in zed. [SPEAKER_02]: That's right. [SPEAKER_02]: So if I now just do item, oh, nav item, right. I think that's it, right? That does look like it. Yep. Missing template. Oh, so we didn't replace the call. So let's jump over to Yes.

01:05:36 pushcxHuh, that's not So we need to find home index where we're rendering.

...48 joeldrapperYeah. So yeah, no, no, all the way under it. It's that above. So we reuse a lot of like that concept of this page has a list of stories on it. is reused across a bunch of things and they all have slightly different texts above and below them. And so there's this, cause everything in action pack is wired together with magic instance variables. We have this above. So you jump to the, I think it's stories controller. It should be. So we were looking at say, and if you go to recent

01:06:37sorry what do you mean yeah look for def recent you had that you showed me just before the stream clever there isn't a recent death oh it's so we're not in stories this is we want to be in home controller sorry this is we have a couple of slightly odd organization things so above yeah so you get what's happening here is we are saying that Above the page, we want to render this story sub nav, which is shared by, we'll also have to update the calls in the new and top methods. Yeah. There's a lot of, there's a lot of like kind of different strategies here for this page is just like this other one, except, and then the the inverse of that where it's this page has the same thing as the other one. And there are so many strategies for that. [SPEAKER_02]: We should be able to just point this to the component, but I just want to highlight how weird it is that we set an instance variable in this controller and it pops up in a view somewhere completely unrelated. Yeah, well, it's because that view is... reused by these various actions so you're going to have to search this also for actually if you just search this file for stories subnav you're going to find two more instances yeah see here's a different one above and then top should have it there you are

01:08:30 pushcxOh, we called it. Yeah. I don't know. Does this guy do fancy tab complete? [SPEAKER_02]: It does if you have Ruby LSP installed.

...42 joeldrapperYeah, I have an error message about Ruby LSP. I did try like gem install, but it's not in the bundle or something. I don't know. I didn't want to. Yeah, I actually have him enough. I don't need to do in a second editor. There's one more of these. [SPEAKER_02]: Well, actually, there's two. [SPEAKER_02]: I just want to double check. yeah we're using this with an equal sign after it so this will work yes yes did you say there are two more yeah so if you go to app views home underscore active yep got it so this one this one could just be replaced out right with yeah so This one, the active view wanted to have a little help text as well. And so it's doing that same pattern up in the home controller where if you looked in the active action, it would be naming this partial in the same way. And then the last one is in the views home newest by user. Ah, you found it. [SPEAKER_00]: Yeah. Great. So over here on recent, I see recent. I see newest and top. There we go. [SPEAKER_00]: Great. So this is a, I don't know if Peter you're watching, but, a regular stream viewer named Peter P I T R has opened an issue about improving the navigation on the site. And one of the things that is jankiest about sub nav is if you watch the baseline of these newest and top, they're going to jump. Cause when that underline disappears from recent, the height of the top thing changes. Peter is not a graphic. Peter, me, is not a graphic designer. And occasionally you see these janky things on the site. And also, because it makes these wider, you can see newest and top kind of shift left and right a little. Anyways, if any actual graphic designer is watching and wants to help, we have a whole... Issues. We have a tag. Label design that is, you know, Peter is bad at design and you can fix all of these things. All right. So this sub nav, this is nice. I like how expressive it is that we are sharing the same structure and because it is Ruby, if I wanted to say. Well, we're just going to pass an array of these links. We could do that. fractaledm1nd Render the borders always for every link, just make the border color transparent when not active and colored when active. No jank. Same UI
And instead of using this, you could, yes. Yeah. [SPEAKER_02]: Yeah. [SPEAKER_02]: If you want it, you could even pick up like, yeah, you could do it like this. pushcx that would be an improvement yep
yeah. And at that point, you know, it's a little bit primitive obsession, but You get exactly where I was going, where it's this is just Ruby. And so we can integrate these different components and this different code reuse however we like. Maybe they come out of a database. I don't know. I just, when I saw it and it kind of clicked, I was like, oh, this is not too clever. This is very comfortable. And I have a whole toolkit of Ruby I can do. Exactly, yeah. [SPEAKER_02]: And if you wanted to put some HTML inside the item, we're just putting text. Yeah, why don't you, I don't know, make that italic or something. [SPEAKER_02]: Right, so to do that, we're currently just taking a positional argument for the text. [SPEAKER_02]: So let's jump back to the subnav. [SPEAKER_02]: What we would probably want to do is remove this argument text and take a block as text. Right. I've seen that's how flex really handles the contents of a tag. And so now we're making this item method look a lot more like the component interface. I see the style. [SPEAKER_02]: Yeah. [SPEAKER_02]: Now, because... [SPEAKER_02]: because we're doing this helper you're going to capture the output of the block you would have to capture it yeah right that's what you can capture just like this if you were using an a tag you could do breath is our path and then oh yes that's very nice well actually hang on a second i would even just rename this to href and then delete this Ooh, hang on. Keep that up. Can you make a vertical split and load up that link to different page helper? [SPEAKER_02]: All right, sure. [SPEAKER_02]: Because there's a button. There's this button up here, right? [SPEAKER_02]: Right. But I think you have to click it because you're driving. I don't want to lose track of your cursor. [SPEAKER_02]: That's fine. [SPEAKER_02]: I'll try. [SPEAKER_02]: I've got it here. [SPEAKER_02]: Let's go split on the right. Oh, yeah. I don't know if you're seeing a split pane, but we're not. [SPEAKER_02]: All right. [SPEAKER_02]: I just wanted to kind of... You have to do it yourself. I don't want to bog down into the editor. Can you just copy and paste the whole method there into our subnav? Because I was thinking about this, and I was like, we're actually really close to just consuming that helper and doing it in pure flex. [SPEAKER_02]: Yeah, let's do it. Oh yeah, there's that whole thing around path or a paging. [SPEAKER_02]: That's okay. [SPEAKER_02]: We can, we can do that in flex. [SPEAKER_02]: Yeah, let's so, so let's, let's get this working. [SPEAKER_02]: So probably is the request path subbing this out and we subbed that. yeah so what's basically happening is it says we we put this slash page at the end of urls for pagination and so it's trying to throw that away from the current request path so that it can compare and say are you already on anything under this sub nav okay so let's get the current And Caius, if you're watching, this is one more bit of jank that would go away if you want to implement pagination. So you want to grab 17 and 18, because that logic stays the same. Actually, and 19, right? [SPEAKER_02]: well it's not so i don't want to grab 19 yet so what we want to do let's let's try and write the code up here first actually so link to text path options we're not going to use link to we're going to use an href so path is going to come down as href text is going to be the block what about options we need to pass in a class here so let's do class and [SPEAKER_02]: I'm going to pass the class in as an array. [SPEAKER_02]: And the reason is it's really easy in an array to wrap up an expression in parentheses. [SPEAKER_02]: And then you can do current page if current page href. [SPEAKER_02]: So now let's make this current page taken href. And let's do... Yeah, there's a style thing here where I would never make a method for this. I'm barely okay with making temporary variables. [SPEAKER_02]: Oh, we can just put it in line if you want. Yes, I do. [SPEAKER_02]: This is... All right, that's fine. [SPEAKER_02]: I would usually make a method for that kind of thing, but... My usual rule of thumb is, am I ever going to call this from somewhere else? And do I need to test it? Which is the same question, except somewhere else is a test. And yeah.

01:17:34[SPEAKER_02]: so then we can just say, if, and let's just put this in line. Did you, you didn't mean to change request dot path, did you? Because this is off of the global request object. Yeah, you're right. And then href is your input. Those multi-cursors, those newfangled multiple cursors. You got to search and replace with ed. All right. So options class, and then we have the, yeah, that's real nice. Does it work? [SPEAKER_02]: What I really like about this, doing classes like this is [SPEAKER_02]: you can just have as many you know yeah you can have there are additions as you want and then if we tried to suck in comment you would really see that because there are so many options for comment but they read so nicely like it the class class is current page if current page equals if current equals href right it just reads like english and [SPEAKER_02]: When class receives an array, if there's a nil in it, it just ignores it. [SPEAKER_02]: So the if, if this is false, this expression will just evaluate to nil. [SPEAKER_02]: So now that's what we need. And then flex throws those away because it's calling compact on the array or something. Sure. [SPEAKER_02]: I like that. [SPEAKER_02]: Yeah, exactly. [SPEAKER_02]: That's very tidy. It encapsulates what's it mean to be a subnav item. So we've got some random Ruby error. Let's figure this out. [SPEAKER_02]: Yeah, it's because, oh.

01:19:20 pushcxIs it on all of these? Yes. fractaledm1nd I don't think you want to `sub!` the passed href in your current check
[SPEAKER_02]: What's the Ruby error? [SPEAKER_02]: Wrong number of arguments can take. [SPEAKER_02]: Oh, yeah, yeah, I know what this is. [SPEAKER_02]: So everywhere we're using the subnav.

...31 joeldrapperOh, fractional bind has caught a bug. fractaledm1nd That mutates the actual path for the link
Yeah. If we mutate the path, we break the link. [SPEAKER_02]: The way this is intentional that did happen in the original. Yeah. Oh, so, back to mine, I see where you're going, but this is the sub nav at the top should always link to the first page. So this is fine. That's correct. It let me think less about passing stuff in. Mm-hmm

01:20:05 pushcxOr maybe it's actually redundant because we never generated it.

...08 joeldrapperYeah, actually, why don't you just comment that out? Yeah, we'll leave that as dead code for a minute. [SPEAKER_02]: I don't think we need it. Yeah, we're experimenting. [SPEAKER_02]: Okay, so we need to jump into the story subnav and change this to... Ah, yes, because now we've changed the interface. [SPEAKER_02]: This, and then we can just put this in a block.

...33 pushcxNow it looks very... [SPEAKER_02]: flexi we just wrap this and because these are now blocks we could put html in here so we could say oh right that's why we started ah look at that and now that will be an m [SPEAKER_02]: The thing about, like, flex blocks is if you have just text in them, nothing else, the text will be output. [SPEAKER_02]: If you had, say, like, if you had, like, hello and then strong world, this would not output the hello. [SPEAKER_02]: This would be lost forever. [SPEAKER_02]: The hello would be lost because of the strong here.

01:21:25 joeldrapperHow would we shove the two together? I saw this helper that was just like, oh, yeah, we're going to just shove a string in there. [SPEAKER_02]: You almost never need to do that. [SPEAKER_02]: You'd usually want to wrap this maybe in a span like that or something. [SPEAKER_02]: But it's available if you want to output plain text explicitly. [SPEAKER_02]: Most of the time, it's fine if it's the only string in the block. [SPEAKER_02]: then it's not a problem. [SPEAKER_02]: It will output it. [SPEAKER_00]: Yeah. [SPEAKER_02]: Let's jump back to the... [SPEAKER_02]: It does your HTML escape. [SPEAKER_02]: It will get rid of this. Right. Can you jump back to the component? Because Flex has caught us peeking in global variables. [SPEAKER_02]: Where? The request. [SPEAKER_02]: Oh, you mean subnav here? [SPEAKER_02]: Yeah. [SPEAKER_02]: Oh, yeah. [SPEAKER_02]: Yeah. [SPEAKER_02]: Okay. [SPEAKER_02]: So we just need to include that one. [SPEAKER_02]: I actually... Did this come up with... Yeah. kind of like that we're being forced to be explicit about, hey, I'm digging around in global variables. Because it is a little strange that our templates were reaching up into the request object to do this. And they were doing it via a helper. So it wasn't super obvious. And it was happening up in the helper, which feels a little less janky, but it's actually not any less janky. Yeah, so wait, if I reload now, I should see. [SPEAKER_02]: Yes, it does. That's beautiful. [SPEAKER_02]: And again, you can put as many of these adapters in base. [SPEAKER_02]: It's up to you. [SPEAKER_02]: And I think after a while, you'd probably end up with maybe five or 10 adapters in there that you use a lot. [SPEAKER_02]: But most of the Rails helpers, you just don't need them. Right. So was it a considered decision to call this components base instead of say application component? Because the real style is like your top level abstract thing is called application blah, and you put the stuff you're sharing across your app. But that's one more of those things that teams tend to grow out of. [SPEAKER_02]: Yeah, I renamed it to base because I didn't like it being components, application component. [SPEAKER_02]: Because I don't like repeating words like that. Yeah. And then it's also got that S hanging on as if otherwise you could only put one thing in. Oh, wait, sorry. I said I'd let it go. Anyway. [SPEAKER_02]: Well, you can think about it as components is your module, right? [SPEAKER_02]: Yes. [SPEAKER_02]: So it's actually reasonable for the module to be named components. [SPEAKER_02]: No, it's not. [SPEAKER_02]: Because that's what it's responsible for. Everything should be singular. Your components module. Everything should be singular. Yeah. All the time. It's fine. As a joke, I'm sure someone has done the joke, but I kind of want to make a gem that just monkey patches the shit out of Rails, and it'll be called Ruby on Rail. Yeah. Yeah. Because I think it would be possible. It would be awful. Yeah. Sorry, you go ahead. [SPEAKER_02]: We're cross-talking. [SPEAKER_02]: I just constantly run into conflicts with model names. Mm-hmm. [SPEAKER_02]: So that's why, that's why I don't usually do it. Right. [SPEAKER_02]: Like, use articles, index, right. [SPEAKER_02]: If that was views article index, then if you reference article in there at all, then you're going to be in trouble because you'd be referencing the module, you know, views article as opposed to. Right. But we're back around to my earlier point that that's only a thing in small rails app, big rails apps kind of. If you namespace, they always learn, they have to namespace modules and stuff. chamlis_ lobste.r?
Cause once you get past like a hundred models, you realize you're drowning in models and your name global namespace is getting cluttered. You have these kinds of issues anyways. All right. [SPEAKER_02]: I imagine that some rails apps are so big that they would actually have, like a namespace for each section. [SPEAKER_02]: So you'd have like, I don't know, front end. [SPEAKER_02]: or whatever, and then you'd have components under that. [SPEAKER_02]: And then you'd have like dashboard and then have components under that. [SPEAKER_02]: I know at Shopify, we had basically that we had like a bunch of top level modules and then within them, it was like a rails app. [SPEAKER_02]: You'd have controllers models. Yeah. [SPEAKER_02]: And everything nested. Yeah. pushcx https://lobste.rs/s/noffmp/seve…
An article just yesterday, 25 things we know. [SPEAKER_02]: Yeah. Shameless. Are you spotting a typo somewhere? chamlis_ nah, singular lobsters
and i mentioned so someone was talking about it and there was a commenter who mentioned that oh singular lobsters dot r that would be great yes lobster where was it yeah they they mentioned that it's an issue in rails David Price- That there are very few tools to organize code that there's no package level separation and. David Price- There was a mentioned by matt's at rails conf that someone is working on namespaces but then also there's homegrown tools like pack work is the one released by shopify and the one. David Price- That I used inside of stripe looked very much like pack work and I am assuming that you know. The Stripe version and the Shopify version had some amount of cross-pollination because, you know, big companies, employees swap back and forth all the time. So like there's the Rails default and then they have a graph of the thing you were describing, right? No, I'm making up a chart. [SPEAKER_02]: I'm inventing a chart, but I'm sure if we wrote about it at some point, I just know that we did this at Shopify. It's up on stream, actually. pushcx https://shopify.engineering/a-p…
So it's visible. Yeah. [SPEAKER_02]: Awesome. [SPEAKER_02]: I just had an idea for how we can improve this without having to go and update all of the other subnames to use this, which is if we go into the application layout. Yeah. [SPEAKER_02]: so we're saying if content for sub nav, then do this, let's copy that. And let's say if content for, Oh, like just make it a different content for flex sub nav, and then you could move header in there. [SPEAKER_02]: Now we can get rid of this. [SPEAKER_02]: And if we go into our component. You can't just put a case statement on content four and peek in here to see what the class is and be like, all right, if it inherits from flex. [SPEAKER_02]: No, because content four returns a string. By that point, it's flat into a string. [SPEAKER_02]: It's already been stringified. It would have been a little evil to do that, but that's OK. [SPEAKER_02]: So here we can now just replace this with doing it in flex. [SPEAKER_02]: That's quite nice. [SPEAKER_02]: We didn't want to replace that. [SPEAKER_02]: What we wanted to do was put it inside here.

01:28:46 pushcxYes. And this is going to receive the block now.

...51 joeldrapperAnd then can I edit it at the same time? Because I can say, oh, look at that. Yeah. Oh, man. [SPEAKER_02]: That should fix it. [SPEAKER_02]: And this is much nicer because now we are able to, you know, we're able to edit our wrapper from the same component that we can edit the items, right? [SPEAKER_02]: If we wanted to add a class to our nav header or whatever, we can do that here on 9.11. right okay so i'm gonna yeah this is a good point to checkpoint well twice like so a good time to checkpoint the code but then also a good point to time check because i am going to run to the restroom real quick which means i'm going to hang up on you because i like you but i don't know this whole thing about we're sharing the file system just feels real iffy and i don't just can't like you could log directly into prod from my machine there's got to be an escape from this sandbox i don't know not that i don't trust you joel it just feels it just feels iffy so and then i know you have a a time curtain as well so you can take a quick break and then we will drag that where i can actually see it i have the hard stop in about half an hour right which is there is no way we convert comment in 30 minutes so why don't you take a second to think about what that's going to be what you want to do with the rest of our 30 minutes and i'll call you back in like two minutes okay all right

01:32:22Frici "I like you but not enough to leave you unsupervised on my machine." is definitely a title of some sort LUL
Frici Seeing how this transforms the codebase, and hearing the explanations while you two are pairing has been quite informative though 👀
all right i don't think it showed up on stream but i set my earbuds down and the cat slapped them off my desk because what is that how dare you so we'll see in the recording let's call joel back

...51 pushcxI don't know if that's ringing. Okay, it thinks it's calling him. Joel probably also stepped away.

01:33:03 joeldrapperOh, yeah. Is that how I set it, Freachy? That's actually a great title. It's better than what I had before. So I had put in the...

...11 pushcxSo strange to have someone on my file system.

...17 joeldrapperYeah, I'm having a lot of fun, too. So I have played with Flex on... a a toy project where i said all right well i'm just going to make a a brand new rails app and i'm going to drop in devise and log in log out and it was going to be like a little wiki and i got some of the functionality working and it's fun to to play around with a new library in that sort of way. But there's nothing like tinkering with an actual code base, because we've immediately run into a bunch of these things like, oh, I guess we could comment that out now. That's state code, right? Frici I could trail the vod, i think you said it more tactfully than I tend to speak but that was the gist of the line.
Oh, hey, look. We've connected. Welcome back. [SPEAKER_02]: You sound really weird now. Frici WB Joel!

01:34:14 pushcxi sound weird now i mean oh i can turn off the the break notice sounds so weird i'm gonna call you back sure oh man if joel's going back to odd audio issues so is it calling i heard a little deedly d but i think he just

...47 joeldrapperso many apps make little doodly dees. Frici That was a discord instead?
I'm not sure if that was Discord, which I usually try to close. That was Discord? Okay. Y'all know it better than me. I was just showing off how not a gamer I am by writing a post on my personal blog called Hey. pushcx https://push.cx/tv-setup
So I was just explaining that I'm not a gamer. So there was the little Discord doodly dee, and then I was like, I don't know what made that beep, and preachy immediately recognized it. But I talked about this. Frici I have that open still but haven't had a chance to read it
And this one was kind of funny, because I noted, I mentioned in here that I basically, if you can hear me, yes, you sound very pitch shifted. [SPEAKER_02]: I'm going to, I don't know what to do, maybe restart Zed. I Frici you sound fine on stream CaitThinking
have no idea what to say to that i'm talking in a normal pitch and didn't even mute my mic when i got up this sounds so funny okay so in my post i mentioned that i don't play 3d shooters which is like all of the aaa blockbusters and so i don't know if those run worse and someone immediately said oh there's this thing with hdr color in video games that linux doesn't do properly And I was like, yeah, well, I'm not a gamer. I am, but not a AAA gamer. chamlis_ currently using raspberry pis running kodi, the apple tv 4k with infuse sure is tempting though
It's also kind of funny because I... Oh, Joel is sharing a product in Zed. [SPEAKER_02]: You sound better now. [SPEAKER_02]: I restarted Zed, but now I'm sharing my thing, so I think... Yeah. Do we want... Frici I thought HDR is much improved on Wayland these days
Okay, so the audio is good, but... Yeah, the audio is good. I guess it doesn't matter who said we have. Frici probably not on X though
So let me grab yours and bring it on stream. And now you're orange instead of magenta. [SPEAKER_02]: I'm not looking at your code. [SPEAKER_02]: I'm looking at flex code. Oh, right. Because that's whatever project you had open. I should call you back, right? [SPEAKER_02]: Yeah. [SPEAKER_02]: I think you can just go to the other thing and hit share at the top, right? Right. So let me bring it back. And it will swap seamlessly.

01:37:10 pushcxIf I hit open. [SPEAKER_02]: Yeah, now we're swapped. Frici Orange you glad his color changed now though? LUL
[SPEAKER_02]: So... Yeah, you still sound fine. [SPEAKER_02]: You sounded like really deep, like completely pitch shifted.

...22 joeldrapperOh, yeah, like I'm trying to impress someone at a frat party in high school, like, whoa. [SPEAKER_02]: No, like you're an alien. [SPEAKER_02]: Oh, okay. [SPEAKER_02]: In Star Wars or something. You have the strangest audio issues. Yeah. Also, do you want to unshare? Because off stream, I have a folder on your, like it is also weird in the other direction that I am editing files on your computer. I don't need this power. [SPEAKER_00]: There you go. There we go. Great. All right. So we're in a known good place. You've had a minute to think about what would we like to do with our last 20 or so minutes. And one thing we could do is if you want to show a layout, we could try consuming the application layout. But maybe you came up with a better idea. Because I was thinking, what are the other big features of Flex? And we didn't show a layout. [SPEAKER_02]: Right. [SPEAKER_02]: Or maybe there's something. [SPEAKER_02]: They're like the last thing you'd want to do. OK. Yeah, because usually. With this kind of conversion, you would start with a very simple thing like subnav to practice and get everything wired up, and then you'd slowly work your way up in complexity. [SPEAKER_02]: Yeah, we could probably... [SPEAKER_02]: I don't know if you want to make a start on the comment thing or start looking at it. How about... We see if we can wrap up a performance benchmark around this. So we had the open question. [SPEAKER_02]: Yeah, that's a good idea. Of how long does it take to render a story with comments? And it occurred to me there's actually a third path. So the three paths would be, let's pull up a story so everybody sees what I'm talking about. if we none of these have comments because it's in that so live story oh oh my god why is my dumb blog post this is one of those like look it's sort of meta this does not deserve 38 upvotes this deserves like three i don't know about 38 so all right i'm distracted by the meta now i just all right so We have the tree of comments and the code base does lots of complicated things to make this performant because this is something like half of the pages that Lobster serves every day. And the naive version of this with, let's take the comments, let's loop, So ignoring the complexity of how do we get the comments from the database? We will leave that complexity alone, even though, even though that complexity literally came up in this performance thread that I shared earlier. It's funny, you, you, I have to talk about it. You mentioned there are just so many interesting little things. You had mentioned somewhere to me that ActiveRecord like regexes strings and We have taken a noticeable performance hit from ActiveRecord because we have this big thing that's a string from before. This is a recursive comment table expression to fetch out all of the comments in their correct tree order. And I wrote it before ActiveRecord had support for recursive CTEs. And so it's just one big string. There is actually a performance string because this is considered a big string, and it is 2% of the runtime on loading an individual story, which is kind of wild to me. That's nuts, yeah. It's totally accidental. [SPEAKER_02]: I don't know. [SPEAKER_02]: The way the Active Record passes SQL is very primitive. [SPEAKER_02]: It's using regular expressions. [SPEAKER_02]: It's no wonder it's slow. Yeah, I... The thing that surprised me actually is... [SPEAKER_02]: They would need a parser for every adapter, right? They'd need a parser for SQLite, MySQL, and Postgres separately, because they all have different... Yeah, the thing that surprised me is that they even tried, because I was just shoving in a big string, and I would expect from this interface that they would just say, is the string you gave me the name of a table? Well, then I know what it is. Or the name of a model, or I guess a join would be a better, then I know what it is. Or not a join, a AR relationship. And if it's none of those things, I'm just going to treat it as an opaque string. And so there must be a active record feature that I am oblivious to that means there is a benefit to trying to regex my nonsense. [SPEAKER_02]: I don't know what it is. And it's so funny that, I feel a little bit like, like someone has walked in on me on the bathroom. Cause like this whole thing with confidence order path is just a heinous hack of bit bashing inside of a string to try and sort comments correctly and work around a limitation of Maria DB. And now it's memorialized in. this big public repo, it's like, ah, come on. I was ashamed of that. All right. So that's how comments come out of the database. Shame. And then they're rendered by... Well, the naive way would be to loop that collection and call render partial for each comment. And we can't call render collection because these things are nested and... each one of these is each comment has a little ol and that's where the indent comes from and i know joel that you know this because we talked about it but i am explaining this because we have like twice as many people watching this on monday so we have to loop and call render partial for each so that's that's let's put this in the scratch and we have to nest them as well and they're already [SPEAKER_02]: What's the structure that you get out of Active Record for this? [SPEAKER_02]: Is it... It's a flat... [SPEAKER_02]: It's like a... [SPEAKER_02]: It's flat? Yes. Well, so... Interesting. And then there is... So let's call this, like, Naive Rails Native... then we will call this the live thing which is right peter awful well yeah heinous heinous is the name so to show this very fast so i had noticed that there was a significant performance overhead to rendering partials this way and I didn't make a whole view component library about it. I did awful things with partials where I said, take the partial and then my thread has this magic comment, because that's how we do things in ERB, you know, magic comment. I made my own magic comment. And this names a partial and then my heinous inline partial, call it a helper, in lines the contents of that partial here. Yeah. And that's a little bit bonkers and awful, but it made a significant performance improvement. And then Lex would be this. So I think it is worth doing it each of the three ways. So we could wrap the instrumentation around number two now. And then I could wrap number one because I can turn this back into the naive thing pretty easy. And then maybe either you do the flex component version offline, or we would do another stream. But I don't know your schedule or anything. So [SPEAKER_02]: We could do a fourth one, if you like, which is to experiment with the new fragment caching that we're putting in Flex. Caching is... It feels a little unfair to do the Rails one without and compare against that, right? Like, that's going to be... [SPEAKER_02]: I mean, the caching is probably going to be... [SPEAKER_02]: Basically, they'll be equivalent because at the end of the day, they'll both be making the same calls out to the Rails cache to get back HTML strings. [SPEAKER_02]: It's only the uncached case that's going to have a difference. I see why you say that, but my theory and practice alarm bells are going off, because every time I say something like that, it's like, oh, it's true in theory, but in practice, one of them uses the interface slightly different, or one of them doesn't evict the same way. In theory, they're the same. [SPEAKER_02]: what's the line in theory theory and practice are the same in practice they're not and so like you just said it and my alarm bell went off so all right yes i mean there will be just as an example there will be an eviction difference between them yeah because i think at least when we first introduced this feature i don't want to write so [SPEAKER_02]: It's very difficult to know, basically, is there anything inside this cached block? [SPEAKER_02]: that depends on another file. [SPEAKER_02]: And if that file gets changed, let's invalidate the cache key. [SPEAKER_02]: You have to build up like a big dependency tree. [SPEAKER_02]: And for flex, that would mean probably writing a prism visitor to look at all of your flex components and see all the dependencies. [SPEAKER_02]: So the naive approach. Yeah, you had talked about that you had hacked in a... just clear the cache when Rails restarts. But right here, we have relative timestamps that we render from Ruby. So there's a need for cache eviction. [SPEAKER_02]: Well, there isn't. [SPEAKER_02]: Well, I guess there is. Like, yes, we could cache just the markdown, but... Yeah. it feels like punting on what the actual design demand is because i don't want to cash half a comment because then i still have to go to the database and everything like i want to cash this whole thing you know i want the cash to be like a logical comment not in the middle because i can do all those kinds of weird in the middle i have done all those in the middle crimes already so [SPEAKER_02]: The solution to that that I've often used is to render from the server an absolute timestamp and have a tiny bit of JavaScript. Yeah, see that right there. [SPEAKER_02]: To a relative timestamp, but we've already lost it. Let's open the... There's no JavaScript. There's no JavaScript. We are a modern website. Look, source. This page has no sources. Look at that. [SPEAKER_02]: That's great. Look at our moral superiority right there. Honestly, it's at this point, it's just this weird point of stubbornness that we don't serve JavaScript unless you're logged in. And I have actually in the last year or two, almost entirely made the site usable without JavaScript. [SPEAKER_02]: Yeah. [SPEAKER_02]: Yeah, you'd only need it for an optimization, right? [SPEAKER_02]: It's not necessary, but if you wanted to have better cache eviction, then... Well, it's not better. It's usable. If we don't have that kind of cache eviction, we can't realistically use the caching because it doesn't... Well, you can. [SPEAKER_02]: The caches would only last for one hour. no it's so the that timer if you so i'm letting you drive again here can you pull up what is it now it's called how long ago it's a helper yeah got it yeah and then you've got to find the i think it's in extras i don't know where i defined it okay this one yeah just doing a search i found it yeah so it is actually not an hour thing it is this big variable thing you could you could put the timestamp [SPEAKER_02]: into this function to generate the cache key though and and the the resulting string could be one of the cache keys yes and that would just so initially it would expire it after five seconds and then after 60 seconds but eventually it would only be expired every hour and then every year at the end yeah so you'd still get a lot of hits i think [SPEAKER_02]: if you were expiring the comments like that and they wouldn't all expire at the same time. [SPEAKER_02]: No. twitchtd didn't realize lobste.rs has no js, that's a surprise
[SPEAKER_02]: Unless you restarted the server and we have like a, the big old server, you know, what did I call it? [SPEAKER_02]: Deployment key. pushcx It's become a fun design constraint, really.
[SPEAKER_02]: My suspicion is that, like, there's barely any HTML here. twitchtd i'm all for it!
[SPEAKER_02]: It should take about… like, it should take less than five milliseconds to render one of these pages without caching. That is absolutely not what it takes. So if you look at the stream, here I have local dev, and let's find… I need a… older story that has okay here we go do we have comments right so even this story that only has three comments on it took 143 milliseconds to render in dev and it's 115 of that is spent in the view well i mean right a little more but interesting And I think it's, I shouldn't have a one plus N or prosopite would be screaming and throwing a 500. So what, you know what it's in, it's in the, the log that we duration, actually we spent, there we go. That's probably better to look at. We spent 38 milliseconds rendering that view. Clearly, this mini profiler is pulling in some amount of database time because those are getting lazily evaluated. Right. Right. And resizing the refresh file. Yeah, we should be able to get that down. The Vim terminal is a little bit imperfect. Five milliseconds or less. Yeah. So we're spending 43 now. -huh. [SPEAKER_02]: Well, I don't know how slow the Markdown rendering is, but assuming the Markdown rendering is already cached somehow, or we use caching on that, the rest of it. Yeah. And we cache it at the database layer, because... So this is stories, but there's the same thing on the comments table. We have a comment field. That's what the user entered. And then we have the rendered markdown because we only need to do that once. And then there is the small issue of if the markdown library changes behavior, it's a little strange to change what the user intended to type and may have seen in their preview. It's not the end of the world, and they're very deliberate around backwards compatibility. But in any case, effectively, we're caching at the database layer. But this does make the comments table twice as... Each row is twice as wide because of that. Actually, a little more, but... Yeah. [SPEAKER_02]: I just loaded the top most commented... [SPEAKER_02]: page on lobsters right now it's your tv setup one with 39 comments there is a 12.58 kilobytes of html on this page so if we extrapolate from micro benchmarks now this won't be entirely accurate then in theory we should be able to render this page 13 000 times a second that oh so you were looking at this specifically the tv setup discussion yes okay so [SPEAKER_02]: Watching… Or in less than a millisecond. Right. [SPEAKER_02]: So any overhead on top of that is likely coming from somewhere else. [SPEAKER_02]: Like, the raw HTML generation performance… Right. [SPEAKER_02]: …could be close to 1.4 gigabits a second, or rendering this whole page in 0.7 milliseconds. [SPEAKER_02]: You know, I honestly just removed 0.07 milliseconds. [SPEAKER_02]: I know we're extrapolating from micro benchmark, and so it's not going to be quite that, but we should be able to achieve something close to that. [SPEAKER_02]: Also, that was on my machine. [SPEAKER_02]: I don't know if it will be the same on yours. Yeah, I have a fairly beefy desktop. Yeah. So would you say, because the number I care about is what's it going to be in prod, would you say that we should benchmark against this number in the log or instrument something in the controller? Because I'm trying to say, yes, we'll be running on my machine instead of prod. But the idea is at least my machine will be consistent about these different strategies. [SPEAKER_02]: If we want to do a non-micro benchmark, like measure the actual meaningful difference. Every time I see a micro benchmark, it immediately devolves into whether or not it's representative. And we don't have to have an argument about whether it's representative. We can actually measure this actual page, and then I can put it in actual production and monitor it there. And if it's worse, it's not the end of the world because it's not like, oh, we just tanked e-commerce conversion. It's fine. Like if we start rendering pages at half speed, well then I'll revert. And it's not like we'll have to open an incident about it or refund people based on SLA. So I don't want a micro benchmark. I want a macro benchmark. [SPEAKER_02]: So let's include everything. [SPEAKER_02]: What we should probably do... [SPEAKER_02]: The easiest way to do this that I can think of off the top of my head is to have... To use something like... What's it called? [SPEAKER_02]: What's the new... [SPEAKER_02]: There's a new... Is Fractal Minds still watching? [SPEAKER_02]: What's the request per second benchmark tool? [SPEAKER_02]: Is it OHA? [SPEAKER_02]: It's OHA. I've never heard of it. [SPEAKER_02]: I can't switch to share my screen, but OHA allows you to... Can you spell the name? [SPEAKER_02]: OHA. Is that a gem? [SPEAKER_02]: If you look at OHA RPS or something like that. [SPEAKER_00]: This guy? [SPEAKER_02]: Yeah, that one. pushcx https://github.com/hatoo/oha
[SPEAKER_02]: So this is pretty good because if you run the Rails app in production mode and then set this on it, it will basically tell you how many requests per second you get. [SPEAKER_02]: And the output's really nice. I mean, it looks very sci-fi movie, sure. [SPEAKER_02]: Yeah. [SPEAKER_02]: So... [SPEAKER_02]: This is like, I mean, request per second is like throughput, not latency. [SPEAKER_02]: I think it does tell you what the average latency is and things like that. [SPEAKER_02]: So this might be a good way to do it is basically like get the application into these different states, run OHA on a page with many comments and see what the actual like throughput is like i would like to run it on a page with the median number of comments yeah because you know everything in social software is log normally distributed where like the average number of comments on a story is three and then somewhere way out on the end of the curve And it's always, every time I do look for outliers in our data, it's always the Microsoft acquires GitHub story. That one got like 300 comments. And so that's why I'm saying, you know, the median number of comments and we could do like a, a median number and then a big number. And I know from having read your discussion that. You think part of the performance cost that's not accurately represented in the micro benchmark is the time it takes to look up a partial by name. So yeah, I'm sympathetic and we can do like one benchmark at eight comments and one at 200. But I do want to be aware that that's a major contributor to where is our load, that we render pages very differently. Even just narrowing down to this one view, this story has an above-median number of comments. Ooh, and I just looked at the clock. You've got to be at time. [SPEAKER_02]: Yeah, I've probably got a couple minutes over. [SPEAKER_02]: But yeah, if... [SPEAKER_02]: If... [SPEAKER_02]: I think this is probably a pretty good way to benchmark this, just in a way that is as much as possible not micro-benchmarky, if that makes sense. [SPEAKER_02]: Yes, that's exactly what I'm trying to express. [SPEAKER_02]: Even if... [SPEAKER_02]: views let's say the views are taking 40 milliseconds right now and we can get that down to four milliseconds but overall the request is taking 500 milliseconds it's not a big it's not a very big difference and that's ultimately what matters it's like does it Hold on. Let me... [SPEAKER_02]: But the theory is that you should be able to render, like, if there was no database or anything involved, you had all the data up front ready to go straight into Flex, and you're not jumping back and forth between Flex and Action View, you should be able to render even this large page, this TV article, TV setup article, literally in less than a millisecond. All right, so one second. I'm grabbing a, there we go. So I'm logged into production, and here's real view times from our, that log that I just showed over on the Rails side. What was the other key here? We got view duration and DB. Oops.

02:04:47any, let's say duration. Well, duration's going to be the overall one, so let's put that first. [SPEAKER_02]: If you want this to be a really realistic benchmark, which I'm sure you do... I never get these damn jq commands correct right off. Does anybody remember this syntax? I want these to be one line. Yeah, it's printing them three lines at a time. That's impossible. Not yet, ask Claude. Can I just... Yeah, I definitely don't have Claude on prod.

02:05:34[SPEAKER_02]: Is it the pipe?

...43Do I? Is it... I need a... No, it's already a dash r. It just keeps reproducing my same... There's a... arh68 maybe [ .duration , .. ] ?
The from JSON is there because... Square brackets? [SPEAKER_00]: Nope. The from JSON is there because there's still some... legacy logging that just blow things up. Yeah. It's gotta be.

02:06:55[SPEAKER_02]: I would even try to benchmark it with Huma running with a sane setup for your laptop. [SPEAKER_02]: So like one process per core, maybe two or three threads, whatever your, something that reflects an equivalent to a reasonable production setup. That's a good point.

02:07:49I was going to throw in some quotes. That seems to matter. Thanks. That's exactly what I wanted. arh68 or maybe { duration, .. } . idk about the escaping I hate thinking about that bit
The answer to every shell problem is just add more backslashes.

02:08:07 pushcxFeels like I typoed something.

...20 joeldrapperJust trying to get real numbers here. Yeah.

...31And I guess we don't have a sometimes there's a, like, I start struggling with one of these nonsense tasks. And there's a expert in the house, but there is not right now. And I did throw it at Claude and it is imagining things.

...52 pushcxIt's not the big R, right? Yeah. There's something about the... [SPEAKER_02]: Didn't you just have it working up here?

02:09:06 joeldrapperAt the top of the screen? [SPEAKER_02]: Yeah. [SPEAKER_02]: But with that, you weren't actually... I was getting... So this was producing... multiple lines per row of input yeah and all right so like this one which of these is duration which of these is view which is if they're not printed on one line this is not usable by a human -huh and i don't know how to get the the thing to print them together so

02:10:09arh68 eh that's .db ... you could repeat .db, .db, .db to at least "line break" lol
[SPEAKER_02]: Can you just do just the view? [SPEAKER_02]: Because I guess that's the only one that we're going to be able to optimize here. Well, but I went down this rabbit hole because you had pointed out that If we optimize the view, we may not actually get a significant amount of perf. Yeah.

...42 pushcxThis says if you want our end.

...57 joeldrapperjson pipe two square brackets i'm gonna i'm gonna have to drop off now go get the kids from school all right and then we can talk about if we want to do the actual performance test and the thing that is most interesting to me is not so much the performance as long as it doesn't fall off a cliff it's do we end up with something that's more maintainable yeah but i do want to know that performance doesn't fall off a cliff because there is a lot of room between flex yeah runs roughly the same speed or flex runs in one millisecond and flex takes four times longer and like yeah i i would be fine with like a 10 hit but like a 4x hit is Not something I want. [SPEAKER_02]: I think if you switch to flex, most of your views would be rendering in sub five milliseconds. Oh, there we go. [SPEAKER_02]: There we go. [SPEAKER_02]: Okay. Like this is going to be a story view. The middle one. [SPEAKER_02]: Okay. [SPEAKER_02]: Right. So this is going to be like about our chat and then the rest of these are, you know, we can just. What did I call it? Path? [SPEAKER_02]: That one, sorry, the 0.3 seconds.

02:12:33 pushcxYeah. Yeah. Yeah.

...47 joeldrapperSo this one, like RSS, that must've been a cache hit, but these other ones. Yeah. [SPEAKER_02]: No, I think we can get it down from, like, 83 milliseconds to 3 milliseconds. [SPEAKER_02]: I'd be amazed if we can't. Here, let's... One more pipe. Alright, I don't want to leave your kids standing on the curb.

02:13:18 pushcxYeah. There we go, there's some stories. [SPEAKER_02]: Yeah, I have to go now, but I will happily come back on and we can finish this if you want.

...36 joeldrapperOkay. All right. Thanks for joining me. It's been a lot of fun. Yeah, it's been fun. All right, catch you later. Take care. Bye. All right. And now let's hang up on Joel so that he doesn't have access to my computer anymore. There we go. Turn off that volume. that was a fun exercise and then yeah here's some realistic performance numbers on the actual thing that we're looking at benchmarking which is these pages are fairly expensive to me because i don't know i i know i'm named after x86 assembly language but i'm like 100 milliseconds to render 10 kilobytes what especially on a modern computer, which our production server definitely is.

02:14:29 pushcxI wonder why this is so stuttery.

...38 joeldrapperThere must be some buffer happening here, because traffic does not actually come in in fits and starts like that, and the server doesn't install requests. So if we could actually And I understand Joel is very enthusiastic about these things and his library. But if we could actually cut these down, there is a meaningful amount of performance to improve between 20 and 80 milliseconds per hit. It seems to be the rough range. No, between 20 and... 20 and 40 or 50 is more typical. Although there are a couple of these, like this story must, oh, this one, it has nothing. I remember this title because I think I deleted this one as business news. So it doesn't render any comments. It just has to render the, this story is gone. All right. There's a very, I was very strongly reminded when he started talking about micro benchmarks about the conversation that went by on blue sky, whereas like that kind of kicked this whole thing off where we don't have to make a micro benchmark.

02:16:00 pushcxWe can just instrument fraud. All right.

...15 joeldrapperAll right. Well, it's been a second since I've done the, The bumper, this is lobsters office hours where we work on lobsters, which looks like this and like this in prod 39 stop uploading this. This guy is a chump. And what else is on my to-do list? So I've talked about that performance thing. arh68 i guess there's no like, grafana / prometheus metrics view already ?
I talked about flex a lot. No, we don't have any kind of APM. I try not to add more services to production. And performance, you know, for the time I spend on it, I sort of think of it like whittling, where it's something to do with my hands, and it's always kind of fun to optimize stuff. But I'm not actually especially rigorous around, are we really getting better in prod? And... I know there are APM services, but they haven't seemed worth springing for. And on a previous stream, I mentioned this and folks mentioned things like Grafana. I spent enough time frustrated by our deployment setup that that's the biggest reason I don't want to add more to it.

02:17:48 pushcxLet's

...52 joeldrapperLet me do pull request review before I do OSA. I don't think there's anything going on with new issues. Yes, so there's this open bug about, we cleaned up the black and white, or the CSS quite a bit. And,

02:18:21 pushcxWe've had some reports that, this one is so strange.

...31 joeldrapperSo apparently there's a validation issue and I kicked this around on the last stream and I thought it was dead, but I had to reopen it. And so there is one screenshot of, what this looks like, because I've heard it described that the colors are wrong. This line here should be kind of a faded gray and the upvote arrow, which is drawn with the border color is just missing. And I know this user is using a like two and a half, three year old version of Safari.

02:19:19 pushcxAnd then

...22 joeldrapperAlso, there are spattered reports that it doesn't work in Chrome, which is what Clemeth here, who has contributed to the site, has noted that it doesn't work in Qt Browser or something like that, which is Chromium under the hood, Qt Browser. But it might be an old version of Chromium. I don't know how closely they track the rendering engine. So we have not really had arh68 any IE6 users out there ? lol I wonder what those metrics look like, user agent &c
the info we need to report it. ARH, if you want, we can grep that out of prodlogs. So let's see. So I'm doing a quick look at the files here. Yeah. And I'm doing it this way off stream because I don't want to throw up somebody's IP address, even if it's just random middle-of-the-day browsing.

02:20:31 pushcxSo I'm going to just cut dash t dash dash f. All right.

...44 joeldrapperLet's clear that, and I can bring it on stream. So here's our prod access log. And we can cut it to user strings if you want, right? Isn't there a, what's the awk? Because awk will handle the quotes, right?

02:21:12 pushcxAnd I'm just doing a quick, someone is.

...23arh68 oh jeez i'm not an awk user
It's OK. We don't have to be an awk user.

...25 joeldrapperWe are Stack Overflow. There we go. And we don't need one. We only need seven. Great. So let's take the first, I don't know. Well, let's take the first 10 of these just to see. And then we'll pipe it to unique-c.

...57 pushcxType to sort RN.

...59 joeldrapperI get a warning out of awk, but okay. So out of the latest 10 hits, those were our top 10.

02:22:13 pushcxSo let's change it to 1,000. And I could grab the full production log.

...31 joeldrapperSo this is the way I end up editing or grabbing things out of log files quite a lot. And what I'm going to do in a second is replace this with cat. And there's this whole meme where people are like, oh, that's a useless use of cat because you could just remove the file name here and put it in cut. And then you don't have an extra process. But what they're missing is I have to build these things up iteratively. And I go back and forth between the final version and the edited and tweak them and tinker. And so it is not actually a useless use of cat. It is what makes the command line usable. All right, hold on. How do I find the edge of this?

02:23:25Okay. arh68 brb making cup of tea
So we're not even attempting to do anything smart about version numbers here. Funny to see Linux this high. There is, what is it, the Go access tool? I think I even have that installed. No, not locally.

...58 pushcxNo, I am not. Good question.

02:24:04 joeldrapperSo I am installing this on prod because I have it locally on my machine, but downloading a significant amount of logs to my machine is going to hit pretty hard.

...26 pushcxSo if I grab that.

...35I'm not on prod anymore.

02:25:01 joeldrapperSo this works on many formats of log and there's a whole thing around parsing the correct timestamp out.

...26 pushcxAnd I don't remember the

...35So I'm finding somebody has talked about what's the... Ah, I must have this config file locally.

...52And I do.

02:26:01 joeldrapperAll right, so let's... Grab this and copy this up to prod.

...25Guess it's been a hot minute since I actually ran this against the production log. I thought that was going to be straightforward. Browser breakdowns, but what's happening on line 10 of production log? Oh, it's not production log. That's what's wrong.

...55 pushcxThere we go. All right, so that's ripping along. And if it's doing 40,000 a second, I peeked at this log file. chamlis_ I didn't know awk could handle quoted fields like that - useful

02:27:09 joeldrapperThere's a thing with the log rotate where the access log only is getting rotated when I... Yeah, awk is the Swiss Army chainsaw.

...27All right, so we're getting there. We'll let that run for a minute, and then I will pull it up and make sure that the output report isn't going to print a bunch of people's IPs or anything, and then we'll have a browser breakdown. So I will let that run while I look at this PR.

...50I had hoped that the lightDart function would be basically universally supported. I'm not surprised that on our user base that someone running a very odd setup showed up pretty quickly. This bug has been really frustrating because it wasn't clear that we have valid HTML, or I'm sorry, valid CSS, and the validator is only just updating for these various formats. Literally in the process of us updating, pushcx https://github.com/lobsters/lob…
Overnight, I pasted a, hey, our CSS isn't valid, and the next day it validated because the validator was updated, and it turned out it was indeed valid. That's... I had just sort of assumed that W3C validator would have been updated at the same time that any spec was getting adopted, but that's a... Naive assumption rather than anything informed by paying close attention to the validator. So Clemeth has opened a pull request to hopefully fix this. So let's go check that out.

02:29:13 pushcxWhich is over in the pull request tab, Peter. All right.

...25 joeldrapperI can get these darn earbuds out of my ear before they run out of battery. That's nicer. I love having Bluetooth earbuds, but they do get tiring eventually.

...51Maybe in a year or so we can come back and restore the light dark, because what it does is actually quite nice.

02:30:00 pushcxAnd exactly the thing we need. So this diff... This is a diff against prod, right? Hmm. Great.

...24 joeldrapperThis was a recent addition by Andy C. We saw that on the last stream.

...34 pushcxSo now we're going to find all of this stuff down here, right?

...39 joeldrapperYeah. Let me know if the font size is too small. This is hard enough to read. Not having it wrap is harder on me.

...52 pushcxSo it's breaking these things out.

...59Why are these numbers changing? Okay.

02:31:10And then making a new style sheet called dark.

...14 joeldrapperAnd a new style sheet called light. So light was dupl... Everything is duplicated.

...20 pushcxThis is...

...31The syntax is SCSS or SAS, right?

...37 joeldrapperThat's not. And CSS doesn't have an import statement, right? I could be out of date here.

...49 pushcxCool, I'm out of date.

02:32:00 joeldrapperI didn't realize you could do that in CSS directly without a preprocessor. So it's not a duplication then, because these are built on the ones that are defined up here. Color BG. And these are... Base BG is then defined in the light tag. Oh man, there's no... clear flow of execution and i know that css is pretty declarative but for this kind of we're making a property and then reusing it later and then not having this be in reading order is a little rough so by default we're light and then we're dark and then if the user requests dark because you can also set it in your user settings page then we will say that. Otherwise, we pull in system. What is system?

02:33:02 pushcxDid you rename?

...10Ah, so system is what?

...14 joeldrapperOkay. So we have more HTTP requests, which I don't love, but we have HTTP2, so they're cheap.

...27 pushcxThis seems totally reasonable.

...37All right. Any other comments on this?

...46 joeldrapperI saw this in email on the, so we also had the bug reported as a story.

...56 pushcxwhich was a little distracting to not have it just in the bug tracker. What broke?

02:34:06Define method split for nil.

...23So if it called stylesheet link tag, this becomes omnil should be replaced with an or system. So this looks like it should work, but it has a bug in it.

...46Let's see this bug again.

...54Oh, they added a spec. Have I been?

02:35:30didn't see. No, he didn't ask. I gotta say they I don't know, Clemeth. Yeah, the spec already exists. So that finished, but let's take a second to

02:36:04Ah, so this is what's happening.

...07 joeldrapperThe way the style sheet, the light dark setting is implemented now is we add a class to the root. And then this test is checking that it's there. The test needs to change. And I think I probably skimmed over a comment from Clemeth where they said, I'm not touching the Ruby test.

...34 pushcxNo.

...38 joeldrapperI'll have to pull that down. Let's take a look to...

02:37:31Okay. So I ran this with Go Access's ignore crawler setting, and it occurred to me it's probably been a year or two since I've run Go Access, and it doesn't... Actually, I didn't scroll all the way down the page to make sure there's no personal info thrown up.

02:38:01 pushcxThere's a visitor host name or IP section. Oh, that's just both traffic, isn't it?

...16 joeldrapperNo, all right. I'm gonna just delete this section of the graph real quick.

...31 pushcxThrow that chart away.

...32 joeldrapperAnd anything else here? No. Great.

...39 pushcxAll right. So failed requests.

...46 joeldrapperI have not seen 35 failed requests in prod. I wonder if that's including 404s. So this is a couple of days of traffic. unique visitors per day is way higher than I expected. The last time I looked at a chart like this, it was like a quarter million a day. Now it's one and a half million.

02:39:13 pushcxAnd is that even a weekend?

...26 joeldrapperNo. Okay, we are seeing the usual week shape where this must be a partial day because Mondays are always highest traffic and then it declines over the week and falls off the cliff for the weekend. So I said everything on the web is log normal. Here is an example of that. People who are requesting the homepage and our most popular things are a huge number and then long tail of everything else. I really did not realize we were seeing... Oh, it's charting hits. That's what the left graph is. That's not useful. Visitors is more like 110, 150. All right, that's the neighborhood I was expecting. It's lower than I was expecting, actually. All right. But it's not ridiculous. And then some 404s hitting static stuff. Browsers did not get correctly parsed. The one thing we made this chart for...

02:40:34 pushcxI don't know what to say here.

...53 joeldrapperbecause I piped the full access log into go access so it had access to the IPs rather than pipe it through awk to strip them off.

02:41:05 pushcxThere may have been something about parsing that that I didn't know.

...23 joeldrapperAha, there is a parse option in my... There is an option in the config file I grabbed for timestamps. Enable a list of user agents by host. It is set to false. Let's set that to true. And was there anything in here where I could just turn off that IP chart so I don't have to pull it off stream and delete it again?

...53 pushcxIP anonymization. Can I just turn off the whole chart? This beats nothing. Don't need to resolve those.

02:42:25I'm doing geo IP sort of actively.

...30 joeldrapperI'm sick of geo IP everything right now.

...37 pushcxI'm going to just regenerate that.

...39 joeldrapperThat's going to run for another few minutes while I break the CSS. No, I was going to pull this down. That's what it was.

...55 pushcxNot breaking things, I'm fixing them.

02:43:47 joeldrapperNormally I would just ask Clemeth to update, but I don't know that they know. Ruby? And this one's been hanging out for days, and I just want the darn thing done.

02:44:17It's dash n and dash e. Now it's logged out.

...24 pushcxOr p. I never remember with RSpec. There we go.

...30 joeldrapperI just heard an interesting interview with the creator of RSpec. It was on the Code with JSON podcast.

...44 pushcxI listened to this last night. Yeah, this one.

...52 joeldrapperThis was a pretty interesting episode. pushcx https://www.codewithjason.com/p…
A lot of why Baker likes TDD so much, and I have never gotten into TDD. He made a pretty strong case for it. I see how he gets that value out of it. That is a weird portrait style where people are sort of transparent from below the chin so i can see the inside of his long hair very strange all right so we have this one spec what i want to check is that we have the one style sheet we expect

02:45:49 pushcxApplication and system. Okay.

02:46:19All right, so we have system. Great.

...30Good.

...55 joeldrapperSo that's the same.

02:47:02And what did it get called? Light and dark. All right, that's easy enough to grab.

...12 pushcxSo this one will have light. And I will probably, yeah.

...25This should pass.

...28 joeldrapperBut I want to add more assertions that the others are also not present. Good.

...38 pushcxAnd I want to add it to all of them, I suppose.

02:48:05 joeldrapperThis one, we should not have system. Really don't want to write a helper for this. I want to keep this very obvious. I'm okay with a little duplication. It's funny, I wrote the little root classes helper because I'm not okay with that kind of plumbing duplication. But making my assertions really obvious is something I'm comfortable with. Oh yeah, the tail end of that thought about the podcast was... I disagreed a lot with a lot of their opinions, but it was really that kind of productive disagreement because they're making really interesting arguments and bringing up things I hadn't considered. And so I have to be serious about it. So I think I can just push to this branch. I cannot just push to... I just created a... Is there a way on GitHub to delete the dumb branch that I just added? So I've got the... Yeah. Can I delete that from here? Oh, I don't want to have to use the Git UI because it's such weird syntax.

02:49:37 pushcxlook there's me remembering how to do it on some other git push dash d git push dash d origin good

02:50:08chamlis_ if you hit "view all branches" on github I think there's a button
Ah, well, thank you, Shamos.

...11 joeldrapperI want to see that UI because I did not feel confident about deleting that. So if I said view all, aha, there's my little trash can. So thank you. I will try to remember that for next time. And you know what? I must have done this before a year ago for that. So I will drop that and repaginate. Oh, that is... When I first started streaming, I took a run. Wow, it's been five months. I took a run at pagination. That's basically a dead branch, but it's not hurting anything. And I don't want to YOLO delete it while I'm juggling a couple other things in my head. All right. So let's switch to master. And then merge.

02:51:09 pushcxwith no force forward and this branch.

...23 joeldrapperAnd then even though I've followed GitHub's instructions, oh, I haven't pushed. I was going to say, sometimes it recognizes that I did the merge locally. Yeah, and then occasionally, I think it's like if I tweak code in the merge commit, it doesn't remember stuff. I don't remember what this is, but it's gotten me a couple of times because I leave pull requests hanging out.

02:52:19And let's grab this and fire off Ansible.

...25 pushcxOh, this.

...41 joeldrapperSomething weird has happened to my history. Oh, I know what it is. So. Ansible is written in Python, and I adopted the UV tool for installing Python, and I haven't run it in a stream shell before, so that's why it wasn't in my history. And I can say, so that it finishes in a reasonable amount of time, tag is lobsters. There we go. That's why I don't usually see this. The web and DB servers have different root passwords for sudo.

02:53:26Oh, and while that's running, GoAccess is finished. So let's fetch that. And then I am going to load that off stream so I can delete the chart that has IP addresses in it.

02:54:00 pushcxSo I can turn, oh, it actually has a little couple of check boxes. There we go. So let's bring that up.

...12 joeldrapperSo on this chart, I just unchecked the chart and the data. That's easier than going to the thing. This did not populate. I figured it would when I pulled in the browser stuff. There must be more config. And browsers, again, did not populate. Did I accidentally? No, I did not edit the local version.

...41All right, so here's I'm logged into prod and you can tell because it has the default in color scheme. Agent list is enabled.

...57My command did not.

02:55:06Did not disable the agent list. And in fact, I passed dash A, so it should have been doing it the whole time. All right, so it doesn't like something about our formatting and I don't want to keep dicking with it on stream. If somebody would really like to see this, let me just... So I'm grabbing a few lines of... The access log. And I'm going to paste them in the scratch so they'll be in the.

...48 pushcxAnd then I'm going to replace these IPS with.

02:56:05 joeldrapperRight. So here is in the access log. sample if anyone wants to offer a working go access why are you people reading this dumb tv setup post you can figure it out and send me your go access rc and i will generate that

...37And then the deployment. Don't need this test anymore. Deployment finished. So let's go and try to close the bug.

02:57:15 pushcxSomebody just asked about this in the chat room this morning.

...25 joeldrapperSo in the thread on lobsters, some people said, oh, I actually like the increased contrast.

...33 pushcxAnd never heard of prefers contrast.

...42 joeldrapperIt sounds like they probably added Oh, this is the GitHub doesn't know stacked PRs. And so until Clemeth pulls master into their branch, I have to see all of the diff repeated. chamlis_ if you're gonna stick around another 20 minutes or so and can stick that log sample in a pastebin I'll take a look

...57 pushcxOh, yeah, sure. Let me, come here.

02:58:10 joeldrapper20 minutes. Yeah, I don't mind running a minute long. Some kind of weird wrapping is happening here. Let me fix that real fast. So let's grab that and just grab this URL for you. Since the whole scratcher, just these lines should be just these lines. pushcx https://gist.github.com/pushcx/…
There you go.

...54chamlis_ cheers
Let me actually, let me grab, do this off stream because it's been ages since I actually looked at my go access RC. So I'm going to make sure that it doesn't have anything interesting in it. And if not, I will also just that just to give you a running start. So yeah, so offhand. pushcx time-format %H:%M:%S
Here's that time format, which is the instant thing it complains about.

02:59:27 pushcxAnd date format is the other part of it.

...44pushcx date-format %d/%b/%Y
This config file is 724 lines long. I must have. generated it somehow. It's also almost entirely commented out.

03:00:01Let's turn on ignore crawlers true so that I don't even have to remember that on the command line. And there is a ignore panels hosts that I can turn on.

...32Almost done. Here we go.

...41Let's reopen my gist. Can I just edit that file?

...56 joeldrapperHere we go. So now you also have My Go Access RC. I saw there's an entry up high called log format. It is either that the, well, no, we know the RC file isn't ignored because as soon as I installed it, the error went away. So it seems real likely that that log format is the smoking gun because it looks like it's listing all of the fields. And it's real possible that like, I have %U instead of %U or something else.

03:01:36 pushcxI can't guess. All right.

...42 joeldrapperSo there's the scratch again. And we were looking at this pull request. So we have a dark contrast.

03:02:22I'm wondering if this is give a mouse a cookie and people are going to say, you know, if we have light and dark and low contrast and high contrast, how about a solarized theme? And I don't know if I want to maintain these.

...56And even just touching the style sheets have prompted some bike shedding about colors.

03:03:13 pushcxYeah, I don't know.

...15 joeldrapperIt's more to maintain. In the discussion, there was three or four people, which is quite a lot of advocacy because, you know, most people won't mention it all. You know... Is it an accessibility feature?

...41 pushcxIf it is, I think that tips me over into, yes, we should. It must be, right?

...57chamlis_ I'm seeing a browser breakdown (only firefox in that example) with `goaccess --no-global-config --output scratch.html --log-format=COMBINED --anonymize-ip --anonymize-level=3 -f scratch`
All right, let me grab your.

03:04:34chamlis_ --no-global-config to hopefully pave over any differences there
Thanks for looking at this channel.

...51 joeldrapperSo we're going to get rid of, this is going to become a blog. And then we are going to output to that same file name. All right, it's thinking about it. We'll come back to it and see if we see it in a minute.

03:05:20I guess I'm just trusting you as opposed to trying to break it out to run on 10 lines of file. It's fine. I've got plenty to do for a minute.

03:07:53tommylbro Do you have discord?
Hey Tommy. No, I do not have a discord. we don't watch it. pushcx https://lobste.rs/chat
arh68 he's got a whole website lol
I don't keep an eye on it on stream, but lobsters has a IRC chat room mostly because we're old. pushcx https://push.cx/discord-vs-irc-…
chamlis_ speak for yourself!
And if you're curious why that's IRC instead of discord, there's like 2000 words about it. Oh, Shamless is young. Let that be noted then. That must be why she has the vigor of youth to take on weird command line things.

03:08:37 pushcxAll right. What else was I going to say?

...49 joeldrapperThat looks pretty good. I think that's all the points I wanted to hit. Anyways, welcome, Tommy Elbro. Tommy Lebro. Must be from Montreal. Oh, man. tommylbro Do you code with VSCode?
So I went to Montreal years ago with a friend. And the friend is the most straight-laced, button-down, kind of nerd. Like, he's one of those guys who, even if he's not going to the office... Like he could be lounging around his house in a button down shirt with a tie. That's just his whole style. So he's very proper, very polite. And we were in Montreal and there is a mountain in the middle of Montreal. Mount Royal. Let's see if Wikipedia finds it for us. Yes. And it's in Montreal or immediately west of downtown. And there's a nice walk you can take where you walk up the mountain, which is, I don't know, call it two miles. It wasn't terribly long. And then you can hop on a bike share from the city's program and ride down the mountain. It's a lot of fun. And they just have trucks that drive bikes up the mountain all day because people love doing this. And we were hiking, we were chatting about the city. And I was like, oh, oh, I get it. Mount Royal. That's what the city is named for. And he was like, what? No, it's a different name. Montreal. And I was like, no, no, no. Say it with a terrible French accent. Mont Royal. It's Montreal. Montreal. And he was like, no, no, it can't be that. It absolutely, that's too stupid. tommylbro g2g see you later just got told that it was time to go bbl
But as you might've spotted on Wikipedia, The city's name is derived from the mountain's name. Montreal is named for Mount Royal. You just have to say it with the hammiest... No, I code with Vim, not VS Code, Tommy. Here's Vim. So yes, in case you have ever wondered, Montreal is named for the mountain. You just have to say it in the hammiest accent you possibly can. It's so dumb and I love it. And if you haven't visited Montreal, it is not at all dumb. It is a charming city and maybe the most European city I have visited in North America.

03:11:37 pushcxThe French Quarter in New Orleans reminds me of Montreal that way.

...49 joeldrapperthat cool and i'm gonna re i'm gonna because i had that weird hassle with i told it not to generate the ips and it generated anyways all right it made the chart anyways but it remembered that i unchecked those things so shamless solved it thank you yeah you're already a vip so So what fun stuff do we see here we see the operating system chart is filled in which.

03:12:26 pushcxI don't believe this. chamlis_ thank goodness
Like I I flatly don't believe.

...33 joeldrapperWe have more than twice as like three times the number of visitors. on linux's windows that's just not plausible i know it's always the year of the linux desktop that cannot be correct your command did not include the ignore crawlers did it yeah that's what it is so these crawlers are either reporting correctly that they're probably running on linux or And I see the crawlers chart over here. I don't believe it. They must be in here. There are just not that many people browsing with Linux on the desktop. I know I am, but that's not plausible. And then let's scroll down here. So with the caveat that there's something here to not believe, and that OS setting is going to come from the user agent string, same as these browsers. So grain of salt.

03:13:43 pushcxCan I sort this by visitors? No, I cannot. chamlis_ I think you can click the column header to sort?

...55 joeldrapperSo you're saying 460 visitors in Chrome versus 74 in Firefox.

03:14:05 pushcxClick the column heading like this guy. I'm not sure which heading. Oh, here, right? Oh nice there we go that was all right fancy animation.

...24 joeldrapperI mean this is interesting, because it implies again this is crawler data, it has to be crawler data. Because it's firefox is such an outlier here. arh68 MSIE PogChamp they live
But it implies that firefox readers are loading what is this five times as many pages per. Oh yeah, here's your MSI-E. 2,500 visitors with MSI-E? I don't believe that number. That's gotta be bots with old-ass user agents. There's no way. The rest? Yeah, it's a pretty plausible breakdown that the web is dominated by Chrome, and then there's the also-rans. I just don't believe.

03:15:13 pushcxEdge. That's a little lower than I would expect, actually.

...19 joeldrapperBut maybe that's, it just looks relatively low because Windows is getting squished down by whatever this oddity is.

...29 pushcxI don't know.

...39 joeldrapperJust gonna keep running this all day, over and over. If anybody asks me why prod is running slow today, it's because we're going to just keep regenerating this report until I get numbers that I believe. You can tell I've worked in business because if the chart doesn't say what you want, you just torture the nerds and you force them to go back and make new charts until you get charts that you want, right? That's how business works. Today, I am a manager. Shamless, make me a reporting dashboard. See? Don't I sound like a manager? All right. chamlis_ aye aye
And I'm not going into these because I know what they are, and Rahul is working on that. Yeah, I've thought about, you know, thank you. I appreciate that, Minion. But I have joked... And I've thought about wiring up this to run on the regular or to dump these in somewhere publicly visible. But no one has ever asked for this kind of data or people only ask infrequently. And it doesn't really change any decisions I would make. If site traffic halved or doubled, yeah, I might do something different. But a lot of this is, what am I going to do with these charts? What decision am I going to make? I mean, a little bit, it could inform, do I want to support Qt Browser? Now, see, it's funny, but even so, since this is a hobby,

03:17:40 pushcxI would take a BSD bug report over an Edge bug report. Will it list all the others? Opera. Yandex. Brave. 14,000 hits and zero visitors.

03:18:03 joeldrapperThat's interesting. I wonder if it's because they're all getting four hundreds that they're not counting as visitors. That number is substantially higher than I thought, 14,000 hits. On the other hand, that's also the user agent for their spider because they don't respect protocols or standards. They honor them. Mostly I like working with lawyers, but sometimes I see their output and I'm like, that's just a fancy way of being misleading. All right. So there's pull request review. Let's add in, you know, we'll come back.

03:19:00 pushcxThis is way more fun than a USA stuff.

...09So. Who's excited for the Online Safety Act?

...21 joeldrapperSo I don't really, I don't have a personal update. Ofcom has continued ignoring my questions. The UK, the US embassy to the UK has not responded. I would assume that without diving into them, arh68 what's the countdown timer, like 40 days ?
or the merits thereof, because there are politics happening with the new administration change, well, the UK ambassador is busy getting replaced and getting new everything. arh68 our own lil brexit :(

...57 pushcxOh, the countdown timer? That's...

03:20:06What is it 40 days from now?

...11 joeldrapperSo that's three 11 good band, but 45 days ish. I am rounding. I am not thinking about time zones. We're going to call that 45 days. So we'll see. So I ran into this blog post online. pushcx https://heatherburns.tech/2022/…
and I was floored. This blog post, which is from 2022, I don't know which one that is because I don't recognize this name, but it's ridiculous. This post is not outdated. This post is the most accurate thing and most useful thing I have read about the Online Safety Act And it's, what, 36 months later? So this part, it opens with this anecdote that she went to Westminster, which is a palace because the UK has a king. It feels so old-timey. I'm aware they're mostly decorative at this point, but every time I see something like Royal Ascent, Throws me off a little. And she mentions that, you know, we're going to go work through these things and the UK's determination to regulate the internet around Facebook. Yes, that is the big problem with the OSA. She nailed it right there, like a hundred words in. And then, or we would have if they had bothered to show up. Only one did. That's it. That's in a nutshell. Like, Ofcom doesn't show up. The MPs don't show up. That is exactly the gripe. chamlis_ royal ascent is climbing the mountain in montreal, no?
It sets the exact right tone. And then I mean, anyone working in a company or project, which will fall in scope of the bill, which is paid software, community, personal hobby, Royal Ascent. Yes. So that is the activity I'm talking about. I don't remember if it has a clever nickname. Royal Assent. Yeah. I can't do a French accent. was part of what made the joke so bad and my friend so offended was i can't actually do a hammy so like again this don't shoot the messenger i've gotten messages from people who are like you're being hysterical why are you well there have been gendered insults in there beyond hysterical but the point of this this post is so good she instantly got it that I am writing for not big tech because they have their compliance departments legal teams and policy specialists, this is. A chore for someone at Facebook scale, this is. Oh, the legal compliance team spent an extra $3 million this year we don't care. But if you are not big tech you don't have a spare $3 million. You just get killed by it. This is so cute. Like, oh, it's already 230 pages. Well, the final version is 350. And then the Ofcom guidance is another 2,800 pages. So yes, the post is outdated, but it's much, much worse. And so like her summary, are you in scope? Yeah. If you can talk online, yep. And she even immediately nailed the, doesn't mean that you have an account. We had to re-derive this. We had to re-figure this out and she had it all 36 months ago. I wish I had seen this post first. If you took this post, just going off that page count, if you took this post and you assumed that everything was 15% worse, 15% longer, 15% dumber, you would absolutely have the best possible quick understanding of the OSA.

03:24:41 pushcxAnd I won't beat it to death and read through the whole post, but this is it. So I loved this post.

...52 joeldrapperIt's sad and awful, but this post is the best introduction to the Online Safety Act. So speaking of good introductions to the Online Safety Act, if you wanted a shorter and more exciting one, this is from Woof Group, which is a Mastodon instance administered by Kyle Kingsbury. Afer, A-P-H-Y-R. And I am not going to bring up his Mastodon instance on stream because when I did that to go follow his admin account, I saw a lot of porn. And this is an excellent set of questions. and ofcom will never if he even gets an email that acknowledges these questions i will be surprised because ofcom doesn't want to admit how terrible this act is because it would be politically embarrassing and ofcom is run by a political appointee so realistically they are going to ignore these questions Because these questions are... Hey, the OSA is incredibly vague. It talks about things like significant number, and it doesn't define significant number, and you haven't bothered to do that. And the answer is... I mean, Ofcom's answer is fuck you. They don't want to lock themselves into something that would actually be useful for sites to comply with. Ofcom... I'm saying this a lot more candidly now because I was giving them a lot of benefit of the doubt. I don't doubt anymore. They want to destroy sites and are happy to destroy small communities. I will notice that... Ifir has made the mistake of thinking the word users means users. The situation is much worse than he expects and has written here because it means visitors.

03:27:39But the note about we can't actually measure how many there are is absolutely true because the Online Safety Act was written without an understanding of how the web works. Here is a point that I wonder if he picked it up from one of my comments, because I mentioned explicitly, lobsters does not have a target market because we are not a business. I don't market lobsters like boy, I wish I had more users from Wales or Shrewsbury or hedge on Battersea by the forest. I don't know, I can't even make up silly UK names.

03:28:26There is lots of moral judgment smuggled into the OSA where it implies that pornography, the existence of pornography, is a harm to people. It uses this phrase, extreme pornography, and says the definition is given their ordinary meanings. I had never heard this phrase, extreme pornography. Maybe I don't hang out in pornography circles enough. I assume if the Lords and members of Parliament are big on extreme pornography, they have an ordinary meaning for it. I sure don't. arh68 they seem to be quite the experts SeriousSloth
So Kyle asks, here is a list of sex acts. Please determine which sex acts are considered extreme. I wonder if, so I don't actually know how, like I have joked about streaming from a hot tub. I don't know how heavy handed Twitch is. If I read a list of weird porn stuff, do I get censored? I don't know. But I really... Ofcom clearly is not going to provide him a list of what is the UK's list of extreme versus acceptable pornographic acts. arh68 all that's like normal gaming trash talk haha
Exactly which sexual positions are permitted. All of this is normal trash talk. Okay. In any case, I mean, I'm showing it on stream. If I'm going to get whacked, I'm going to get whacked.

03:30:05 pushcxBut This certainly affects Twitch, right?

...17chamlis_ do you know if Libera have looked into this?
How are we to interpret reports of...

...20 joeldrapperI have no idea if Libera have looked into this. You would have to inquire yourself, Shamless. I don't... I think they're based in Sweden? I don't know why I think that. Yeah, I have no idea where Libera is based. I remember Freenode was in the UK before... Let's not go down that rabbit hole.

...52 pushcxBefore it was murdered.

...59So then, and I kind of love Afer for this, Frici they are a bit distributed as a network but the org itself is a swedish nonprofit yeah

03:31:07 joeldrapperso there's this graph of dear authors of the osa and ofcom you don't know how the web works let me draw you a chart of different things about how the web works they're distributed as a network but the org itself is swedish okay so i'm i'm remembering correctly every once in a while i get to be correct thank you freaky and then there is another one that is about You don't understand how the web actually works and how data is shared and cached. All of this stuff gets more complicated when you realize most services have something like Akamai or Cloudflare in front of them. Oh, yes, Kyle actually makes this point even better with, what do you think about online services like AWS 3? AWS S3 would be, by my reading of the OSA, under its jurisdiction because users can post content that other users can see. God, can you imagine? I wonder what Amazon's thoughts are. Like, oh, wait a minute. Anything viewed on S3, we need to make a list of acceptable extreme pornography acts?

03:32:34And then this one, this one is just sad. So it is not that the question is sad, it is that the question has to be act is sad. Ofcom sort of has levels of formality in their responses where through like friends of a friend, I have heard these kind of vague assurances that, oh, well, we're gonna scale our responses to The amount of scale of the service or harm of the service so clearly we won't drop an $18 million fine on the little guys. But they will only say that off the record third hand anonymously they won't actually write it down in their guidance in print.

03:33:30 pushcxAnd I feel like.

...34 joeldrapperThere's a famous saying in American politics that is, at long last, sir, have you no sense of decency? And this is, you know, Ofcom doesn't have a sense of decency, or they would respond to this question and say, of course, we're going to scale appropriately. Of course, we are going to tell random guy who runs a random small website, please do this thing before we come in with the criminal penalties. But don't and then hey here's my question my big burning question of how do you intend to force the osa against the us corporations and citizens i could just show you the answer they gave me which is the osa says that we have power over the entire world and because the osa says it has power it has power i mean i could write that on a napkin and have exactly the same amount of authority over the entire world and every web service in the world, but Ofcom feels like they're special. We don't get to move on just yet. There's one last OSA bullshit thing. So anyways, this is an excellent criticism formed with a question mark at the end. They won't answer any of this. If they even respond to his email, pushcx https://www.thetimes.com/articl…
won't give him a straight answer at best he will get some statements so last link and i know i try and usually be fairly non-judgmental and upbeat but i'm burned out on the osa like the fact that they're not acting in good faith and not trying to collaborate and are actively seeking the destruction of some of the most wonderful things in the world Eh, kind of a downer. We'll move on in a minute. The last thing worth highlighting here is Labor, which is the name of the party currently in power, has, as a public statement said, ministers are concerned that making significant changes could delay the implementation, so they will review it when it's in force. And Or was it? I thought there was something that said, either I'm misremembering it or it was a different page, but they basically said, we're going to leave it up for a year and then we're going to survey the devastation. We're going to let it fuck everything up and kill all of these communities. And then afterwards, we're going to survey them. So that's pretty much end game. Like I still have a contact out to a US Senate office that does some online policy stuff. Maybe they can nudge international diplomacy along better, but this is pretty much a commitment from the current government that needs to act to prevent the destruction making a i know what a public commitment from a politician is worth especially without someone's name specifically attached to it but this thing is not getting a lot of attention so and there's all of this nonsense of social media we're going to blame that for political unrest so the uk i will miss you in 45 days

03:37:33 pushcxYeah, see, even with ignore crawlers, it put to Linux first and I still don't believe it.

...52 joeldrapperI also kind of don't believe this chart. chamlis_ it still has a crawlers category
arh68 this one feels more certain than the tiktok ban, but ... idk it's still like getting run over by a glacier
There must be some crawler it's not recognizing.

03:38:00 pushcxIt still has a crawlers category. Oh, did you see that up here?

...10Did I? You know what?

...14 joeldrapperThat's because I didn't fetch a fresh copy of it. I rendered it, but I didn't. All right. So this says 49k crawlers and 417 Linux. Let's reload. Now it has more on Linux and it got rid of the crawlers. Yeah. Okay. So small changes. So can I reorganize?

...48 pushcxNo, I would like to put these two things next to each other.

03:39:02 joeldrapperSo my best guess for the bits of these charts that I cannot believe is that there is some crawler. It's not recognizing that it's a substantial percentage of all site traffic that identifies as Firefox on Linux. That's the only way because the rest of the numbers are, you know, I look at their relative scales. chamlis_ can you view less aggregated user agents?

...32 pushcxCan I view less aggregated? Well, it's got a pagination list if that's what you're looking for. chamlis_ maybe there's a real outlier ancient firefox version that a crawler is uring

...40 joeldrapperOr are you saying you want like a cross tab where it's like user agent by platform? chamlis_ using*
Real ancient. Yeah. That's really common that they have old versions. I don't see anything about browser version in here, right? No. You know, this is enough of an outlier. I'm going to just. I'm going to grab off screen, but.

03:40:25You know, I shouldn't have closed the tab where I had that. unique because we saw Firefox for Linux in there. So I already had it broken out. Yeah, it's not in my scroll back anymore. So I'm doing a quick grep for Linux dot star Firefox in lobsters access log. Let's pipe that to head 20. yes so i immediately found oh it's the enumerator it's the enumerator all right where's my awk let me let me show you why this why i immediately recognize this let's put these two command lines together

03:41:30 pushcxWhy is arc not stripping off?

...55 joeldrapperI'm not getting the arc right. It's messed up somehow.

03:42:09Let's just, come here, let me do it in Vim. I can just edit it by hand faster than I can repair the awk invocation. So let me select these lines.

...24 pushcxYeah. And make a new buffer. Paste.

...43 joeldrapperAnd then you will see why I'm groaning in a moment.

...54chamlis_ incidentally, perhaps you could use your geoip database to exclude non-residential IPs too
And there's some word wrapping. So if you see an extra space... Actually, I can just set... If you see an extra space in some of these user agents, please ignore it. It's me word wrapping.

03:43:18So Shamless, you are dead on about maybe it's a real ancient outlier Firefox version that a crawler is using. And not only do I recognize that, I recognize the crawler. So I figured if it's such a significant percentage of site traffic, if I just grep for Firefox on Linux, it's going to be in the top 20, right? And it's literally the first line. as soon as i saw a couple i recognized it so this user agent is a bot that in my head i call the enumerator all it does is enumerate user accounts all day long this is not so there are a couple of these services online that are like put in a username and we will be creepers and go find if it exists at 200 social networks it is not those because the IP address, which I've deleted this, is almost always, it's either Tor or a VPN. And this is some attacker who has enormous dumps with tons of usernames and they occasionally have garbage. So here's the enumerator. Atlas, Concord, Corso, something like space, Italia, they have garbage in their database. The enumerator... Oh, this is even an older browser, Firefox 24. Firefox 6. Oh, this is the... It's 64. This is that extra space in the user agent because I had wrapping issues pasting this. But there is an aggressive, aggressive enumerator. And it's some kind of very large system because when I put in... chamlis_ the 64s are HTTP2, 24 is HTTP1.1
rack attack to rate limit hits it tuned to the rate limit so it stopped getting 400s automatically but even though it's been what like a year since i updated to the tilde urls it has not and so you also see the enumerators sure You also see the enumerator always asks with the old style of URL, which I would guess means that at one point, either the bot made a look at the site and realized what our URL structure was for accounts or a human made that judgment. And I could believe a bot did that automatically. And then they hard coded that. And then they have the logic so that if they do a get and they get a 301, they do follow it. But that's it, that's the enumerator. They are a significant percent of site traffic. So if I took that log file and I filtered out anything that said Firefox and didn't say slash one, I would drop all their traffic on the floor and a significant percent of site traffic would drop out of that chart.

03:46:38 pushcxCome here, Bim, what state are you in?

...59So, no, not that one.

03:47:13 joeldrapperHelp if I could type. I'm running long, so, you know, I'm falling off the clip. cat-lobstrack-rep-v firefox slash 64. And what was the other one? Firefox 24. I don't remember which syntax grep supports. I think this is fine.

...53 pushcxSo I'm going to where that finishes.

03:48:05 joeldrapperBecause I'm old, every time I... In my head, every time I think of the enumerator and I think about them for a minute... chamlis_ it was something like 8M before, watch it be 2M with the filtering
I've done a couple of things to try and block them, but... Visiting... Yeah, that's why I'm copying the file to another file name, is I want to be able to page back and forth. Every time I think about them, there's this... pushcx https://www.youtube.com/watch?v…
ymeynot45 Aww
really cheesy 90s song that I am not going to punish the actual stream with but this is what passed for music when dinosaurs roamed the earth and I have constantly had this you could play it yourself right this is what passes for for culture or what did I call it with enumerator

03:49:04And then we'll just open the new one in a second when it's done. If anybody wants to see anything else while we're waiting for this report to be generated, or if you want to place your bets, what's the over under on how many more?

...27God, if I could make Twitch give me an API key, you insecure fools. ymeynot45 WTF was that song, LOL?
Because then I could make a betting pool and we could all place our bets on how many hits are going to go away. The enumerator is huge. It is a significant percentage of site traffic. The hits per day is 1.4 million. I wouldn't be shocked if that dropped by a million. I'm not joking. It is aggressive. It hits the site a couple of times a second. It comes out of every exit node on Tor. There's some code in just to... Yeah, like I said. So I think that song was a novelty because it was some of the earliest, I guess you'd call it hip hop or EDM, and just sounding so different than other music is why it was popular, but very repetitive. So if you've ever seen Strong Bad, they make a fake techno song where he chants, the system is down and the terribleness and repetitiveness is a direct reference to that song, the percolator.

03:51:05 pushcxSo config, where did I put it? Is it an initializer? Yeah.

...16 joeldrapperSo we have rack attack for rate limiting and You can see me here on lines 25 and 29 talking about the user enumerator is here. ymeynot45 I can't believe how often I have heard about Strong Bad in the last 6 months after hearing nothing about it in the 15 years prior.
And it's painful because Strong Bad woke up and started making new stuff a couple of months ago. I think that's probably why it's coming up. Or maybe you just started volunteering at the old folks home.

...51 pushcxAnd yeah.

...59 joeldrapperSo there are so many links out there on the web to old user pages, but it's so characteristic of the enumerator that I get tempted to make this really strict. I should probably just put the user agent in here. Yeah. So there was a, This SEM rush thing, can you believe an SEO tool was not a well-behaved netizen?

03:52:34The hassle with a lot of these filters is the enumerator does get and if I say you can only do so many gets to this, or I will think you're the enumerator and drop you on the floor, is someone else could make a page, some random blog post, and in it, they could put 10 one by one pixel image tags that would load slash URLs, and then every visitor to that blog post would be misrecognized as the enumerator. But if I put in the user agent, I can be aggressive about it. chamlis_ would you ever consider trying to give them fake data? like pretending every username exists
So like when log4j was going around, I mean, mostly I would like them to fuck off. If I give them nonsense, I don't know what they're doing with it. They can't be actually paying close attention to lobsters because they never noticed the URL change. so they haven't noticed hey on this one site a hundred percent of the time we get a 301 redirect and they're not immediately looking at their thing and so in case anybody doesn't get it what happens is sites get hacked they lose their database In the database is things like passwords if they can crack some of those passwords now they know a valid email and password. And at some point, they can come along and try those everywhere, and so the archetypal thing is like say they hacked lobsters. Then they would have oh push CX used this password because maybe it's a common enough one that it's easy to crack. Now let's go log into EveryBank and PayPal and Twitter with pushcx and hunter2 as the password, now that we know that. So the enumerator is an attacker who's sophisticated enough to realize in the future there will be more hacks and dumps. So rather than wait to get the dump, I can enumerate all of these accounts now, and then when I go to make an attack and steal accounts, Number one, I recognize more usernames because I've checked lobsters. But then second, I can immediately pop both accounts because I have the username and I know very well. And I don't have to show my hand because it's, you know, it's iffy if somebody logs in from one IP to 200 different accounts or attempts to log into 5,000 accounts. So...

03:55:28That's what's happening there. Let's hack that out, actually. Let's put that in the scratch.

...40chamlis_ they really can't have tried hard, because every user is on the invite tree page, right?
Yep, you got it, Shamus. That's why one of the reasons I know they're not specifically targeting us because you just would never do it this way. Did I not reload? Just looks unchanged. So like 27, a full day of data, 1.72, 1.75. Boy, my estimation was really off. There's only 20,000. I don't believe that. Cause I see like one a second. Where did that log go? chamlis_ is the grep working?
Ooh, not that one. What time am I at? That's a good question. The grep must be wrong.

03:56:35 pushcxI should have tested it separate. Here, let me do it this way because I have to...

03:57:04 joeldrapperno the grip worked so i looked at the top 20 lines of the log file and lines are top 10 lines and lines 9 and 10 are the enumerator and with the grep those lines are missing and replaced by innocuous traffic

...53just ran scp again off screen nope all right so i don't immediately know what's wrong there that's a little bit of a puzzle match the fonts yeah come on a hundred thousand a day well that's maybe not so wrong i said that there's like one a second and there are 186, wait, what is it? 186,000 seconds in a day? I never remember this. arh68 86400
86,400, okay. Yeah, it's one of those like, as soon as I see the number, I'm like, right, right. So if I have felt there's about one a second, No, maybe I'm a little off, but it's in the right neighborhood. Maybe there's one every one and a half seconds.

03:59:04 pushcxWhy are these charts so different? Because I resized the browser and it stayed put. That's interesting.

...17 joeldrapperOnly the last day is moving. enumerator is active all day every day maybe the enumerator is rolling its user agents and i'm not recognizing but that does explain this yeah see so if linux was where it more likely is probably somewhere below mac os we would have 400,000 fewer hits, right? Eh, 380,000 fewer, right?

04:00:00 pushcxI'm just kind of eyeballing.

...09All right.

...59 joeldrapperDid we? No, I have it off screen.

04:01:19Going to grab the enumerators, Firefox versions real quick. Was it Firefox 64?

...37 pushcxYeah.

...46 joeldrapperSo there's there's shameless is very good question of do we want to drop their traffic on the floor, which might prompt them to get a little smarter? Or do we want to give them bullshit?

04:02:17See rails is so coupled to the database and active record objects are so big.

...28 pushcxSo what I'm wondering is, so this does the, chamlis_ send them a 302 to /users hehe

...46 joeldrapperload showing user and so here we find the user and so like what's tempting is and i put a string on the left because i don't remember the immediate path but it's not the I want to make so if you are the enumerator right because if you are writing the enumerator you probably check that the username you're checking for appears on the page then this looks like you know if what call it one percent if one percent chance say yes otherwise showing user equals nil the hassle here is if i call user.new well you know i got local dev running let's go let's just see if it blows up So let's say if true, place with 1% chance, and let's ask Stack Overflow via Google, request user agent. Which I'm also going to change to say true. So I can just see this in my local browser. So let's go to slash use slash enumerator. Jerk. We might have somebody named enumerator. Yeah. So I'm getting up on you're in prod buddy.

04:05:15Why am I cycling color schemes? That was disconcerting.

...26So I just said if true, right? Oh, I didn't save the controller. Then it's... Standard RB reorganized that a little. Oh, this needs to be a else.

...56OK. Yeah, so now we're just going to blow up for things like when was the user created? And then I just have to. chase blowup after blowup until I have everything the view expects filled in as not nil.

04:06:48Right. It always says just now I thought I could just say, Oh, I want, all right.

...59 pushcxGrand no brand is in the wrong place.

04:07:11 joeldrapperWhat's the reds even has a nice thing for it, right? Zero 99 random.

...24I guess I was right the first time.

...33 pushcxFor an instance of float, that's application helper blowing up.

...48 joeldrapperUsers show line 40. That's when they were created.

04:08:30How am I hitting this label from here? Oh, cause I started pasting stuff and it ended up in the buffer and I didn't clear it properly.

...38 pushcxSo we get a user with a.

...59 joeldrapperMatt isn't working oh because I subtracted so it's I could have just said that that's what it is. cool so.

04:09:11 pushcxthat's enough.

...34Let's do the other cool number.

...44 joeldrapperI'm not going to sit here and reload this 100 times, but... Oh, and I won't see it at all because I don't have that user agent. There we go. So...

04:10:22The enumerator is... I occasionally get tempted to block Tor when dealing with abuse. This is one of those of like, why can't we have nice things? Because there are jerks like this that use Tor this way.

...51Because it's otherwise kind of sophisticated with using multiple IPs, which means if it's using multiple Tor exit nodes, it must be constantly refreshing that and trying to spread out and then making sure it's got different exit nodes for its different hits. Because it never uses the same IP a bunch of times in a row to hit a rate limit. Some care went into this. Kind of strange that it has such a dumb user agent.

04:11:41And I know when I touch the user show, I'm going to blow up in strange ways. Oh well. I just didn't want to have to have a user persisted to the database. I'm trying to lighten the load here. What's different? Let's just drop that.

04:12:12How's that look Shamless? Is that what you were thinking of? Let's fib to them. chamlis_ yep looks great to me

...42 pushcxWhy am I not on master?

04:13:03OK, there we go. That's the graph I expected.

...08 joeldrapperYou know, I did it again. Because I'm getting tired. I didn't run the build. I know I can make it a pre-push hook. There's a bunch of moving parts, and when I want to push stuff fast, I really don't want that getting in my way. There isn't a nice bypass. And if I had a nice bypass command line, I would probably just end up pulling that out of shell history, and I would be back in the same boat.

...38 pushcxlet's hope i didn't break master in prod again being sloppy oh yep i did what's brakeman mad about oh that's i'm allowed to do that why did that come up One of those is interesting.

04:14:15 joeldrapperOh, because of the last couple hours, and then... So, breakman is really quite clever, and it looks at the place that a variable came from. And so we had already a warning about this, which is fine, because we know this is... there's some metadata around whether the comments created in the last couple of months. And so we're hitting that query and Brakeman is noticing that the model is coming from a different code path. And it's either, it's not one of these test models, it's just because The structure of it changed where the actual model that comes from the user.findby now has a different structure around it. So that's why there's two of them. All right.

04:15:40arh68 well i enjoyed the dive into metrics but i gotta go. have a good weekend yall HahaGingercat
Have a good one, ARH.

...48chamlis_ see ya
This is the one I want to grab the one of these hashes is going to already be present. I just want to reuse that note.

04:16:09And then this one's the new one.

...36You know, it's coming from linkified about. Oh, so this one is actually, this one's also going to be, this other obsolete entry is going to be here. It's not mad about that you're passing params into a model. It's on the output. We have markdown. So the node's going to be, yeah, rendered markdown. So both of them, it's noticing that

04:17:09Both of them, it's noticing that the way the user is loaded changed.

...26Cool. Let's make the enumerator sad by lying to it. This is one of those things where There is a thing where if you look at your prod logs, you will see weird bots and bad behavior and abuse. And you can spend an infinite amount of time trying to run that stuff down because there is an infinite amount of it online. And I've already spent a bunch of time on the enumerator. But lying to them is really satisfying because they've been around for

04:18:09 pushcxAt least a couple of years at this point, like two or three.

...13 joeldrapperWhich says a lot about how many sites get popped that they just test tens of thousands of usernames a day and they never run out. I haven't tried to see if they're looping around and coming back to old ones. Yeah.

...40All right. That's all I got. We did extra stuff. We ran long. We had fun. ymeynot45 It was fun
We'll see if Joel hacks up this benchmark. chamlis_ thanks for the stream, as ever
See, it's funny because I would like Joel to set up a benchmarking tool But he keeps wanting to make it really narrow and a micro benchmark. And that is the exact opposite of what I want. So maybe I'm going to have to do it. We'll see how that goes. So next stream is, as usual, next Monday. Let me double check my calendar. I will be here and be square. I'll say be there or be square. It's like the percolator. All right. Well, take care. Enjoy good techno. I appreciate the questions and commentary. I will see you all in a few days. Later.