I should be doing this in production

20Good morning. It is Thursday. And this is the Lobster's Office Hours. I didn't do my usual start the stream and give people a second. Just kind of jumping right into it. I've got the nice stream manager up. Got my notes up. Folks will trickle in as they see things on. Excuse me. I'll have to get up for water in a minute. As they... As we start the day here with... Everything. You'll see the notifications on Blue Sky and Mastodon and Twitch and Lube and Clang and all of the... Everything. God, it's funny. This is like... four lines and I know I'm forgetting something that basically to say almost nothing, unfortunately. So good morning. Please do sound off. You can say hi. This is the office hours for lobsters that I do twice a week. Lobsters is this site. And if you have any questions about how the site works or how the code base works, or how the community works, or a moderation decision I made, or you want to run a query against the production database. We can do those things. Right here, right now, as we said in the 90s. And then otherwise, I work on the code base and do other admin-y things. PR review is the top of my list. So let's jump in. I don't think the mic picked it up, but that's the boss. Occasionally. My cat Raz will grace us with his presence as supervisor. We'll see if that comes along. Hey, buddy. Yeah, you're adoring fans away. See if he works it out. We made it to the last stream. That was nice. That's maybe only the third time he's made it on stream. And I got some of that insulating foam in the gap in the window. So it's actually warmer in the office and hopefully warmer by his favorite perch. So we'll see. Cats are pretty great. Alrighty. So... The first thing I wanted to talk about in pull requests was that I merged something. Yay. This one, most recently updated, which should be the default. I should just bookmark this. And maybe Firefox will auto-complete it for me.

03:38So there have been two. Continuing the trend of me not being able to predict what people want to volunteer to work on, two bits of plumbing got merged.

04:02Throwing these in the notes, if you can hear my keyboard rattling.

...13so this one by rahul is for his other full request where he split out some mod stuff or no he's working on the active job back end and we wanted that to be available to mods and you have to kind of mix into the controller. Rails calls it mission control for the job. So that's his pull request. 14.15. And you know, I see that the build is green on that, so maybe I can merge it. So this one, we ended up getting into a big tutorial about what's going on with Brakeman. That's what prompted the... other issue where I want to monkey patch Brakeman. So let's throw a link in here. Great.

05:42So that got merged. It's plumbing. I'm very happy to have it. I wanted to pull that down. So I'm not sure I have it locally. Yeah, didn't yet have it. I was tinkering. What's my state? I was tinkering with this stuff last night. And that's all code that can get thrown away. We'll come back to that. Before I shoot myself in the foot, let's turn that cache back off. Kind of kills me that there's a command to toggle the cache, but there isn't a command to just turn it off. Yeah. I like idempotent commands. So the concern, where did it land? It's indicatable. I don't super love the Rails practice of making concerns adjectival. I think you end up with silly fake words like this, but it wasn't worth fighting about. So on these. I looked at this flow and I realized so the first rule of off code is if it works, don't touch it. Right. You don't want to introduce issues when you think you're fixing something else. But I looked at this and I really don't like this flow where there's an if and then a nested if and it's not clear where. The if not user goes. So what's happening is this if implicitly returns nil for the else branch because the else branch isn't present. And then the last statement in a Ruby function is implicitly returned. So then that nil gets returned and that holds the before action chain. And I don't like that. And I think they just don't even get this message. So I wanted to throw on a test, and I didn't want to make Rahul do it. So let's go to look in the tests. Specs.

08:26Yeah. It's got to be a spec. Let's go to the login spec.

...39So this is a... Let's start another one. I really do want to just test mod stuff.

...59I'm just double checking real quick that there isn't another spec. No, there isn't. Funny that this auth helper didn't get a good spec. That's not great for key functionality.

09:27So I want to have a user. I want to have a moderator. I want to describe a mod path.

...53So that's just slash mod. And that goes to mod index. Great.

10:29It doesn't load for a non-mod user. It doesn't load for a visitor. I think that's pretty good. So let's go ahead and say if we sign in as user,

...58and we do a get to slash mod we expect the response well we don't actually say bad request so we will just say we expect that we get a redirect and we expect that there is something in the flash error and i don't really care what that is yeah so i could just put that in here

11:35Let's see if that wants to run. Let's clear this stuff out. Unspecified method email for an instance of symbol. I passed a symbol and not a model.

12:14So this test is basically the same without the sign in. And this test is the same, except we say moderator. And doing this, I'll have the confidence to refactor. Why do I have two failures? Oh, because this one should not fail. Two, what is it called? To be successful. Be nil. Be billed. To be a brother-in-law. That slipped pretty well, so the typo count shouldn't be wild. Alright, so that's better, but...

13:18I get the redirect.

...29Oh, that's interesting. So it's actually a different layer that's firing.

...45And I actually don't. I actually do care a little bit which message they get.

14:28All right, so then this should run. Good. So that's what's happening here. That's why this is firing. So now I can get rid of that. Just say this, and now there is one fewer branch. And if this works, which I expect it to, because that's pretty minor refactoring, And the logged in user should catch it. Hi. The require logged in user isn't halting the redirect chain.

15:31soulcuttr Mornin
I think if I return false, it'll halt the redirect change. Boy, I don't love changing auth code. All right. Hey, good morning, Soul Cutter. Welcome back.

...52Yeah. So I'm going to just put that back and be happy with my additional test. Assuming the test runs, let's run the whole suite. So I'm working through pull requests here before I get into bigger updates because people kind of trickle in over the first half an hour of the stream.

16:34So I'm happy with the code.

...43What is it called? soulcuttr I'm adding fields to an API endpoint, and then I need to go on a query optimization adventure
And I didn't write any tests for admin. There are only like three routes that are admin only. And the code is such a close parallelism. If you kind of It's given between these two functions. It's obviously the same function, so I'm not too worried about that. Adding fields to an API endpoint, and then I need to go on a query optimization adventure. Ah, we are also going to have some performance stuff here on stream.

17:24Is it a REST API or like the clever GraphQL one?

...34pushcx https://github.com/lobsters/lob…
So if I got that merged, the other pull request that I merged yesterday was this one, which again, I should throw these links in the chat. This pull request is one of the big story controller refactoring ones where I really cannot predict what people are interested in doing, but this one is a pretty unexciting, hey, let's refactor this controller and split all these concerns out. soulcuttr We discovered this week that an endpoint is making about 4,000 queries. LOL woops. This is a REST endpoint, but we do want to get it into our Apollo graph
Concerns. All these different actions and verbs out to be their own controllers and kind of lean on the Rails resourceful routing because woof is story controller, big and complicated. Ah, yeah, 4,000 queries. is a bit of an n plus one i usually say one plus n good luck with that is this a rails app for you too because apollo is the front end a lot of people use for graphql but i don't remember what the back end a lot of people use for graphql is This was so nice. This really... So, Utkarsh, if you're watching, thank you for this. You really simplified that controller. soulcuttr they're fast queries at least. It's not entirely n+1, there's a lot of duplicateddd queries too. plucks all over. This is a Rails app, yeah
Oh, and he jumped through a hoop here where I asked for a slightly different naming because he couldn't have known the name. Got that merged down. Real happy with this. Oh, plucks all over. Okay. Well, good luck with that. Have you seen prosopite?

19:39pushcx https://github.com/charkost/pro…
This thing. It is overkill for your problem. soulcuttr I have used Bullet in the past - I did see that recently but haven't used it
And when you add it, like half of your site will break. There may be a better way to incrementally add it where you can just turn it on like one controller at the time. is nice, and we used Bullet on lobsters for years, but Prosopite does what Bullet wants to do. So Bullet is not perfectly reliable. I have seen it miss N plus ones. I haven't seen false positives. soulcuttr I will give prosopite a shot!
Prosopite is just a little more clever, and it runs as a rack middleware, I believe. I'm really impressed with this. yeah the if there's anything here about yeah so if you are adding it to a significant large app you will probably want to turn off this raise i believe it's on by default in development and test so that you're forced to see these In a large app, you will have to turn that off at first and make yourself a burndown chart because it will catch one plus end queries that you have no idea. Like lobsters is not particularly big and it caught, oh man, not only is lobsters not particularly good, or I'm sorry, unintentionally revealing. Not only is the lobsters code base not especially big, the two primary maintainers, JCS and me are both like aware of this problem and very particular about trying to get rid of it. And it still caught, I want to say more than a dozen n plus one queries. You'd have to look back about two years in a commit history to when I did it. soulcuttr sounds like Prosopite is the next generation of bullet
But they all either included prosopite or one plus n in the commit message. And there was just so many. I think there are still some that are getting logged in production because we don't have anything like 100% test coverage. soulcuttr no commits for 2 years though? hmmm
yeah it really is i think the the idea of it being a next generation is is true or you know promise fulfilled you might even turn it off in production because adding it to a significant app will be a project in and of itself but oh it's so so useful I have it set on lobsters to log to its own file and for a while I would log in every couple of days and see another one and you know 900 instances of another one and I would just fix one or two and clear the log and then see what showed up next there are probably still more that I need to do that I haven't but it's getting there all right all these icons so I think check means I approved and that means it's a draft okay that says it's a draft redundant all right so this one oh yeah this one was very odd i don't know if i reviewed it on stream but there were some unexpected changes here and including database stuff. So they must have had a hassle getting the dev site running. I can't easily check out the branch and play with it to help with the bug they got stuck on until they remove the extra stuff. So that one's kind of strange. All right. So this is waiting on locks or conflicts and He wanted to see the sprockets merge, so let's take a look at that. I think this one I get to click merge on.

24:14I'm going to just drop down to the end here.

...22So this one kind of, Rahul ran into a big hassle here where our CSS used, I didn't know, one sec. goreng hello hello
I didn't notice that Precipite hasn't gotten a bug fix in two years. It has been working very reliably for me. So hey, Grang. Welcome back. How unmaintained does it look? Six open pull requests, some of which have been hanging around for a minute. 14 open issues, some of which are pretty minor. Well, that's concerning.

25:25soulcuttr sometimes when libraries work well, but haven't had activity, instead of calling them abandoned I will say they "have achieved perfection"
I don't know this person.

...36Oh, hi, Italica. When I am on stream, I actually don't get notifications.

...52All right, I have checked my phone. One sec, I have to send another text message.

26:22All right. Okay. Thanks for dropping by the stream to tell me to look at my phone. That worked pretty well. All right. Have achieved perfection. Yeah, I do like the idea of libraries that can be done. If prosopite is throwing some warnings though, there's just that always, there's always that bit of churn. so for dark and light mode we have a little bit of redundancy with our css and we had solved that with erb to template it out but prop shaft won't let us use erb in that way and so rahul had looked at refactoring it a couple of different ways and unfortunately broke one of the modes and then I think has come in and yeah, duplicated all of the dark color scheme stuff to keep the functionality working. And I asked about the prospect of using, there's a new CSS function And when I say new, I guess I mean really new, because I had thought it was a year or two old. And it's not even a year. But it was a function to allow you to minimize. OK, so this, you could have two color schemes. And then what lets you toggle between them?

28:27This may not even address our problem because our problem is we want to use the lighter dark mode that the browser requests, but we also want to override it when the user demands. And so this code is very similar to what we have where we declare a bunch of CSS variables and then

29:02So if we declared all the variables up front. Actually, this does look like what we want.

...27Yeah, it would be nice to. I know when I deploy PropShaft, it's going to break broad because changing the asset pipeline always breaks in production. Even if I run my local dev in production mode, actual production does something slightly different. That's just the nature of the asset pipeline. Yeah.

...59So if we pull all of these colors up into root and then said HTML color scheme dark. Yeah. So I'm tempted to do this for Rahul just because I know he's in India and the 12 hour time zone means at best he's not going to look at it till tonight. Yeah. But I don't want to step on his toes. He's been already just a valuable contributor. So let's grab, is there a table of contents here?

31:30What is this selector? HTML color scheme dark. Has color scheme dark.

32:10pushcx https://github.com/lobsters/lob…
So that's great. Let's look at Rahul's other pull request here. And just to share the link, that's this one. See if this one's ready to go. I don't remember the state of this one. So this one was waiting on authenticatable. And also that helper. So he's resolved some conflicts. He must have had his own feature branches. Oh, he had this idea of... Yeah. So I merged that. Let's see the details here, because I would... I really wonder what his setup is. Because I've seen him have issues with standard RB. And most people just configure their editor to run it automatically. And then breakman there is pretty nonsense. All right. So not nonsense. It's just the touching some stuff has caused it to change signatures. So here we have the that, this. OK. All right, so this can be removed. Actually, this whole controller can be deleted. Yeah.

34:35And he's already assuming it here. And this is redundant with his pull request, so he's going to catch some conflicts about it. Yeah, so Rails is Rails. GitHub does not really understand the idea of stacked PRs, so we're seeing everything that's in his authenticatable PR again, which is kind of confusing and annoying. Nice, nice admin helper. I would rather not.

35:18I would rather not have empty files hanging around. So you renamed that. Honestly, we have so few background jobs that I'm not real worried about that. I wonder if. So rahul has said a couple of things that imply to me he's not especially senior developer and I don't want him to get blocked on this giant scale thing.

36:29So let's be specific.

37:57I don't want to discourage Caius, but I also don't want to discourage Rahul. When did I become a manager? Huh? All right. So we've got how long ago echoed through, and then this is all very nice.

38:40We could get rid of that. And then we have solid queue and solid queue. It's a funny presentation of the diff. I'll take it, right? And then recurring. This is so much nicer.

39:09What is 4am? This is going to be 4am server time. And though the website runs on Chicago time, the server runs on UTC. So 4am is going to be about 10pm or 11pm depending on the time of year. That's a low traffic time of day. That's fine. And probably Rahul is taking that directly from our So this is resolved.

...53And then this is all the new schema. Great. This also has to happen.

40:59Submit that. Great. I wanted to... Come here. Can I go to this and just resolve it? All right. Good. So this one, I'm looking forward to merging this because we actually have a couple of places where there's something that happens sort of in the background or we try to fill a cache. off screen the two that immediately came to mind once i was thinking about getting this in were the traffic helper is on a cron job so that it doesn't get dog piled the traffic helper is an easter egg here our logo changes color changes brightness depending on how busy the site is but we of course don't want to do that for every single hit because that's A lot of SQL. And if we can just fill that in a background job every five minutes, that's much nicer than a cron job. And then the other part of it is when you're logged in, there is the count of how many messages are in your inbox. And your inbox is both direct messages and comments that are replying to you. that latter part comments that are replying to you hits a database view that's actually fairly expensive so it would be nice if when you are logging in or honestly at the moment you attempt to log in and we see your username file a background job to start filling that queue because then by the time you know, 100 milliseconds later when we've verified your password and rendered the page back to you, probably that cache will be full. As it is, when you log in, that first hit can take a second. Also, loading the replies page should enqueue one of those jobs. So you see the old number, but then in the background it goes and refills the cache. All right. so then i don't think anything has moved on the issues since the last stream no let's go back to the open issues and double check that and bookmark that clearly oh i added some notes to this and paginate that's worth talking about a little bit i already showed this stories controller thing so on rails 8 i broke this up a bit because we've been adding these things this one is happening over in the pull request i was just in that was a nice little touch there github To know that I would probably want to link the pull request that I was just leaving a comment in someone was clever.

44:32Can I just. Oh interesting I can break them out into their own issues clever clever that's new to me.

...51So I added, I ran into a Go Rails video. I think I saw this on Blue Sky, where there is a new params expect feature. So you get fewer 500s when vulnerability scanners barf into your forms. That will be nice. We can say fresh win on many of our views. Actually, let's just say, I really keep expecting that to have a single edit.

45:27What's the home controller that has all of the, yeah.

46:06Let's say this differently. Every action, the important ones are. So site traffic is really disproportionate, where of course, some pages are viewed much more than others. And if we get the The various lists of stories, that's what's in Home Controller. And the single story view, that's Stories Controller Show. Like right there, that's, oh man, more than 80% of traffic just off the top of my head. So.

...53And then I still have to go through those release notes. There must be more things worth doing. It's just a lot of detailed reading.

47:07So let's bring back up my notes oh I didn't mean to close that issue yeah. pushcx https://github.com/lobsters/lob…
This is still ticking along very nicely. This other one let's share it this. issue that I wrote here is.

...35It would be great to paginate people's threads, like this URL has actually changed since this was filed seven years ago. Not exactly an incredibly fast moving project, our codebase.

...58There are lots of places we do pagination and I was chatting in the IRC room. So that's, if you're curious, that's over here. We have a live chat room. You can join anytime. And I was chatting with Caius about this. So Caius posts the weekly threads, like what are you doing this week? And as a experienced Rails developer who's contributed a little bit recently, which is wonderful. And so I am trying to get him to do more, hey wouldn't you like to break these things and it's been hanging out for a while i took a whack at this on stream and it wasn't as straightforward as i had hoped and in talking with it I had said that the code base's general strategy is to not take a dependency when 20 or 30 lines will solve a narrow problem. It would be very reluctant to take on large dependencies that have a lot of complexity we don't need. The churn is expensive. The bigger a dependency is, the more often it's going to update and have bug fixes. And then I wrote a little more about it, that Pagi is 2367 lines, which is getting up there for a fairly narrow feature that what we want is a fairly narrow feature, but it actually has a bunch of things like a, a calendar feature and a bunch of front end helpers and two styles of pagination where it'll do limit offset pagination. And then it also does the, I call it cursor based pagination. Caius calls it seek. And I think Pagi calls it like key set. I'm going to keep saying cursor because that's what it is to me. So I spent a few minutes, because Caius really liked Pagie, I spent a few minutes reviewing it and I was really impressed by how few open issues it has and how few issues get opened, period. It seems to be they've written a very reliable piece of software and a chunk of that is keeping your feature set small and focused. Our biggest bug minimization strategy for lobsters is not tests. It is just not having features. The best code is the code you don't have. So I've been really careful about adding features. And in the Rails world, it's really tempting to pull in dependencies and say, like, I'm going to add this gem and then I'm going to add like 19 features. The problem is five years later, if that gem is unmaintained, now you have to code those 19 features yourself. And so I wrote a little bit about that here, that we've outlived some of our dependencies. And when we keep them well maintained, we minify our internal use of their API, which makes them a lot easier to replace. Oh, my God. And then second, if it's directly exposed to users, like in the comment box or in URLs, we want to take extra care that we have an interface we're going to live with. So Caius liked this. I'm glad he read it.

51:28And honestly, it's hung out for seven years, and I've taken three different swings of pagination in the, like, can we just write 20 or 30 reusable lines? And I don't want to let it be my white whale. So if this is what works and gets us along the line, this is what works. Great. This was not my philosophy towards Rails apps before I started maintaining the Lobster's code base in, what, 2017? Yeah, fall of 2017, when I became the admin. I was contributing to the code base before then, and so it was like starting to change my opinion. And now maintaining the code base, boy, have I really come around to this philosophy of, just write a really small, narrow version and keep it well contained with a limited API, because it's very easy to go from that to replace with an external dependency of some kind, let alone just not having the churn. And that's why lobsters keeps getting picked up and reused as like a Rails performance benchmark or an example of how to write tests or how not to write tests. Yeah. All right. So I've been going about 50 minutes. We've gone through the pull request review, and I talked about that code base philosophy. Now is a good time for me to do, you know, the bumper that this is the office hours for lobsters, this website, and the code base. If you have any questions about the site or the code base or the moderation, you want to run queries against the database or speak up any time you want. You can chime in any time. In worst case, I will put you in the to-do list I was just showing to answer in a few minutes if I can't take the interruption. This exists because I wanted folks to have one more medium to answer questions and see behind the curtain that, much like Oz, there is in fact no magic. There is no great and powerful Oz. There is just a guy who makes lots of typos in VIN. That's all it takes to build an online community is to make slightly more correct commits than bug commits.

54:03There is the occasional bit of nonsense. pushcx https://lobste.rs/s/ukosa1/
And if nobody is piping up, there is the UK has created a online safety act. And you can read all about it there. But basically the United Kingdom has said, we are going to regulate every forum, every bug tracker, every website where one user can talk to another that is not email, e-commerce or VoIP. And Somehow, they are allowed to institute these rules over every website if it gets read in the UK. And my very pointed question for UK Parliament is, do you feel that this is a reciprocal right? That every country in the world should be able to censor UK publications if they might have readers in their countries? So, for example, did anything happen in Tiananmen Square, say, May 15 through June 4, 1989? The BBC says yes and has written many stories and made many documentaries about this. If those are consumed in a country that believes nothing happened in 1989, Should they be able to censor these UK publications like the BBC? The jurisdictional claim in the Online Safety Act says yes, yes, absolutely. One country gets to censor authors in another if it might read their things, which is very strange. And where was it?

56:17I wrote about it. I wrote about it here, but the, I noted that lobsters is not a UK site and the answer is obviously no, that the OSA should not apply, but then their tools and their writing seems to imply the UK has global authority. I may have missed the change in international policy that would permit the UK to exercise such jurisdiction and Ofcom has failed to reference one in their guidance. pushcx https://en.wikipedia.org/wiki/W…
And this link goes to a Wikipedia page, pardon the bright mode, I'm not logged in or anything, to the Westphalian system, which is kind of the, there are a little historical inaccuracies in calling it this, but there was a treaty in 1648 that says, if you are a country, however big or small, you get to run things in your own borders. And this sounds kind of funny to the modern ear, because it is the system we have had for, you know, coming up on five centuries. But it wasn't at all clear at the time. And a lot of that is tied up with the history of the medieval churches. That also said that they had authority over what happened in different places. But this is pretty foundational to the modern international order and the USA pretty clearly disrupts it. Without any kind of justification or even mentioned that it's doing so. And I'm kind of beating this up because I keep raising this point and then I keep getting reactions where people are like, but here's one weird hack to ignore that. Or here's someone saying, well, obviously Ofcom isn't going to come after a little guy like you. You're just American big tech. The thing that the bill is written to target. And no one is engaging on the jurisdiction question, which is very frustrating.

58:35I especially would love to hear Ofcom and Parliament engage on the jurisdiction question. pushcx https://lobste.rs/s/ukosa1/uk_u…
So I actually have it linked here. So maybe really useful. I wrote up a comment on the OSA yesterday because I finally found avid stream watchers will remember me looking for this page on their site and not being able to find it. This page is as best as I can find an overview of all of Ofcom's guidance. And a couple of the early links are repeated, so this isn't super clear. But if you go through and expand all of these sections, you will see all of these PDFs. So if you would like to comply with the Online Safety Act, all you have to do is read all of these PDFs. And I went through and totaled it up. This is only 2,426 pages of dense, interconnected legalese with complicated cross references. That's it. 2,426 pages to understand this bill. then of course also you do need to read the Online Safety Act which is 350 more pages because a lot of stuff references the Act or you just need it to understand the structure of that and it's funny I was talking about this with someone and I realized that if you add 2426 and 350 pages for the Act you get 2,776? Which is kind of funny, because I keep harping on this jurisdictional question, and I believe something relevant happened in 1,776. Daphne counseled me accurately that I should not wave this flag too much, but it was just a very funny coincidence, and I can't not see it. I do think she was correct that coming off as, you know, I'm going to wave the flag and be patriotic and put on my stars and stripes bandana and be a prick about it is not going to be rhetorically successful. chaelcodes lol, I'm sorry that 1776 comment was hilarious
The more polite version is like, come on guys, do you believe that every country should be able to censor your authors? There are, just to pick a famous UK work of art off the top of my head, there's Harry Potter, which contains immoral scenes of women who leave the house without male chaperones from their immediate family. Would you like other countries who think that's inappropriate to censor your famous cultural works? I'm going to guess the answer is no.

01:01:57However, this 2,776... Well, if I said something funny, we can use it as the title. Otherwise, I'll have to dig back in the transcript. Oh, hey, Chill. Nice to see you again. Just clicked with me what your username is. So this page count of 2,776 is only temporary because... they are starting phase two of producing documentation. And so for all the people who are disappointed that 2,776 pages is not enough, they will have another guide this month, another guide next month, codes and guidance in May. They're going to do consultation on additional codes of practice. chaelcodes 👋
And what I understand is consultation is They go and talk to the five biggest companies that they wrote this law for, and then other people tell them they're making mistakes, and then they ignore all of it and write guidance that only is appropriate for the five biggest companies. And then they're gonna publish some more and publish some more and publish some more. So in case 2,776 pages is not enough, don't worry. It's just the beginning. There's phase two, there's phase three. This is like the Marvel movies of online regulations. The Ofcom cinematic internet has multiple phases. I don't know that, you know, in this analogy though, Ofcom is Thanos and is snapping half of the internet out of existence.

01:03:52And honest to God, if they would write, they can put it in a PDF if they want, but if they would write a one-page statement that said, obviously we will be prioritizing and working from the largest sites down. If you run a small site, we'll see you around 2040. It would basically kick the can long enough for them to build up some amount of trust with everyone. they've written now is hey everyone you're all but they've written it like kind of a polite vague legalese because they are writing for their big political targets in a very universal way that means all of us little unimportant people are also targeted they could fix this in I haven't actually tried to write it, but I think in two sentences for sure, and maybe for one, especially if they want to use a semicolon. Same for the jurisdiction issue. That also could be a sentence or two. So I guess I'm going a little out of order. Someone introduced me to Ofcom. I described this two streams ago. I had gotten an out of office reply. I have sent a like, Hey guys, how's that going? You back from your break? Let's talk about how easy this would be for you to put out this fire. I have not yet gotten any acknowledgement of that. I am continuing to chase that and email continues. I'm going to email more people at Ofcom because I have a couple more email addresses and I will do my own cold emailing. One of the things that working at the Washington Post taught me is offices full of powerful people are exactly the same as every office you walk into that is like the middle management for the Near East region of Starbucks. It's just a bunch of people with Excel and Outlook. And in the government, most of them care Because almost everybody who works in the government has taken a pay cut to do so and cares about the mission, and I am assuming that is true of the UK as well. And if you talk to them usually people are pretty darn responsive and very happy that someone is paying attention to their thing and trying to help improve it. This is even true when you are a muckraking journalist. Oh man I there was one time I did I worked on a story that was about nutrition and I talked to folks who are at the food and drug administration, which is. The American regulatory agency that makes sure there isn't too much sandpaper in your bread. and talking to them. It's not just that they were friendly. It was like I adopted a puppy because they were so thrilled that someone was using their data and trying to present more about their work and make sure people understood what they were seeing and what they were doing and how the regulation worked. And usually Usually I got a pretty positive response when I was talking to folks in government. The FDA was like I adopted a puppy where they just kept following up. When is it going to run? Is it going to be in this Sunday? When's the website up? Hey, have you seen this? They were so great. I loved the FDA. Or at least that part. What was that? Food and Nutrition Science section? I don't remember the name of the specific department anymore. I had been hoping for a response like that from Ofcom. I will keep hassling. One of the other things about having been a journalist is just keep hassling. Oh boy, it works. So the next one is forum owner response. I have been contacting people I know who run forums, and that is both folks who are in the UK And the folks who are in the UK have said, we don't really know anything about the OSA. It's not on our radar. That was a little scary because they are most, they're like first in line. They don't have the jurisdictional protection.

01:08:55And then I talked to large American forums and I didn't ask about whether they minded if I shared their response. So I'm going to be, I'm not going to name names, but they are a large, well-resourced American forum and answered that it's not on their radar. So I sent them some links and they are figuring that out now. They actually have lawyers around, so. maybe they will come up with something. Speaking of actually having lawyers around, I have mentioned on stream that I've talked to this lawyer in the UK, Neil Brown, who has previously, I think has previously worked with Ofcom as an external party and has started this website, onlinesafetyact.co.uk where he is writing his, oh, I believe the stock is new, since I just loaded this earlier this morning or yesterday. So he is writing for small sites, especially located in the UK. He hasn't yet written anything for non-UK sites that I'm aware of. And it is basically a human's guide to the Online Safety Act. And yes, I know all about that link by that guy, Russ something. caiushardcore Russ Garrett
I would like to never see that link again, because it is just a guy who is trying to make sense of this. And there are so many pitfalls to playing lawyer if you are not actually a lawyer. So this is really encouraging, where he is starting to say, what does it mean to do these risk assessments and compare against Ofcom's guidelines for which categories you fit in and which hoops you have to jump through. It is still very early. So I could summarize these three samples very quickly. All three of these samples, Russ Garrett, yes. I don't at all mean to insult him and I appreciate that he tried to write an assessment or an abbreviation of it. But he is not a lawyer and people keep sending it to me, I cannot rely on something that not a lawyer wrote. It is so easy as a non lawyer to make very big mistakes, because you don't know legal jargon and you don't know. legal assumptions about what it means for a contract to be valid, let alone in another country. So it's just not enough to have the risk tolerance reduced. caiushardcore Also hi :)
So anyways, Caius, welcome. It's very nice to see you, and I was just talking about you. I don't know if you were here earlier, but I talked through... Come here. See, I'm not even getting my new bookmark. I made a bookmark for nothing. I was talking through that long comment I made to you about the philosophy behind taking on dependencies. The only thing that didn't make it into this comment was me realizing live as I was talking that I've actually worked on pagination three times now over the years, and it is becoming a white whale. And you know what? If taking a dependency is what gets this damn thing done, do it. The other one was there was an interaction that we had over in the pull request about ActiveJob where you left a really thoughtful note about... Now, where is it? I resolved it. You left a really thoughtful note about basing queues on service level obligations and you had a great post. I said, let's punt on this because Rahul strikes me as maybe a junior developer. caiushardcore Hah, fair enough. Comment on dependencies was a good clarification, thanks for clarifying it further.
He's commented on being new to Rails and I don't want to overwhelm him with this scaling thing that yes, we should, but this is a lot to take in. especially if he is a junior developer for all of web dev rather than just for Rails. I don't know. Some of the things he's done, like in the prop shaft pull request, he kept trying to go different directions or add features to it in a way that I don't usually see senior developers do. Senior developers are usually like, how much can I throw out of scope and not do? caiushardcore Cool, lets not bikeshed it - we can easily change the queues later if we need to
And he was like, what if we rewrote the CSS to Tailwind? And I was like, that's a lot. We're not taking on Tailwind. And so, you know, I'm just trying to eyeball his experience. Yeah. So... If you want to wait, I am hoping he comes back to this in the next day or two to finish off this because there's just a couple of little things that need to finish for this job system. And if you want to open an issue afterwards or even better, just PR it, that would be wonderful. But I don't want him to get blocked because he doesn't understand all the implications of this. Because I read this and I have seen queues operated like this and I'm like, what would I change? So anyways. Great to see you here very great to see you in the issue tracker and pull requests.

01:15:14caiushardcore No problem, glad to be of help!
So where I was was saying that. Neil brown's write ups are very promising they are not super applicable and the spoiler on all three of these is. Because these are. for internal use by a business or just by him, the OSA doesn't apply to any of them, which all feels totally plausible. I hope he writes up a sample risk assessment for a small forum, you know, obviously. I hope he writes about the jurisdictional issue. I am watching his Mastodon and this page very closely because He is an actual lawyer who is writing about this stuff. And even if, you know, there's a whole thing called giving legal advice where is a lawyer giving you advice for your particular situation? And if a lawyer is writing on a general webpage that is published to the world, obviously they are, obvious to me, they are not speaking to you and you cannot treat them like they have become your lawyer or, that they are writing specific to your situation. And that's why you see these kinds of disclaimers. Even so, seeing an actual lawyer write about this stuff, that's why I don't know this link, is very promising. This one ends in MD and the others don't. Is this too much? Yeah. Let's tell Neil this link is broken. I'm going to... I'm pulling up Mastodon off screen. Because I want to say... How do I... So I've got Mastodon off screen. And I'm going to send Neil a quick message that he has a broken link.

01:17:54Just to say that there shouldn't be an href on the end of this link. So with that done,

01:18:18Where did my comment go? I ran into on Mastodon somebody talking to Neil about this.

01:19:07So I am leaving that slightly, I mean, can I pull this in, in a way that's not going to show DMS or something? I hate working off stream and being like, I swear I'm doing something. so I gave Neil that link and then if I, you want this on the screen, somebody wrote this is what i got nerd sniped by but neil it's only several 200 page documents of corpo language you have to sift through it's super accessible for everyone and oh this is only 12 minutes ago so neil is active right now so hopefully he sees that that broken link and i pointed out caiushardcore Madness
that i totaled it up and it's actually quite long all right so you missed the earlier joke if you add all of that off-com guidance the act itself is 350 pages so i wrote it here If you add 2,426 to 350 pages for the law, you get 2,776. And as an American, that's a very funny number.

01:20:50So the last thing, this is not the code. So this is the Lobster's office hours. If anybody has questions about the site or code base, you can ask any time. When I am not reviewing pull requests and answering questions, which is whatever you'd like to talk about here, I work on the code base. And the thing I have to work on is that the OSA has caused performance issues. Of all the nonsense. So if I start the server in the background, It'll take a second to come up. There we go. So at the bottom of this page, there's this big dumb banner. This footer that I put on. This footer says, hey look, I have geocoded you to the UK and you're about to get blocked because I can't comply with this law and I don't want to be the test case for this law and I don't want to be like, oh yeah, let me just spend the next couple years of my life defending myself against extradition to the UK because I run a forum. And... That's so fucking ridiculous. I'm not going to beat it to death. The current bad strategy, and it is in the code base right now, is that before the law takes effect, the UK will be geo-blocked. So to the best of our ability, no, we will not have any readers in the UK. Which sucks. That is awful. That is, .. That is the UK burning down the village to save it. To use a very American metaphor. I don't know if other people know that one.

01:22:57caiushardcore Cut off your nose to spite your face would be the UK equivalent I think
So I put this banner on to say UK and it's visible in development mode because I wanted to be able to see it. And I am trying to get UK users to get involved and talk to their members of parliament and talk to Ofcom because I haven't had any success with those things. And obviously I am not a constituent. Cut off your nose to spite your face. Yeah, that's a classic. Certainly heard that one growing up. So I've tried to get UK users to help out. That is the link here, slash UKOSA1. Obviously, I did that by hand in the terminal. Because I had to link it all over the place, and I can't memorize these random story slugs.

01:24:02chaelcodes Would a VPN help here? For UK readers?
So the problem here is this banner should appear to all users. pushcx https://lobste.rs/s/ukosa1
Would a VPN help here? Let me throw you this link because you're hitting one of the facts. So I discussed it up here that... Where is it?

...36So I wrote it in the original post, so this is what paragraph six or so. Our current bad plan is that we'll geo block the UK before the law takes effect because. Even being subject to even having to prove that an American site run by an American in America is not subject to UK jurisdiction is ruinously expensive and disruptive. caiushardcore That reminds me, I need a tailscale exit node outside the UK. Gotta run, seeya!
then i note that while the inaccuracy of ip databases and availability of vpns mean that this can't be perfectly accurate unambiguously blocking uk occupants as effectively as we can is the only course i can see to substantively reduce the risk that the osa is enforced against this site so sorry caius Good luck with that thanks for dropping in so chill. it's this ugly situation where, as a practical reality, I know that. So kaios here if he sets up a tail scale exit node on a friend in america's home connection. There is. effectively nothing lobsters can do to recognize that because he would look exactly like someone coming in off of a residential isp in america and there is a a programmer thing where programmers are like oh if something is imperfect it basically doesn't exist and i think this comes from we spend so much time of if the code has chaelcodes Intentions.
a bug it effectively doesn't work at all and that kind of extreme universalism is not at all how the law works where the law is just kind of built out of traditions and exceptions and exceptions to the exceptions and case law and other kinds of stuff so the programmer mentality is very misleading because the programmer mentality might say Well, you unambiguously, yeah, intention is what I'm getting at. You can't perfectly block visitors from the UK, and you know that a visitor might sneak around your block. But if Ofcom, which is headed by a political appointee of the current ruling party of the UK, decides to make a political point out of this, is much harder to make that political point against someone who is doing the best that they can to tell you to go piss up a rope by geo blocking because this says if the uk believes that they get to have jurisdiction over american sites who have uk readers it is harder to claim with a straight face that also we believe it applies to anyone who tries to get rid of uk readers and i don't believe that being honest about the limitations of these technologies in any way limits the risks because i am sure that ofcom knows that vpns exist someone sent me a message saying like why did you say this If you admit that it's imperfect, aren't you giving up the legal defense? From having worked with lawyers, my best understanding is, I mean, they wouldn't write this kind of thing, but I just would lose all credibility with technical people if I pretended that VPNs don't exist, or I pretended that IP databases are perfectly accurate. Part of this is, chaelcodes Yeah, lawyers don't want to say anything.
I am sure Netflix and giant content providers that want to do geo blocking have enormously more expensive and sophisticated ways of enforcing their geo blocks. And even those don't work. vinitkme Hello there!
But I am a guy running a forum for beer money, having to spend $50 on a GOIP database is actually a significant expense. Oh, hey, Vinitkma. It's been a minute. Nice to see you again. vinitkme yeah, been busy with a new job
So, Chael, the other thing that is worth seeing in this post, and it's down in the comments a bunch, is there is not one weird hack. that we can do to tell the UK to get out from under this. Oh, hey, congrats on your new job. There is not one like if we hang a sign out that says like, or if we say UK users can only read the site, not just visit. If we say we're not a forum, we're a multiple author blog. Like there's no clever get out of jail free card. vinitkme Thank you! New EU/UK regulation bulls**t?
gmem_ As someone currently living in the UK, running a Mastodon instance with a bunch of UK users, the OSA gives me a migraine whenever I think about it
And I know as programmers, we're always looking for that kind of loophole. There is not one. And we kind of beat to death in the comments, the couple first ones that people have come up with. Yeah, GMM, I don't know if you were here earlier, but there's a comment down here from me. If you search for the number 350, you'll find it where I found their table of contents page that I was thinking of. So, if you want to work through Ofcom's guidance, there is the 2,426 pages that you can read. Yeah, FNICMA, it's UK, because the UK is not currently in the EU. And basically, they claim authority over the entire internet. And this is pretty foundational to the law. I linked it in another comment down there, but they are very explicit that they believe if UK occupants, if someone has their butt physically within the borders of the UK, they believe that that gives them authority to regulate, say, American sites in this instance.

01:32:06So the UK is causing, the Online Safety Act is causing performance problems because I am trying to show this banner to get UK users involved to the users who are in the UK. Like all sites, Lobster's has, so on a daily basis, we have around 2,500 users vinitkme Can't you put cloudflare in front of it and use that to geoblock stuff?
who do something actively on the site, like submit a story, write a comment, or upvote a story or comment. Just using that definition, we have just ballpark 2,500 active users on a US workday. Could we put Cloudflare? I have no idea. I've tried to avoid taking Cloudflare on as a dependency, and I've already written the code. If you look in Where's the commit? Right here. So here, the geo block is already pending. I don't need any help writing that code. That is the current state of affairs right now.

01:33:31So if we have 2,500 users who are logged in and do something, we have, just because it's the usual round number, probably 25,000 who visit and don't interact. No, that's probably high for us. Maybe call it 10 or 15,000. Feels more like my intuition. This is not something I've rigorously tried to check. who are logged in but don't interact in a way that adds database records. pushcx https://en.wikipedia.org/wiki/1…
And then because of our invite system, there's a, what is it called? Wikipedia calls it the 1% rule. I've always heard it called the 1,990 rule, where like 1% of users add things and like 9% are doing something. frici Timezone Appropriate Greetings to everyone asecre3Hi
yeah ninety percent consume nine percent change or update one percent add we actually have a slightly more extreme ratio oh we have a slightly more extreme ratio because of our invite system so where we have something like hey welcome back free chain well where we have something like 2500 interacting users per day we see something like 250 000 visitors per day so if we have a quarter million people visit the site every day they are not logged in They are not doing stuff, so we have done full page caching. So this page, if I were on production and not logged in, this whole page would be cached for 60 seconds just to reduce our database load and our web server load. And this geo-based banner is fucking it up. On last stream, I turned it off. Maybe it was after a stream. I had to turn off the full page caching. because the cache would get filled by whoever landed on the page. And so whether you saw this banner was based on whether the person who filled the cache, which is just, you know, lottery style, whatever visitor visited in that 60 seconds when it came up, whether they were in the UK determined whether everybody for the next 60 seconds would see this banner on the footer. And that's just for users who are not logged in. If you are logged in, we skip the cache, we render each page artisanally. And the web server is starting to get sluggish in the middle of US workdays because of it. And clicking around in production is slow, and posting comments is slow. There's a... chaelcodes If you add a query param, does it store as a different page?
gmem_ People should be working in the middle of the workday! Kappa
This one has been such a... Speaking of white whale... We see deadlocks sometimes when people post comments. Yes, if you add a query param trail, it stores it as a different page, and then that gets wiped every 60 seconds to two minutes. I don't remember the timer. chaelcodes ?uk=true?
It differs slightly.

01:37:09It also skips the cache dr3ig How does caching interact with serving from multiple threads?
Yeah, Chael, that doesn't work for someone's first page load. And then I almost can't. I guess it's an aesthetic concern that adding that is kind of ugly. How does caching interact with serving from multiple threads? It's Nginx's problem. So it's before we hit the threaded Puma workers. So if you look at a Lobster's Ansible. pushcx https://github.com/lobsters/lob…
And we look at my commit from a day or two ago. So we use the Rails gem to write that full page cache. And then this is the bit that's commented out. And there's some, it's expressed indirectly because Nginx config for performance region is very, So we have to do this thing where we build up a string.

01:38:34But what happens is Rails serves a page and you might see dogpiling where two threads will generate the page and two threads will write the page. That's last one wins. so if we were to see a huge influx to a particular page just as the cache expired yeah they would all generate and all try and write it so it is not a like google scale kind of solution to the problem it is a lobster scale solution to the problem and then nginx just serves if it has the file and it doesn't have a session cookie or a filter cookie and we see the HTML file on disk, NGINX will serve it directly. And that is so much cheaper than having it done by the Rails workers that it's a big performance win for us. I hope that answers your question. I feel a little bit like I'm rambling about the concept of threading and caching. So I had to turn off the full page cache so that this banner would be correctly visible to UK users who I need to help with the Online Safety Act. And the production server is like CPU pin during the US workday now. I actually thought posting a comment was broken because I clicked the Post button. And instead of responding in the usual 500 milliseconds, it took five seconds. And where did I put it? I lost track of the tab, didn't I?

01:40:48Where's the so rails. This gem action pack page caching was a built in feature for rails for years. And then four years ago, they spun it four or five, they spun it out to its own gem. Yeah, removed in 4.0. So however many years ago that was. And it hasn't even been touched in four years, which is a little concerning because there's things it could be doing to use Rails 8 features a little better. But I have that like, oh, it's that vibe of, oh, we have this dependency and we are starting to outlive this dependency. It works right now. Will it work next week? Will it work next year? Probably not. And I say probably not because Ruby 3.4 is doing that stuff with frozen string literals. This whole thing is string munging. I don't think Ruby is making a backwards incompatible change anytime. No, they are. When they enable it, that'll change a bunch of warnings. And so I haven't checked if this emits warnings. I looked at the issues. Yeah, so there's stuff about Kamal here. That's a Rails 8 feature. The answer to this one was no, I think. And then this is a something in the rails plumbing has changed. So I think realistically, let me grab this for the scratch.

01:42:51This is, you know, this has lost the mandate of heaven and is no longer part of Rails. And so my options are, where is it? I can never find, what's the GitHub UI for, oh, it's insights. I wanted to see if there was an active fork. Well, updated two months ago and two years ago are definitely more recent. This looks like someone who tweaked something that personally bothered them. Remove deleting slash. So this is removing a trailing slash off the URL. I have no idea offhand why they did that. And this one, do you have multiple branches? No. Why did GitHub think this was active? I guess the fork was just created a couple of months ago, whatever that last page said. So it doesn't look like there's an actively maintained version of that gem. I don't want to actively maintain it. So with Rails 8, I have been trying to lean more into the Rails 8 features. And if we look at the guides,

01:44:30pushcx https://guides.rubyonrails.org/…
The current guide about caching says. I'm adding it to the scratch. Basically, it says. You take your views and you shotgun these calls to the cache method into them. And I don't love that because our use case We really do have page caching. It works so well for us.

01:45:09On the other hand, the site is slowing down in production, so I've got to do something. And that something I'm going to do is definitely not upsize the web server. It's already, I mean, until I turned off this page caching, it was overbuilt.

...37I know a lot of developers love performance work. I don't super love performance work. So I've done a bunch on it here. Why did this render slash about when I loaded this page? That doesn't make any sense.

01:46:01Yeah. Maybe I should just port the whole view layer to flex for performance.

...16I have done so much stuff to remove n plus one queries.

...28Who was that? To Soulcutter's current project.

...36I've done a bunch of stuff to knock down our rendering time. But if the home page is taking 800 milliseconds total, what is that, 747 milliseconds total in local dev, it's going to be more in prod. Yeah, see, you did some kind of caching. I don't know what caching this is that fired, but now I'm getting a normal number, a reasonable number.

01:47:09So that's about that. I either need to... The boss is forcing his way into the office. Hello, sir. You want to come join the stream? You got kibble up on your favorite spot. We'll see if he comes in. He may have just been mad that any door was shut against him. chamlis_ can you do the IP lookup in nginx and add the banner there?
It's like that, I think it's Mark Twain's definition of a door. What a cat is perpetually on the wrong side of. Hey, Shameless, welcome back. I've looked at that.

01:48:09And there is a, so they have two modules. I think this is the crash. This isn't even the page I was looking at yesterday. There's this module you have to add to tell it. HTTP geo IP. Let's see if production is built with this.

...52HP, OK, so it is, that's genuinely surprising to me. I sort of assumed it wasn't going to be built by default on Ubuntu. So we could do that if, yeah, so the question was either do that or do more of this kind of caching in the view layer. And I was leaning towards doing caching in the view layer because it's the Rails 8 way. It's the way the whole thing is pushing and I want to lean into that. frici the times you get happy Ubuntu builds packages including the kitchen sink LUL
But I don't even know where to start with that because it's not clear which sections of these pages are expensive. I mean, obviously the homepage view.

01:49:45Yeah, that's fair. If this is built in, That's a little less daunting. Let's see if we can figure this out. So I looked at this and is it the, that's all the NGINX documentation. There was another, chamlis_ I don't know how to safely insert HTML with nginx though, all I immediately find is a straight string replacement which scares me a little
page yeah so this talks about getting it installed and then this is different docs than i saw i don't actually want to do the restriction here what i want to do is

01:50:30want to have one of those nginx ifs and i want to split the page in two or just have the uk bypass the cache that would actually be totally sufficient i don't need to so yeah so shameless what's the quickest way to do this so if i go to edit code ansible roles nginx dr3ig I don't what's the performance impact, probably more of a memory issue, but I believe in the `Maxmind` module, the `MaxMindDB` database is instantiated on first request (and presumably in every thread) rather than when the app starts and the module gets eager loaded.
files and Jeanette no it's in production because there's this whole thing we don't use all right here is our config file so on this one we build things up if I could build in like add a letter for the UK I must have commented this out in broad too and it didn't make it into Ansible let me check that real quick I'm already on lobsters.

01:51:48Yeah, I had to comment that out in prod, and I didn't get it back into the repo to commit it.

01:52:00So if we added a letter, So what I'd ideally like to have here is if, and then, you know, check if UK. So if I can write that, if I can say set use file Ash to, what do we want to call this? chamlis_ british flag emoji
So I have S F I H and O in use, I guess you for UK. let's just grab this whole thing because otherwise i will typo british flag emoji all right there we go so if we can figure out what we put in that if we have the goip database we have the module presumably i need to write some kind of why am i doing this here I should be doing this in production.

01:53:14Yeah. Because there is going to be a cycle of running nginx-t to make sure that this is happy. Thanks for that warning. So if we look at chamlis_ stream title? "I should be doing this in production"
We're living dangerous. Now I'm logged in as root on prod. No, it's not maintenance. It's... What did I call this file?

...51Hold on. I'm not listing all the files because I might have left some notes here. Ah, here we go. so the current version of the database is seven megs which is survivable oh thank you yeah it's that or the 1776 if i said anything quotable there all right

01:54:29Man, if you needed a reminder that lobsters is not a commercial service, I'm logged in as prod on root and hacking out nginx config. chaelcodes I thought "When did I become a manager?" was pretty good too.
frici I didn't gasp at all sir, idk what you mean LUL
That's lobsters. Where's the... Why is that not this file?

...57There we go.

01:55:07frici scary prompt
Wrong clipboard. Scheme of the show. Still the wrong clipboard. There we go. Oh, great. The Vim in production is indenting with tabs.

...33That ought to fix that. So if we can fill in this if,

...50frici root, in prod, on stream, with an unconfigured vim. I too like living on the edge
chaelcodes LUL
Then this, then we could edit these to say, is that an Austin Powers reference? Man, speaking of old memes, I think that's, I too like to live dangerously. frici and yes it is.
I actually only just saw Austin Powers for the first time two weeks ago. All right, so here's this, and then this comment can move up here. All right, there we go. Yeah, the really important things, when you're logged in as prod, stop and make sure you're correctly quoting Austin Powers. I mean, you gotta be meticulous about this stuff in production. chamlis_ your if condition and U comment seem to be in opposition, btw
So Shamless, you sent us down this route. Have you used this nginx module in prod? Because I've skimmed these docs, and I didn't see the example. My if and you seem to be in opposition. Well, you're a VIP, so you've probably caught a bug. chamlis_ "check if UK" vs "not in UK"
So the way a lot of these work is if the cookie is blank, You get the S and then so if it's blank. Yeah, so I am saying we are going to use the cache if you're not in the UK. Because I am assuming that UK users are not a majority of site visitors. chamlis_ oh so the if is if you're *not* in the UK? my bad if so
And so if they bypass the cache, it's much less load than if all of. The rest of the Anglosphere and the bots. yeah okay i appreciate you checking it is a little strange but if you look at these we are checking to see you get the s if you don't have a session cookie you get the f if you don't have a filter you get the i if you are asking for an index that exists and then these things are written to say well if you get enough letters chamlis_ I read the condition placeholder too literally
chamlis_ I've never done this nginx geoip stuff before, I'm afraid
not enough but if you satisfy all of those conditions and since nginx is not super expressive that's why we have to do this build up a string thing so how do we tell it all right so we have to say load module in the main configuration level okay so let's grab that Ah, yeah, check if... So this has to be up here, is my understanding. And probably there is somebody who actually knows nginx-config who is screaming that I've done this in the site file instead of in the other, but... Just trying to hack this out, and hopefully... I really, really hope this code only has to live for a couple of weeks until we get this thing sorted out. And you know, I say that, and that's how you know this code is going to live for 20 years. Because if you think it's going to live for a week, oh yeah, it's immortal. All right, so I have to do it in an HTTP block, which is out here. And so I can say, And then I guess this is the path to my file. So that's serve lobsters. What did I call it? Better to copy and paste. save myself some hassle so I just say that and then we have to figure out what goes in there yeah if we can get if we can get this working it's like five lines of code as opposed to let's shotgun the views with caches and you know I say that but I was just talking about how like In the other issue, the single story view and the list of stories views, those actions make up at least 80% of our traffic. That's actually only two cache blocks to add, one around all the comments, which is expensive to produce in templating, and then one around the loop for list detail with heinous inline partial.

02:01:03Hmm, I'm going to continue down this road. Let's see how this goes.

...16All right, so we want to pass to map or split clients directives. So maybe I don't end up writing the if, I end up writing a split clients and then, you know, the UK gets a branch where there's no caching and then everybody else gets the branch where there's page caching. The only variable I care about is we'll call it GOIP to data. And they don't have it. We'll say country ISO code. I'm trying to really closely match their example. chamlis_ s/county/country/
And if I look, my custom key bindings are missing. That's going to be super great. That's not going to drive me insane or anything. And if I look at the... Okay, so it's over in the max mind extra. That's what it was. So we are looking up the field country.iso code. So I think that's going to match. These databases are pretty well standardized.

02:02:38Can't scroll here. So we don't have a, I don't think I have a stream section. I have an upstream section. And we don't have a map section. And I have no idea what they do. So we're going to have to figure that out. Scenario choosing the newest server. Did I typo country? Thank you. I should have copied and pasted. It took me years to learn to copy and paste trivial stuff, and I still don't always manage to do it.

02:03:29Now, here we go. This will create the nearest server variable. So if I went down here, I said, I can't click. Well that, all right, there we go. I at least have the mouse working now. So if I said map geo IP database, ISO country code two is UK.

02:04:13The default is that people get... What a weird... Are these string literals? I guess? Because I want to have a... The default is going to be the letter U, and then UK should map to... I don't know, ASDF. Just anything that's not U is enough to break it, because then this says... chamlis_ is this the right module documentation? I think it might be the Nginx Plus one, whereas the nginx.org one has different syntax
if is uk equals u and like i am aware that i could collapse these two things together but i'm trying to go step by step create the server block which will pass the request to one of the upstream server groups hopefully i can just have an if

02:05:17chamlis_ I think you want https://nginx.org/en/docs/http/…
What is Nginx Plus? frici oh yes it is
Oh no, that's the commercial version? I, shit. Good catch, Shamless. All right, let me throw that away then. That would have been a lot of pain. This one also Plus? chamlis_ it's .com vs .org
Yes, so. All right, so let's start over, I guess.

...53It's converse org. Thank you, I wasn't aware. It tells you how often I do nginx config, right? I know just enough to be very dangerous in production. That's why I'm logged in as root. So this one says in the HTTP section, I should say, what, GYP country, and then give the file. dr3ig is country iso_code "UK" or "GB"?
Because what I want is GOIP country code. The following variables are available when using this database. Oh, Greg, good catch. It is GB, not UK. But I think we're going to lose this map, right? So for historical reasons, it's GB. And I would like to isolate that gotcha to the lowest level, I think. So if that's going to pre-fill GOIP country code, then I don't need this section. chamlis_ feeling bad about this, but s/geoip2_county/geoip_country/ (remove 2, add r)
And then here, I would say if GOIP country code equals, because they do single equals testing, if this one equals GB, Oh man, I can't type the word country, huh?

02:07:32There isn't a two anymore? You're right, there isn't a two. All right, thank you. You shouldn't feel bad about this. You are catching really annoying typos and misreads. And I've mentioned this on stream, but keeping up the running patter knocks like a third off of my coding ability so i make these kinds of mistakes much more often so i really do appreciate that folks point these out so if i'm allowed to say if on that variable then i can say we set the u if you are not in the uk Yeah, I'm never going to... chamlis_ should be != "GB"?
I think part of what threw Shamless here, let's say... Let's call it W if you are in the world. You know. Let's not be snarky. Yes, it should be. Except I... Is that their syntax? Yes.

02:08:47How we do a not equals, because it might be, because they do single equals here. Is it like this? Or is it like slash equals? Seems okay so far. All right. So if you're not in GB, you get the W. So you get to be world. And then here, here, here are updated. This looks pretty short. All right, let's see if nginx-t likes it. Come here, don't retype stuff. You're logged in as root. nginx-t, so I'm not going to load that just yet. All right, load module is not allowed here. So that is the thing we added up top that was based on the pro doc. Do I have to load anything here, or is the module loaded by default? It should be enabled with the configuration parameter. What is that, the systemd config file? I'll get a different error, right? Yeah, unknown directive. All right, so we're not. chamlis_ I think that's a compilation option
frici this says its not built by default, but its built for you
What if I said with HTTP Geo IP module? Can't do that. All right. chamlis_ it's built as dynamic, so you'll need to load it
You think that's a compilation? Hmm. But if it was built, I'll need to load it. OK, how do I load it?

02:10:33So I wrote this. What if we said bin? Presumably, it's going to have that name, is what you're saying. And then I'm probably going to get that same error that I can't have a load where I have it. That's fine. So let's move it over to the main config file, which is in conf.d. No. Where do I have a load module?

02:11:27And we have this module available, module enabled.

...48Anybody remember the apt command offhand to list all the files in a package? I'd like to figure out where nginx is.

02:12:10Yeah, so this says.

...51frici apt-file list $package?
chaelcodes So, we've been making quite a few changes in production... that seem difficult to test.
chamlis_ https://ubuntu.com/server/docs/…
I have to install the package to whatever.

...59frici dpkg-query -L should do similar as well iirc instead of installing something new
We're installing Perl. We need Perl for that?

02:13:18chaelcodes I know we were joking about it, but why are we doing it in prod?
We are doing it in prod so we can run nginx-t because I do not have nginx locally. And this is going to depend on things like prod paths and prod modules that I don't believe that I can immediately match up. Lobsters as a hobby project does not have an entirely separate staging environment that tries to be a clone of prod.

...48chaelcodes Okay.
Therefore, YOLO. I should have done this from the start. I'm wondering if we even have the module built or it's... Let's look at Shamless's link. When all else fails, read the docs.

02:14:30chamlis_ oh wait sorry, probably install https://packages.ubuntu.com/nob…
I don't know that these are.

...50Yeah, I think these are built in because when I run engine X dash V, it lists it as here no no here we go with hp goip equals dynamic

02:15:55chamlis_ that package includes the so file, so I guess it doesn't load dynamic modules unless they exist?
Really?

02:16:32gtfrvz try the load_module directive in your nginx.cfg and not your lobste.rs.cfg
So I can't load module there. Unknown directive still. chamlis_ and then I think you can symlink the file from modules-available into modules-enabled
OK. You think up top here, outside of any section?

02:17:05gtfrvz yes, on the same level like events block
Not there. Not there. On the same level like the events now. It said it doesn't like it. Or no, it actually didn't. Oh, I missed it. The error changed, and it's not the load module. We're just back to the GOIP is not allowed. So do I have to move that? Does this go up here? No, it does not. All right. I'm going to run to the restroom real quick so that I can think. And then we will keep hacking this out.

02:20:20Oh, I kind of hoped that by the time I made it back to the computer, Shamless would have been like, oh, here's a worked example. He's usually pretty good at finding those resources. All right.

...39chamlis_ are your DBs GeoIP2? I didn't realise that was different and that would need libnginx-mod-http-geoip2
So it doesn't like the GOIP country wherever I put it, huh?

...52Before we get too far down this.

02:21:02I thought MaxMind database and GOIP2 were synonymous, but maybe I'm wrong.

...32chamlis_ ahh then I think that was the wrong package earlier
You think that was the wrong package.

...39All this we're running down the wrong thing I was going to grab this and put it in the answer before I forgot.

02:22:06chamlis_ that same package with a 2 on the end might be the one
Yeah, so this used MaxMindDB in a synonymous way. Okay. Let's add the two.

...31Come on.

...44chamlis_ and then I think rather than editing nginx.conf you can symlink the .conf in modules-available to modules-enabled
why did you apt apt you prick it ran in nginx reload did that just knock us offline okay good you never heard of installing things that didn't

02:23:38frici right... debian/ububtu likes to start/restart/services whenever it feels like ... I keep forgetting they keep doing that even if people tell them no its a bad idea.
I'm not seeing anything that looks like a named nginx module.

02:24:06Type.

...24So we're not getting the load module error. gtfrvz 2
And there is a modhtp-goip2.conf.

...44chamlis_ maybe also in /etc/nginx/modules-available/mod-http-geoip2.conf
And that has the load, which is the line that I reproduced. I think rather than link it, I would rather reproduce this load module. It's less pain with our Ansible setup because Ansible claims to be declarative and very much is not. And so if I ever delete a file, I have to create a task to delete the file and then run it and then delete that from The Ansible config, it's just this whole chore. gtfrvz You're now using geoip2, so add 2 again i thikn
Whereas if I add a line to a file and I delete a line from a file, it's straightforward.

02:25:40All right. So now we just got to figure out where it wants this GOIP call. So I have our browser. If I swap GOIP for GOIP2, I get a weird 404 page.

02:26:36gtfrvz only 2 in module, not in db loading
OK, this syntax looks quite a bit different. So let's go to here. And it's in an HTTP block, which means it probably will work enough. So we'll say GOIP2. And then grab that path and not retype it so I don't spell country wrong again. And we'll say auto reload. How often do we want this to reload? I'm going to replace these monthly, so I think probably one day is fine. And I only care about GeoIP2. Let's do this. We're going to do that. Look at me not adding typos. So I only care about the one field. We will imagine that people are in the US by default. The source is, what is this variable with IP?

02:27:44What is it trying to say?

...50chamlis_ you can put $remote_addr etc in there
gtfrvz $remote_addr?
Oh, but it said, it's the thing it was saying about based on the client IP or from a specific variable. So we can just throw this part away. Yeah. I'm just going to throw it away and see that it, because these other ones, they don't include it. So I think it's just saying that there is an example of how we can override. chamlis_ > If source is not specified, $remote_addr will be used to perform the lookup.
Then we could make these fastCGI parameters. So if I really had wanted to, I could pass this down to the Rails app, but I don't want to interact that. Yeah.

02:28:48All right, so I'm going to close that so I can see what I'm doing. Let's say if I have GOIP2 data country code. Ooh, come on, apt. GOIP2 directive is not allowed here.

02:29:23This is this is a location block. All right. So let's move it up to the. HDP blocks, we got to. Where do I have that? I have that up in the file I just closed. I don't have a separate HPS block, right? It's just talking generally. Yeah. So let's put that up here.

02:30:06Hey, look at this, folks. Nginx is happy with this.

...20chamlis_ :D
Okay.

...28Nginx is practically a... Hindley-Milner-typed language where if it compiles, it works. I joke, but this is short enough that I actually feel pretty good about it. So it's G-O-I-P. Let's make sure I typo it. Yeah. Yeah, so those are the same.

02:31:05So we grab the country's ISO code to this variable. chamlis_ if you want to stick something in a response header I can test that from the UK
If it is not GB, you pick up the W. This seems legit. Sorry, some kind of gnat here. I hope the compressor kept that from being too loud.

...41I want to stick something in a response header.

02:32:25Don't remember these. Yeah, I think I can just say like that. All right. frici incidentally the bio message is still there
Who's feeling lucky? Shall we reload nginx? The bio message is still there. What's the bio message? frici the bio break text source
Oh, oh, the taking a break message. Yeah, let's turn that off. Thank you.

02:33:10frici and I guess at thiss point we can only test by restarting nginx yes HahaSweat
Sorry, it's fairly small on my little review and I didn't notice it. All right. I feel pretty good about this.

...26Well, I don't have to restart. I can get away with just reload. So if I curl from the server itself, yeah, I believe this is in CA. chamlis_ I see "x-geoip2-data-country-code GB"
pushcx curl -I https://lobste.rs
All right, anybody in the UK and want to run this command?

...53frici chamlis_, said they are in the UK i believe
pushcx curl -I https://lobste.rs | grep -i geoip
That's not the thing I wanted to copy. Where are we?

02:34:02Yeah, Shamless, did you say you were UK?

...10chamlis_ yeah I see GB
Oh, CA. I'm thinking CA, the state. The server is not in Canada. I wish you could meet my server. My server who lives in Canada. UCGB. frici didn't we find last time that the database is slightly inaccurate sometimes ?
Great. Well, if you see GB, then you don't see the cache. Yes, we have definitely found out last time that the database is slightly inaccurate. And I did more testing with folks who are in the chat room and It was pretty good for desktop IPs, but then someone had a mobile phone where because their mobile provider is doing something clever, it would be like every third page load, they would see the message or not. It was not predictable because they were routing different messages very differently.

02:35:26So.

...35All right. Let's stop jobs. Let's log out of root. And get these files back into the Ansible repo.

02:36:21If those are back in and I already added the missing package, then we can look at the full scope of this. This feels pretty good. chamlis_ (maybe don't leave the header around before people start using lobsters as a geolocation API?)
This looks reasonable. Okay.

02:37:04Oh, the header.

...10Yeah. Yeah. The header is useful for, it's useful for a lot of things. But if I don't delete it, I'm never going to remember. So, okay. Yeah.

...57Let's push that up to GitHub.

02:38:09And let's run the playbook. This Ansible playbook run has been brought to you by UV. So it's mad about not having the right password for DB01. That's fine.

...29But I just want to run it and see that it spits those things out. And I'm going to bounce nginx again in prod because it's so easy for Ansible to slip out of sync. I could have run just the nginx role, but there's also the pull request and such that I merged. That stuff ought to go out at some point. Why not now while I'm paying lots of attention? Yeah, so this... Thank you for all the help, folks. Shamless for G and GTFRVZ. I always have to look at your name. I don't have that memorized. But I appreciate that you watched over my shoulder and convinced on this, because that would have actually taken me significantly longer to do myself. gtfrvz it's easy, just add an z to gtrfv
The difference between regular and NGINX Plus, that would have chewed up a half an hour. All right, so we ought to see that it's bouncing. frici good point
chamlis_ haha
Oh, what a great mnemonic. Thank you. Is that like your initials? How did you come up with that handle?

02:40:06frici their cat named them during adoption
I feel like I've asked this before, but I sure don't remember the answer.

...15gtfrvz i used gtrfv for twitch just hitting the keyboard. but then i lost my login so i added another letter
You know, that actually... Fritchie, you joke, but if they're on a QWERTY keyboard, GTFR are all close together?

...31gtfrvz technically i have that old account back now
So that is your mnemonic for your name? Oh, GTR, no. soulcuttr could be worse lol
You typed your own handle wrong, GTRFV, or I'm reading wrong. frici yeah they typed their own name LUL
Yeah, that's not actually just a Z added back on the end. frici typoed*
There's a big meme running around on Blue Sky where people explain how they got their usernames, and mine is x86 assembly was my second programming language. And then my maiden name, Herkins, was common enough that somebody took it as a username before me on everything by the time we hit 2005.

02:41:28I tried to do the old school hacker thing of your initials, but I didn't have the three letter initials.

...45soulcuttr I have the gmail for my real name, and I hate it because I occasionally get emails clearly intended for other people that share my name
Why am I ahead one? Oh, I added that spec. Let's add that. What a bit of a slog. soulcuttr I barely use gmail these days, but still annoying
Can we just take a second to admire how ridiculous it is that a law that shouldn't apply to the site caused performance problems two months before it took effect? chamlis_ rule britannia
frici I get my own name taken from me usually if (assumingly) a hungarian comes on the site/service first. since Frici is more common in Hungary than other places for diminutive instead of say fred/freddy/similar.
Of all the ridiculous root causes for a bug, an unpassed law or an unenacted law in another country causes the site to slow down was not on my list. Oh, these are great. Yeah, I have a old coworker named Derek Willis, and I forget what his Gmail is, but it's common enough that other people named Derek Willis, which is not a rare name, or maybe it's even D Willis's thing. He gets lots of crap on Gmail for other people.

02:43:05Hmm. Oh, Fritchie is based on, it's a diminutive for Fred. frici for Friedrich
In what language? I mean, Hungarian, obviously, but it sounds the way you're phrasing that maybe you're not for Friedrich. Ah.

...29Okay.

...52soulcuttr My name / spelling is fairly rare at least. There's a Louisiana Astronomy professor and a blue collar guy
Oh, for those kind of doppelgangers, the absolute best one was on My Maiden Name. I mean, you know, it's not objectively the best, but I loved it. On Star Trek Voyager, there is a Commander Peter Harkins. Guy was in like seven or eight episodes, but the... The Star Trek nerds interlink so much that they had all of the page rank. And so I couldn't... Two episodes. I swear he was in more. soulcuttr I don't want high page rank - I cede that to the professor
chamlis_ chamlis is a scifi character, nothing original
Maybe I'm just blowing it up in my mind because I was excited to see a TV character with my name.

02:44:48soulcuttr it is cool to have a Star Trek character
chamlis_ the star trek wikis are a true pinnacle of human endeavour
But yeah, for the longest time, if you looked up me, you found this guy. And you know, as I get more middle-aged, at some point, I just gotta get the yellow turtleneck and I'll be able to cosplay this guy.

02:45:16soulcuttr you _have_ to cosplay that guy. The best costumes are ones you have to explain
Alright, so... So let's be overeager. I was thinking about this fragment caching. The best costumes are the ones you have to explain. Yeah. I mean, people would just assume I'm generically cosplaying a character. Because on Voyager, it was like everybody had the same costume and a different color turtleneck underneath. So I could be cosplaying anybody in engineering. I could be B'Elanna Torres. All right, logged out as root. Can close that. All right, let's... So I was saying that that fragment caching... See, the one thing that... soulcuttr One of my recent costumes was a Counter from the korean tv show Uncanny Counter
The place this doesn't fit really well is I actually don't want to cache for users who are logged in. soulcuttr zero recognition
One of your reasons was a counter from the Korean TV show Uncanny Counter. That doesn't mean anything to me.

02:47:05Ah, paranormal hunters who hide evil spirits from the afterlife. Legit. And they look like they all wear a shirt with two stripes. soulcuttr it's a track suit with two stripes. Easy costume
Alright. Snazzy.

...30So I only want to do this for folks who... aren't logged in. Because if we see a couple thousand, if I put the user ID in the cache key, here we go. Cache if.

02:48:02So what if, all right.

...11let's enable that in dev and then oh it already was restarting if i enable that in dev then on the page where's the comments so this section

...43And then where's the story display at the top? That's list detail, right? How are you not calling list detail? It's missing. Ah, it's list. Really, Rails? So Rails does some file name munging when you're looking for partials where it slaps an underscore on the front. But I swear list detail is list underscore detail, which is a... Oh no, it's not. Why did I think there was an underscore in the middle?

02:49:32Maybe I'm thinking of form underscore errors on the story editing. That gets rendered a bunch of different places. Hmm. Man, I'll take it. So we got the list detail up here. So I could throw

02:50:04I lost track of where I was here so cache unless all right so we want to say cache unless the user is present and then we want to base on so the hassle here is I want to base this cache on the last updated at on any comment on the story and we don't have yeah i'm not going to be able to hack this out in five minutes all right so there's because there's this whole thing you have to do with touch true any action for a game this is going to make our deadlock problem worse where So the problem that happens right now, we see these deadlocks and they got worse when the server slowed down. I don't think I ever finished that thought. It used to be like a once a month thing, but it's happened once a day since I turned off the page cache. The database deadlocks when you are creating a comment because it's locking tables in a different order than someone who's upvoting. So when you vote, You insert a record in the votes table, and then that also has to update the hotness on the story. And when you insert a row in the comment database, you also insert a vote, and then you also recalculate the story. And something about those two flows is causing it to happen in a different order where Like, upvote wants to lock story and then votes, and comment creation wants to lock votes and then stories. And I don't know exactly what the sequence is or which one is doing which tables in which order, but we're getting infrequent deadlocks and not quite enough debugging to run it down. And I feel like this is just going to make that worse. Which touches on. thing I wanted to mention.

02:53:47So I was going to say, did I already write it up here? Yeah.

02:54:22What's the comment method? Is it confidence? So the way we rank things is based on an algorithm from Reddit that tries to estimate this idea of confidence, which is based on votes. But you want to be a little sophisticated about it because If someone, if a comment has one upvote, well 100% of its reviews are positive. batman9459 no syntax highlighting is mad
But that probably should be ranked behind a comment that has 10 upvotes and one flag, because that comment is probably better than a comment that only has one thing. And this algorithm is less applicable to us because we've slowly gotten rid of down votes.

02:55:31There's not no syntax highlight. You can see the spell checker here is concerned about these terms.

...44Why is this one? Why don't you like the word confidence? Maybe it's just too long. I don't know. Yeah, it's trying to highlight long line maybe. There is some syntax highlighting, but it's real minimum. So like this italics on comments and string literals is syntax highlighting. Knowing that something is a method or a variable is actually not useful for ruby and you know putting def i don't need angry fruit salad it's not going to tell me anything all right and comments calculated confidence into item potent database queries so we'd call something like story dot So we'd have these code paths, insert votes, and then call story.calculateConfidence, which would issue a minimal update that would need to lock tables data is read and written back in a transaction

02:57:43So the reason I got here from that caching, yeah, is that then, so what's this, 1238?

02:58:24Forget that number.

...55Comments belongs to story. Added a touch through.

03:00:34still show, and around. I didn't actually look at the other view. View home index, yes.

03:02:06Let's see. March down.

03:03:37what could we actually do if they changed so like if instead of entering two not two new lines in a row does that make two br tags or does that make a paragraph tag like that would cause the test to fail that's pretty harmless and there's nothing to do different i guess really i'm thinking of Markdown is all about just keyboard smashing punctuation together and if some combination of asterisks and underscores changes meaning we change comments. But then we can't really revert.

03:04:59Other thing is...

03:06:55Big brown keyboard. So this is just kind of what I'm thinking of. Where is fragment caching going to be the most useful? And biggest bang for our buck is our two big hot paths. And then once we have that, also we can avoid dogpiling by pre-filling the cache. once a minute that would be nice and then as long as we're not going to take on oh god what is markdown going to do over there okay that would be nice if we have this that might be a big performance win enough to just totally delete the page cache, which I should mention.

03:08:02Currently do full page caching of the action pack. What is it? Dash or underscore dash?

...42all right all right that's actually that's a good write-up i don't want to actually a really clear chain of things where we'll do we'll fix the deadlock by

03:09:14this method that's been hanging out on my personal to-do list for ages of moving these calculations into the database.

...29All right, so it's a multi-step process, but it's pretty straightforward. I don't know what all this sub issue stuff is. It's all very new to me. Is there a document here somewhere? sub issue okay i mean i have enough issues good i suppose it's useful for things like the rails 8 update I wish I could make this remember my sorting.

03:10:19Yeah, I can't just click to convert. Oh, OK. So the ones that aren't finished can be converted to sub-issues? I'll have to read about. GitHub sub-issues at some point. Because that's clearly a new thing. So I'll put that off screen. Come here. Get off of my desktop. And then... Alrighty. So then the last topic was...

03:11:03hotness to deadlock to fragment caching chain of issues it's always funny like i just spent what 20 minutes half an hour writing that up and especially if someone wants to take this on twitchtd what version of mariadb are you using
half an hour not coding, just figuring out what the heck something is in its dependency chain is actually pretty darn productive time spent as opposed to what you saw me doing, which was starting to add the cache call and then realizing, oh, this touches on the other stuff. I try and avoid starting like 19 versions of things.

03:12:02pushcx mariadb Ver 15.1 Distrib 10.11.8-MariaDB, for debian-linux-gnu (x86_64) using EditLine wrapper
We are using version 15.1. Here, I will just paste this here.

...20twitchtd ty
Why do you ask?

...29twitchtd just curious
So I know when the Rails folks grabbed lobsters to use as one of their performance tests, that they swapped out and used SQLite, which mostly just meant commenting out some of the aggregation queries that we do by hand in mod controllers. and the replying comments view that we kind of want to get away from anyways. soulcuttr hours ago I started adding fields to an API. I am _still_ freaking working on it. It's a 2 line change to the controller. It's like a 500 line change to 50 files in the test suite. WTF
And if it was that easy, I would probably be happier if we ran Postgres in production. It's a lot more featureful and includes a few surprising things. Wow, that's a big change. Soul cutter. It's funny how those things spiral. All right. So that's, that's really everything I wanted to do on stream. And I think I'm going to wind down. So this has been Lobster's office hours. If you have any last questions about the site or the code base, Now is your chance to get them in under the wire like Twitch TD just did, asking about our production MariaDB version. Sorry, the cat did not appear on stream today, but even so, this was real important. I have, you can't see it, but I have HTOP running off stream and it's gone from pretty much all of the CPUs on prod were pinned pretty close down to our normal, it ticks along at like 10 or 20% on average. So the page caching because of the law is actually working out quite well immediately. All right. chamlis_ thanks!
So absent that, I'm going to roll out, go find myself some lunch. soulcuttr (╯°□°)╯︵ ┻━┻
I hope you all have a good rest of your day. Thanks for joining me. The next stream will be on Monday afternoon at 2 p.m. Chicago time, where hopefully by then Soulcutter will have finished his two line change to his controller.

03:15:20frici thanks and take care !
And hopefully by then, Ofcom will have got back to me about, oh, of course, we would never imagine that we're the boss of the world. Haha, can you... I'm going to step on that and not be spicy. We'll see how that goes. All right. See you all on Monday. Take care.