In the U.S. we have a system of mandatory football

03:45Happy Monday. It is time for Lobster's Office Hours. And I'm Peter Bott Harkins. I run this site, Lobsters. This is a good time to drop in and ask anything that you would like about the site or the code base. We can even run queries against the prod database and logs as long as we're not barfing up personal info or making worst of lists oh and here is a little feature that i hacked out what two streams ago one or two to make it obvious when there are stories merged in oh i didn't i gotta hide the stream for a second i didn't put up my usual Going live, tweet, and whatever. Blue sky post and Mastodon post. There we go. All right. And then I guess the other thing to do is turn on the footer. So let's take a look at that in a second. Why is the footer on the left? That is... So there is the footer. That's a brand new thing. On Friday, I had hacked out the materialization for depth and reply count. When running the migration, the code was not in sync with the database, which made it pretty mad. All comments were just flat there was no tree of comments for about 90 minutes because migrating the comments table is slow. and So I threw up a header up here that said something like. You know sorry about that we know about the bug drop by irc. And I don't know a couple dozen people came by irc maybe up to 100 I wasn't really watching the numbers, but it was a noticeable amount of people drop by, especially for a Friday afternoon. And so then I thought like, well, also on Friday, somebody had, or Thursday, somebody had come by this stream and been very surprised to find Lobster's office hours. And so I was like, well, these two things are related. So I thought, let's add a Twitter that mentions that office hours is live when it's live. And I keep highlighting that and I left a tweet about it and I mentioned it in the chat room and, I just keep talking about it because it's just kind of awkward around anything that smells like self-promo. But I also added it to the About page here. So what was I going to look at? I was going to look at that CSS, because when I tweaked this CSS, I wanted this to stay on the right. So let's get that fixed right now before I really jump into anything. So folks can feel free to pipe up any time with questions and comments. But it's still open, yeah. So this was spaced between... I always have to check the cheat sheet for Flexbox, because I can never remember its properties. twitchtd oh, caught the stream early today :)
What is that site? There it is. Found the URL off stream. manateemon o7 good evening people
Oh, hey, Twitch. Yeah, I managed to. twitchtd o7
So I have a whole checklist of posting a tweet and posting a Mastodon post and editing the stream description and all that. And today, I actually ran through and hit all the checkboxes. So what I wanted was that by default, things should be on the right. So justify content. So it's spaced between, but if there's only one thing, it goes left, huh?

07:59manateemon justify-content: flex-end for one item. I think
I don't want a line.

08:26You know what I should just do? I think I can just have it say, if I flex items from the right, when there's a second item, like if I swapped the order, it would end up putting it, oh man, flexbox. Right, so mantiamon, the problem here is that there is sometimes one and sometimes two.

09:01So here's the CSS on the left. manateemon Ah, I see
So if the file exists, I guess I could just swap these. And then the item is always there, but then usually doesn't have a...

...20I guess that would work, right?

...26Then it stays structurally the same. So let's check local host.

...37Okay. So that exists. And then I remove the file name it's checking for and reload these stay on the right. And then let's double check the small view. Good. What is this? Let me touch the file again. That's... The centering is weird. Why are they centered? Where did it get that text align from? Oh, text align is used for... Wait. What are... I was thinking to center the big page div. And I don't even see that in here. Oh, it's down in the, this must be down in the media.

10:53Okay, so we'll remove that. And that's a no op, because now there's no more float used for the footer. That's an acceptable ugly, I guess. I guess let me make the nav right align, and we'll call that good.

11:24That seems fine. All right, so with that fixed up,

...59manateemon a slight increase on the line height could probably maybe be useful for touch screens. But I might just have fat fingers haha.
That's a good call.

12:09So that's just this. And then let's look at what the line height is now. But I look at that, and I think you're totally right. So we'll grab you. See, this is pretty close. But when it's two links next to each other, so where is this guy getting its line height? The main footer is overriding it. So let's just get rid of that. Yeah, so here's the main footer. If we get rid of the line height. And actually, I don't think we need an explicit height either, because we're not floating that anymore. Keep the padding. Nice. That breathes a little better. Good call. Thanks, Mantiman.

13:15manateemon Ahh, I can breathe again :)
Oh, I didn't mean to add that. So let's... There we are. Toss that out. Oh, and I can see that didn't use the right indenting, which I have to Castle people about regularly, so that would have been embarrassing. Did I already? Yeah. So let's go. Ooh, that's exciting. I wasn't planning on spending much time here on stream with pull requests and such, but let's see what my Dependabot pull request says. All right, so Ansible's deploying the site, and I wanted to peek at this off stream in case it was some kind of awful remote code execution, but I I'm not too worried about people submitting sites, submitting links to malicious servers in the next 10 seconds.

14:47That

...58That doesn't net IMAP. Oh, this is IMAP. Someone would have to email the site.

15:08No, they wouldn't. Why don't we even load net IMAP for? I can't imagine a code path that would get somebody to this gem. yeah this is probably not a code path we use because we're not using net imap to host a server but we'll bump it regardless and that is exactly what Dependabot was saying, and we'll grab this ID.

16:29Yeah, I'm not sure offhand if it's even possible for us to hit that, if it's like hosting an IMAP server or if it's as we're sending email. Either way, easy enough to bump that out. All right, so I don't need this. So that's the thrilling excitement of office hours. If nobody has questions, I hack on the site. Sometimes that's CSS bugs, sometimes that's depend about issues. This is the very exciting janitorial work of running lobsters. pushcx https://bsky.app/profile/rubyce…
Where's my scratch? Oh, yeah. I had an announcement I wanted to share. Let me grab this link here. So I figured I have some people who like Rails coding on the stream, because that's what I'm doing. And Ruby Central has announced that the CFP is open for RailsConf. So you've got 18 days, just over two weeks, to submit a talk. I talked at RailsConf years ago and it was a very well-run conference. They were very friendly. They wanted to make sure speakers, especially first-time speakers, had a good time and put on a good show. If you are watching this stream and you would like me to review your CFP proposal, send it to me in the next, I don't know, 10 days. You got to give me a day or two to see it and respond rather than, you know, 1130 on February 28th, you're not getting an answer from me. so I've previously reviewed talk proposals for Madison Ruby and I've spoken at a bunch of conferences. So I know roughly what I'm talking about. And, happy to help people get their talk proposals accepted. It's a lot of fun. If you've never spoken at a conference, a little scary if you don't know public speaking, because that was my first big public speaking and they put me in the big room, but a lot of fun. pushcx https://github.com/lobsters/lob…
It's a great excuse to deep dive a topic. Speaking of deep dives,

18:56manateemon Are you American? I feel you as a people are very good at public speaking in general
So a year ago, oh, almost to the day, I added prosopite for catching 1 plus n errors in our SQL. manateemon here in Sweden we almost never practice it in school
And we have been real happy to use it. Us as a people. AnakimLuke what's a 1+n query?
Yes, I am an American. I live in Chicago. yeah we do it is really common to practice it oh hey anakin i'll see i'll answer in a second if you click on this link it'll tell you too but it is really very normal in american i guess you'd call it primary school but manateemon "as a people" was a bad way of phrasing it, dont get me wrong :)
in classes to be required to give presentations to the class as if you were the speaker and they're short typically especially when you're a kid they're they're very young so like at the ages five and six it's called show and tell sometimes where you just bring in something you like like a favorite toy and you explain why you like it oh no i get where you're going it's a it's a cultural thing and i'm not assuming that you're making some kind of weird generalization no so yeah there's that and then before i spoke at conferences i spoke at local user groups and it was really easy as an on-ramp from AnakimLuke do you think they'll let me talk about my fav toy at railsconf? 🙃
high school and college into speaking at user groups because you know i had talked to 20 people before in class so talking to 20 people at a user group was like all right it's more of the same that's a lot easier because people are a lot more interested in what you're saying at a user group than board students didn't link to the that is weird github What is this markup? Oh, I just said the word prosopipe. I swapped these around. That's what it is. manateemon Solid advice. Heading out for a while, hopefully catch the tail end of the stream. hf!
I will never stop being slightly confused by these because in anchor tags, the link comes first and in markdown, the text comes first.

21:30So, Anna Kim, I don't know what your favorite toy is, but there are regularly talks that are, this is my favorite toy library. Oh, cool. See you later, Manitima. Thanks for that CSS help. So, Anna Kim, a one plus N query is when you, so as an example, like, you look up some record and then you find the associated records. For us, a thing that happens often is if you look up this list of stories and you want to also print the username of people, you have to hit the user table. So it's very easy to have one query to find the stories and n queries where n on our page is 25, 25 queries looking up usernames. And when you're database like ours is on a separate VPS, you get killed by the round trip overhead for the network. It is the big foot gun in Rails, I would say in Rails performance, where it is very, very easy to add them because of the way active record works. And the API yet presents and doesn't present like the your classes map very closely to tables rather than being any projection out of SQL in the relational algebra sense. A couple of weeks ago, Byroot, who has dropped by the stream before, I guess he maintains the lobsters using widget or I'm sorry, the widgets fork of lobsters for use in their performance suite. And they're having a really neat thread about that. So where's my jet. So if we find this blog post, you know, let me just pull it out of my personal browser.

23:58pushcx https://github.com/Shopify/yjit…
There it is.

24:09All right. So because lobsters is open source and has real world complexity, it is used in Shopify's benchmark of why JIT. And this is a super interesting thread. If you're interested in why JIT or, you know, the general topic of JIT and Ruby, and it has a couple of interesting trivia things. hono4kami Hi, good morning! (it's like 3am)
Even if you're not super into compiler details, because I couldn't tell you much about how YJIT actually works under the hood, I just benefit from it. And we've done some, oh yeah, so he pinged me, 3 AM. Well, good morning, Hono.

...55So let's see. he caught a performance regression because they updated the code base they kind of forked and pinned it so that they could have a stable thing to benchmark against and caught what they consider performance regression so i offered some notes about how we're thinking about our database and our view layer So if you saw, if anybody saw the streams with Joel Draper, maybe a week or so ago, we didn't get to the point of actually benchmarking that, but that was a fun experiment. I don't know if Joel's going to be interested in finishing enough of that benchmark work that we would have to do. So Where was it? So he found some specific issues in different tools that we use that are standard Rails stuff or standard library stuff for Ruby. And then our replacement of the ERB that mostly generated invalid RSS with a DSL that produce is valid. Ruby, it is much slower. I think there was also, I actually learned of Flex from a Lobster's thread where Beirut and Joel, Jean and Joel talked about performance in Ruby DSLs and they disagreed pretty fundamentally. So I think this may be a little bit of that continuing. The

26:55Oh, he noticed that I shoved a lot of work into the database to try and make things run faster, like this monster that's come up on stream more than a few times. And then...

27:11So, Jean wasn't sure whether to update because pushing more work into the database means less time is spent in Ruby code, which is not so useful for Yjet's needs. And then Maxime here said, on the other hand, we kind of need as much Ruby code as possible to focus our optimization efforts, I feel. When YJIT team members try to implement small optimizations, we look at performance differences on lobsters a lot because it's the biggest benchmark. If the difference is below 1%, it's hard for us to make the call. So that was really neat. I didn't realize that lobsters was getting used that frequently for what are otherwise going to be fairly small code operations. Lots of fun Ruby performance details here, if you are curious about it. And this is open, so if you know lots about YJIT, you could probably say something smart in this thread. I can't.

28:15So, back here, Jean mentioned that... Hey, did you notice that Presopyte adds overhead even to production. And yes, we did know that it was on in production because I have it logging one plus end queries. And he pointed out that there is this feature called strict loading that doesn't get a lot of attention in the guides. But if you try to say, if you load a user strictly, so you call this strict loading before the query method, whether that's for an individual record or multiple, and then you try and walk associations, you get a raise, which is not very different than the behavior we see now on a prosopite in development. And I was kind of stumped on how to actually implement that though, because Naively, you would have to just find every single mention of looking things up by user. And always remember to add it, not by user, by model. And so I can sort of imagine a cop, but not a reliable one. And the query API is enormous and broad. And then Biroot suggested making it a default scope on a per model basis or just right on application record. And default scopes are such a foot gun that I was really reluctant to use them. And I ended up starting to write a initializer that would turn them off so that in the development and the production console, I wouldn't be hitting my head on it because you've seen how many times on stream probably I've looked data up and I've done user.comment.something. And while I was trying to figure out how to customize the error message that users would see, or I should say new developers would see, I found an undocumented option called strict loading by default that does this, but without using the default scope functionality. So it's a separate way to do that. And I thought it would be neat to figure out if that's something I could actually enable.

31:02Because I've really liked press a pipe. But if we can drop a dependency, especially if it gets us a small performance speedup, that's pretty nice. Actually, all. No, this does get used to improv. All right, so handle JLo. And then.

...48So the ActiveRecord stuff is roughly together. Actually, let's see. I'll leave it. I'll turn it on in dev.

32:17Yeah.

...29But I'm not going to turn it on in prod because I don't need to watch 500 rolls in. That's environment and test and environment prod.

...52Nothing about active record in here. I guess I'll just put it at the end. Never know how to sort this file.

33:13these two, I guess.

...29Let's say only in dev test.

...43Looks pretty good. Let's see if we can turn that on. What's weird is I read, oh, we've got some kind of error there. Server immediately exited. Oh, because the Precipite code is hanging around still.

34:13So if that's going to boot, I'm going to go ahead and run it in that background tab for Vim. And then I guess the next obvious question is, do the tests run, right? Because this has slightly different behavior than Precipite. The tests do not run. Oh, my God. That's a lot. Okay. That's just every single thing. Oh, so Precipite has a... I should have thought of this when I got rid of the initializer. You kind of have to shotgun statements to permit. Oh, this is gonna fail, isn't it? In dev comments where... yeah and here's a difference we can't unless i can turn off strict loading this is a non-starter because i don't want to refactor all of this code right this second so this is the very infrequently used code path for when users delete their accounts or get banned and so it's fine that there's a one plus n in that Getting rid of them is mostly about getting them off of the home page. Yeah. So is there a way to? So if I can say strict loading values equals false.

36:34One in domain. Oh, yeah. Where clearly I learned the block syntax.

...51Let's see. I don't know if this is actually going to work. Because this one's on the association rather than on self, I guess, is where we want to call it. Yeah, let's see if I can move that up. We can clean that out. And then here, oh, this thing for finding users.

37:48huh that's actually interesting because of the way this is looping and finding users repeatedly strict loading can't find that because it only prosopite looks at all your queries so it can find you know

38:16PrestoPyte would highlight this as inappropriate. And strict loading can't find it, because strict loading is only about associations.

...34All right, so that's all of those matches. Let's see if we have anything else failing. I don't know what this is, since that big hassle with... Okay, so this looks like a legitimate failure. So let's try and see how much of the suite is going to throw errors. A lot of the suite.

39:19One more.

...41So that code I deleted, I need to not reference it. All right, now let's see how much of the suite

...53This has got to be more of this config stuff. No, it's not. OK, so these two are the change I just made. And whatever this is in updateStoryText, let's find that.

40:46Something I'm wondering is, is strict loading a method only on the class? In which case, we're in trouble. Or is it also on the instance?

41:11Let's find my user and say.

...34Okay, so instead, it's not a chaining method, we just turn it off individually for the record. So it doesn't have the API as I assumed.

42:24So this one, let's try that.

...39So that's still loading.

...46And it works. Okay. So that one's fine.

...56AnakimLuke so, it seems to me that n+1 queries only happen when delegating responsibility to an ORM? I find it hard to think of how it could come up by writing bare SQL. does that make sense?
This one should be fine because it's that same pattern of calling it on the association. This one's the one I just did. This one needs to look like what I just, no, this one could be over here. N plus one queries.

43:21I see what you're asking. happens anytime you have an abstraction layer about SQL, whether it's a ORM or a data mapper or any place that you can put a while loop and fire off queries. So if you have external control structures, you're going to have the risk of one plus n queries.

...58But there are things that database libraries can do to make them less common. So some of those got fixed.

44:22Why would these things all start failing?

...33So strict loading works really different, and it's any time you access an attribute, there must be a rescue in this code path that it's falling into. Yeah, strict loading.

45:02Even if these aren't n plus one queries, especially with tests instantiating objects, the instantiated objects are all, yeah, this is,

...41Yeah.

48:27Snacks have to be big.

49:17Does, do we get anything better? Not really.

50:37We actually can't say it's from the controller level because it's worse. We'd also have to do it from the jobs, from the scripts, from the email inputs, from action mailbox. Yeah, so the top of the control flow is not, there are so many entrance points for that kind of refactoring.

51:57I should have mentioned that.

52:35I wonder if the other way to get rid of, I guess to, yeah.

54:00Yeah, so if Lobster's had a lot more dev time, that would be more plausible. I don't know. All right, so there's the end of that topic. So we're coming up on about an hour. So I will remind, this is Lobster's office hours. If you have any ideas about stuff you'd like to talk about, questions about the site, moderation, pop up anytime with questions.

...35AnakimLuke !lurk OhMyDog
So there was this interesting bug on Friday. dpk0 what’s your latest update on plans for dealing with the OSA? still expecting to block UK users five weeks from now?
And... See you later, Anakin. Oh hey DPK, welcome back. So the current plans for dealing with the OSA... I think my last comment... Yes. pushcx https://lobste.rs/s/ukosa1/uk_u…
So I posted this on stream a week ago. The...

55:23I contacted Senator Rod Wyden's office because he's done a lot with internet law in the United States. There has been no progress there, unfortunately. Not even an acknowledgement. It could be because I'm not a constituent. It could be that they're just very busy with other politics. AnakimLuke lurk means the person have the stream up, but is working on something else in the meanwhile
So... I'm out of things to try myself. I recently... Ah, okay, Anna Kim, I thought it meant you were leaving. So where was I? In dev mode here, I did just yesterday tweak this to say more explicitly, please ask your MP. And I gave a link to where people can look up their members of parliament and do that. And I've tried to use this language of what's going on with the OSA which seems to be activating. I don't know. dpk0 i wonder what will happen if Parliament does refuse to reject the rules
hono4kami Oh wow, UK seem to be serious about going forward with OSA, huh
I'm a little uncomfortable with this because I don't want to tell people how to run their politics and what their country's politics are, but at the same time, it's now the only way out of this mess. The

56:49Yeah, and I linked on a previous stream here. I don't know, Hono, if you clicked on... Oh, I didn't put it in here. So on the stream where I wrote that comment, I linked to...

57:17Where is the damn link? Was it not that one? pushcx https://push.cx/stream/2025-01-…
Was it before that one? Before that one. Okay, so here, let me grab this link. So you have these. dpk0 doubtless the supporters of the law (whom you’ve mentioned before include some real pieces of work) will launch a judicial review and allege a suspending and dispensing, but since it has to ask Parliament i’m not sure what the government could do other than repeal the Act (which seems unlikely)
The party that's in power, which is the Labor Party, has said that They don't want to address this anytime soon. They want to let it go into effect and wait to see whether it's good or bad. So I have, unfortunately, pretty low hopes that this gets averted.

58:12Yeah. dpk0 hmm
DPK, you can predict anything better than I can because I don't know how the legal process works in the UK of something becoming a law or getting repealed, and I don't know anything about the local politics of who thinks what. So my opinions about what's likely are not worth very much, but I'm sharing them because it matters. AnakimLuke why would lobsters need to geoblock?
So Ofcom... dpk0 i think seeing this article (the Times one you just had one screen) confirms to me that having a lot of sites geoblock the UK would be the best possible outcome if the Ofcom rules are approved
pushcx https://lobste.rs/s/ukosa1/uk_u…
did some where was it i even asked people to stop linking me to it there's the tv thread the anakin because the you should check out this page but the short version is that the law has said that they are in charge of the entire web that if Anyone... Oops, that shouldn't make noise. Anyone in the world can read... I'm sorry, anyone in the UK can read a website, they believe that gives them authority over that website. dpk0 because nothing short of that will create enough political pressure for the UK govt to care
This is a very novel legal argument. pushcx https://www.thetimes.com/articl…
AnakimLuke hmm
Yeah. Oh, I should... Here, I'll share the direct times link. AnakimLuke that's.. odd
And I said this... where did i say it i set it in email yeah yeah it's more than odd it's unfortunately i think dpk0 are you being very quiet at the moment or has my steam audio broken?
it's a bad idea. So I think it's really unlikely to stand up in court. But a small nonprofit site can't afford to be the test case. I don't think I'm being very quiet and my volume meter looks the same. AnakimLuke sounds good for me
dpk0 okay, then it’s me
chamlis_ they're trying a similar thing with the iCloud encrypted storage stuff too, it seems
If anybody could chime in, if I sound normal or very quiet, I would appreciate the feedback because I don't actually have a studio monitor. dpk0 page refresh!
Okay, so we're going to hope that that's only Daphne. Yeah, Shamless, I saw. dr3ig you're fine
dpk0 okay, now it’s working
Yeah, funny how turn it off and turn it on fixes everything. So... Okay, good. Glad you're sorted. So I think actually it's... It's also bad for the UK if they succeed, because if they succeed, they have established a precedent that a country can pass laws over the entire world based on local readership. And if the UK thinks that their censorship is great, I guarantee there are countries in the world that will want to impose censorship rules that the UK will not enjoy, like Does your fiction include women who go out of the house unsupervised by male chaperones from their family, do you have political parties that do not. hono4kami @chamlis_ This is the newest one where they wanna install a backdoor on encrypted iCloud right?
align with our great leaders thinking on how the world should be run. So. yeah so. Oh no, yes, I think that's what shameless is referring to, and we just had a thread about that it's.

01:02:05It didn't get tagged long.

...28I thought this was submitted. I know I saw it go by on Hacker News.

...42hono4kami definitely saw it on hackernews
chamlis_ https://lobste.rs/s/07jh0c
pushcx https://news.ycombinator.com/it…
Actually, if you do read Hacker News, there was just a great thread where... Ah, here it is. Ah, yeah, and it was on lobsters. I don't know why I couldn't... So it was tagged as security and not Apple or log. Hold on, so I'm going to... chamlis_ (I went to the homepage and ctrl-f'd apple, haha)
That's definitely missing tags.

01:03:20So I'm going to add law and iOS rather than Apple, because I guess it's related to the desktop, but iCloud is mostly their iOS thing. I don't know. Is it still on our homepage? Thank you, Shamless. I appreciate you finding the link. Yeah, there it is.

...55So... I am griping about the OSA because it claims to affect us. I'm going to stay out of expressing an opinion on the politics of That one. Because generally I don't talk politics online, but it feels important because for the OSA, because it is related to the continued viability of the site.

01:04:31So DPK, that is your answer, unfortunately, is there's not much news. Oh, the other thing I mentioned was on, let's grab that back because I had it.

...51There was a... I know it came up on... Maybe it's just fallen off of my list of recent threads because I've actually been able to comment on some... non-osa things so offcom can't find it which means that chanlis will find it 10 seconds after i say it right so there is they did a webinar last week or a series of them over days where they said hey drop on by and we will explain everything this is for small sites and chamlis_ https://lu.ma/qijpvfnl
We are probably a multi-risk site is my best understanding.

01:06:11Yes, that OpCom session for small low-risk community sites, Shamus, you're correct. chamlis_ I'm actually programming on the side this time so I'm a wee-bit slow ;)
That is exactly it. It came up somewhere here in the threads and I can't find it again. pushcx https://lobste.rs/s/ukosa1/uk_u…
session there it is all right so this is the comment i was referring to so they did a webinar and it was very discouraging i didn't watch the whole thing i read a bunch about it and i talked with people who were watching it and they AnakimLuke lol
talked about small low-risk community sites as if they were small business and their examples of like a small gaming site was one with 15 000 live users which puts a total user base in the millions so even in their small low-risk community sites they were just incredibly out of touch and They reiterated their belief that their law applies to the entire world and that their guidance suffices, which is hilarious, both of those. And then also they said, if you have any questions about this, well, you can just email us. And that's just a lie. And that's part of why I have gotten so discouraged and pessimistic about the Online Safety Act. because i have tried to talk to ofcom i have used their official email addresses and talked directly to them and they don't answer questions and they don't answer anything that might be politically embarrassing which is half of this law because you know let me go light on personal opinion on that one It is very easy to ask questions that are politically embarrassing, as I tried to do last week with the whole idea of, because they have this concept of extreme pornography, it means they must have lists of acceptable sex acts to determine what is and isn't extreme pornography, and they refuse to share that, which is to say they have a secret list of sex acts that are not permitted. And they will never answer questions about that. And someone who runs a Mastodon instance in the US that is, I don't know how to characterize it. Oh yeah, I characterized it and I think it was DPK laughed about soundboarding me reading their about page because they're pretty active, enthusiastic, amateur pornographers. And not to sound too fuddy-duddy and out of touch about it, but they did eventually respond and they just ignored all of his questions and when I've written in they've ignored my questions and they have explicitly said so they said in their webinar oh you can write to us and we will answer your questions so that you can comply and then what I've seen in their replies to sites who write in is they refuse to answer any questions and they refuse specifically on the grounds that they won't give advice to individual sites and they call it bespoke or they call it individual and they use that excuse not only to avoid all of their responsibilities and break the commitment they literally just made on Wednesday, but also they use that as a catch-all excuse to avoid actually answering any questions about their process and their guidelines. So if you go to the Woof group folks, so that was the one who, here we go, announcements.

01:10:27In today's live stream, Ofcom's James Proudfoot told service providers, if people reach out, what's useful is we can talk to you a bit more specifically about your service and the regulations. It's a lot easier to talk through a one-on-one conversation. And his colleague chimed in, if you reach out to us, we want to have conversations about what applies to your service. pushcx https://blog.woof.group/announc…
And does he quote it? I think he quotes it here. Let's share this link.

...57As the regulator, we are not able to provide individual service providers with bespoke instructions. So, Ofcom is talking out both sides of their mouth. And if you've seen me get very discouraged and very pessimistic about them in the last two months that this has been running, this is a very clear public example of how they are not acting in good faith. There are lots more examples that are not mine to share. If you would like more examples, you should just email Ofcom yourself and try to ask questions. And if you do that once or twice, you will stop thinking that they are serious about this. If you reach out, we want to have conversations. That's not a true statement. AnakimLuke who'd this law benefit anyway?
This is not a true statement. They just will not. They will blow you off and tell you to go piss up a rope and go read dpk0 pearl-clutching busybodies
250 pages of law and 2,850 pages of guidance. Well, Anakim, obviously it's for the children.

01:12:10That too.

...20So, I really appreciate that Kyle Kingsbury individually, because I believe this is his post, and Wolf Group generally, to the extent it's more than just him, I think it's a personal thing. I don't know. I'm not going to read the about page more on stream, even if it is very funny to some people when I say these things. I think this is an excellent public documentation of pushcx https://blog.woof.group/announc…
what the actual process of trying to comply is because if you read their previous post, their letters to Ofcom were, this is a dozen very serious, very real questions that you get if you actually start reading the law and the guidance and you want to conform. Even if you want to spend dozens of hours down that rabbit hole, everything is too vague and too legalistic and too self-contradictory. chamlis_ plays in the UK used to be censored at one person's whim, I wonder if the scandalous list even exists https://en.wikipedia.org/wiki/L…
It is not possible to comply. And then if you talk to them, they tell you to fuck off. And they say it very politely, but they tell you to fuck off. Lord Chamberlain's theater censorship. This is not one I know.

01:14:02Is the Lord Chamberlain an office or a guy or both?

...14chamlis_ yes, that
For many decades, so I'm going to guess an office held by one guy rather than, I don't know, a literal office. chamlis_ guess what year this ended
Hmm.

...32That's, .. Well, now, yeah. Now, instead of having one guy who has a very nice smoking jacket, they have, presumably, a literal office with some folks in cubicles. Guess what year this ended? You're saying this Licensing Act of 1737? I don't know. I'm gonna guess 1985. dr3ig wonder why his eyebrows are pointing up or down
dr3ig is it like caesar thumb ?
1968 oh man i was a little pessimistic it contained a distinction between legitimate theater and illegitimate theater this is exactly the the distinction they make now between pornography and extreme pornography i don't know i'm not going to dunk on anybody's eyebrows i'm Not on a particularly high horse on that front. So unfortunately, no news about the OSA. So today is February 10. I did on February 1st or 2nd update our database. So where was this? pushcx https://db-ip.com/db/lite.php
dr3ig i guess it's not about his eyebrows i as much as the painter decided to paint them that way
pdb it's in the damn footer why am i searching oh because this one this browser has slightly different dns settings and so it doesn't want to load this page but if you look this up you can see this is the free database we are using right now and it gets updated month monthly Yeah, I can't guess what was in vogue with painters. I have seen someone make the point that having a painter was like the ultimate Instagram filter because especially if you were the Lord Chamberlain and the portrait painter doesn't depict you in a flattering light, you can probably call him an illegal painter and throw him in prison.

01:17:04chronocidetwitch I just joined the stream, but I want to say thank you @pushcx for being so vocal about OSA. I learned about it through lobste.rs and been talking about it with colleagues here in the UK.
chamlis_ the modern equivalent is commissioning a profile picture I suppose
So I would assume that whatever the Lord Chamberlain looked like, it was probably slightly less attractive than that official portrait.

...18Oh, hey, Chronicide. pushcx https://www.writetothem.com/
Well, thank you for joining, and grab this link down in the bottom to please contact your MP about it. because apparently you don't talk to your person who's in the House of Lords. I'm learning so much about how UK politics work and what the legal structures are, and it's all against my will. I have one friend who is AnakimLuke LUL
Really into Final Fantasy 14 like puts a lot of time into Final Fantasy 14 and I'm being forced to learn things about Final Fantasy 14 by osmosis against my will. And that's what the experience of the Online Safety Act is I'm being forced to learn things about UK politics by osmosis and against my will. Oh, so date. dpk0 you don’t have an individual representative in the House of Lords. although it can be useful to find a Lord who is interested in a particular topic and bring a particular issue to their attention, but i’m sure everyone in the Lords who’s interested in this already knows about it
I did the February update of the database. The March update of the database is going to be in 20 or so days. chronocidetwitch I'm an immigrant myself so I know very little about UK law, learning about how to fight against the OSA as well
And when that happens, probably to give it a week or two, you don't have individual representative. Oh, interesting. See, look, I'm learning more facts in real time that I could have gone very happy about. through my life without actually knowing I'm pretty hands off of you know you guys run your country and we will run ours oh god there was yeah so I'm thinking probably when I do the March update is probably when I'm going to buy the commercial database that should have the improved accuracy now that I've chamlis_ I think US folks have a way more involved civics education than we do, I'm learning a ton here too
fixed all the caching bugs related to this footer. This was inadvertently a very useful feature test. dpk0 oh hey, i learned from the writetothem.com page that the local government council members (for my parents’ house, where i’m still theoretically represented as a UK citizen) are Green
I think we're gonna be good and way more involved civics education than we do. So we do have civics classes coming up in What do you folks call it on the continent? Primary schooling? Primary schooling? Secondary schooling? The schooling you are required to take until you're roughly 18. It is either called civics or it is called social studies. The two term chamlis_ primary is 5-11, secondary is 11-16
dpk0 moreoever, i met them when i was a member of the German Green Party and went to a Green Party of England and Wales meeting while back home in the UK once. they’re pretty nutty conspiracy theorists!
pushcx https://push.cx/tags#washington…
are used pretty interchangeably in the education level ah dbk everybody's learning things about politics on stream and then after but shameless i'm kind of a weird example because i have been a political journalist here in the united states as long as i was shown off my blog i previously worked at the washington post And I covered politics, including the 2008 presidential election. So if you are curious, you can read about, and these are in order actually. chamlis_ oh nice
So I worked on, oh man, dozens of stories over the couple of years that I was there. And there's a lot of, politics stuff. So I'm actually, for someone who doesn't talk about politics much in public, I'm actually pretty well informed. I like to know how this stuff works. I think it is incredibly important. And I do see, let's click over to the site, I do see lobsters as an extension of my journalism work. So there's a bunch of kinds of journalism. So if you see, what is it? There's usually like seven or eight categories people break it out into, and I'm not going to remember all of them off the top of my head. I was not expecting a pop quiz, let alone that I would be implementing it. But, you know, there's like the main journalism you think of, you know, breaking news, something happened in the world. And then there's investigative journalism, which is why did that thing happen in the world? There's enterprise journalism, which is we're going to spend months figuring out pushcx https://en.wikipedia.org/wiki/S…
what happened in the world so actually if you've heard of the movie all the president's men that's an example of enterprise journalism where people were working a story for months there was another really good movie what's that movie spotlight so this is one there's gonna be two no so if you're curious about enterprise journalism This is a great introduction. It's about the newsroom in Boston that broke the story of the Catholic Church protecting priests who were sexually abusing children for decades. And not just in Boston, but it spiraled into a global story. And so this movie is about the actual nuts and bolts of doing that kind of work. Another kind of journalism is trade journalism, which is writing about a particular industry for that industry. And I see lobsters as trade journalism. We are writing about the field of computing software and hardware and network design and the actual practices that go into it and helping the field understand itself. And some of that is breaking news like Someone has invented a new graphics algorithm. Something may be happening with the next standard for C++. dpk0 heh, you can’t use writetothem.com as a UK citizen living abroad any more, because you have to give a UK address to write back to
And then some of that is more general things like, I have an opinion about how to write this kind of code, or here's a thing I accomplished. AnakimLuke how do journalists in general see journalists that work for fox news? I ask because I'm brazilian and learning that such news outlet even exists is a surprise
Oh, no. Well, I mentioned on stream that the US has this same issue because representative representation federally is so local. The number the US expats also lack representation there's. Theoretically, they continue to be represented by wherever their last home in the US was, but in practice. dpk0 yeah, same in the UK. i can still vote for 15 years after i left. (4 years to go …)
no representative has enough of those that it's going to influence their decisions designing systems is hard and the idea of citizens who live abroad indefinitely was maybe not well considered at the time the actual u.s constitution was written they're pretty busy

01:24:40How do journalists see journalists that work for Fox News?

...49Well, the ones who work there like it quite a lot. pushcx https://en.wikipedia.org/wiki/Y…
dpk0 although i didn’t vote in the general election last year. i don’t feel UK elections are relevant to me any more
So there's a phrase in journalism called yellow journalism that is worth reading about. dpk0 ‘boulevard journalism’ is the term in most non-English languages, i believe
And I say this not to insult Fox, because this is now considered an insult, but this is Oh, they have a page on objectivity. That's exactly the thing I'm gonna talk about. dpk0 Boulevardpresse in German
Journalism in the United States, which is the only history I'm even a little qualified to talk about, has generally had a very strong opinion embedded in the paper itself. And there was a historical period in the 1900s from, Oh gosh, I don't know. Call it about 1900 to... Actually, just call it all of the 1900s, like from zero to zero. Where there was an idea that we were going to write objectively, that the news was just going to be about informing people and not also carry editorial judgment and say, Not only did this thing happen in the news, but this is evidence that politician A or B is bad and politician C or D is good. And it's kind of a historical aberration if you look at anything in journalism. And we've exited that area. Or I'm sorry, we've exited that era. And Fox News was famous in the U.S. for sort of being large and first about exiting from the idea of objectivity. Although not super honest about it, they did say that we're fair and balanced and that they were more accurate. I mean, everybody claims they're more accurate than everybody else. There is a similar criticism of

01:27:07This is the, so this is, the book is confusing. What I'm looking for, the phrase has been used as a critique. Here we go. pushcx https://www.wnycstudios.org/pod…
Yeah, this is the one. This is a good intro to it. The last five or 10 years, journalism has really been reckoning with this idea of, do we want to maintain objectivity as an ideal, or should we be more explicit that journalists are writing from their personal history? They have their own, not just biases, which is a negative idea, but they have their own opinions. And that it is proper for them to express their own opinions, not just in carefully demarcated opinion pieces in the editorial section of the paper, but in every piece about the news. So Anna Kim, you are asking about a very big topic. I hope that answers your question. A lot of my things, when you get me off topic, I tend to ramble about history and the structure of things rather than have a simple yes or no answer.

01:28:38dpk0 pushcx rambling ++
Actually, let's go... Do I have it here? No. AnakimLuke yeah I'd love to hear more about it!
Can I just search for the comment I just left? Yes. pushcx https://news.ycombinator.com/it…
So there was a big meta thread on Acker News here that I wanted to share. I mentioned this in passing. And in it, Dang wrote a whole bunch about how he brought all of the meta stuff together for folks who didn't want to get into it. And if you read down this thread, he has a lot of really interesting comments about his moderation philosophy and the guidelines for the site and why things work the way they do on Hacker News. And then there's also a lot of really clear criticism of him of, Hey, you say you're doing X, but you're not. And then he says, well, I think you're wrong or it works like this, or here's some more examples. So I thought this was a really great discussion in a bunch of ways. If you search down, you'll find me in there somewhere. because someone said, hey, HN, why don't you, yeah, lobsters has what they call merge stories where the moderator will merge a single page into a single page, the links for a few submissions, as well as the comments from all submissions. And so I gave a little introduction to it, and it's on, I mean, heck, it's even on my topic list as the thing I'll get to if people don't keep me rambling all stream. It's office hours, that's what it's for. And so I mentioned how I've been doing it on stream and trying to improve it, and I've even joked for a couple years that I wish HN would copy the feature so that NoGoodNick_ hi, how often do you stream?
wouldn't have to explain it all the time and we wouldn't actually have to document it because then people would assume it hey no good nick it's on the stream and i stream speaking from chicago time i stream monday afternoons and thursday mornings typically about three hours each because that's about as long as i can keep a coherent thought and stay talking because I try not to just have dead air for on and on as I work. So if you have any questions about lobsters or Ruby on Rails or the code base, feel free to ask anytime. And then when people aren't asking questions and pulling me into discussions of the history of industries, I work on the code base or I work on the community. And often I break things. So this bug report from Friday was some code that I don't think I wrote on stream. And Dreg, who was just talking here a minute ago, Dreg, who is a channel VIP, because he... Oh, Dreg, I don't remember your gender. Too many names. I had to make a cheat sheet somewhere. I would build it into Twitch if they would give me an API key. So... AnakimLuke I'll let you code and ask about ad-supported journalism next stream :D
Drag opened a PR with a fix of this code and I took a different strategy and I called touch having a foot gun because when you touch a record, let me code and ask about ads. Well, ad supported journalism got shot in the gut by Craigslist in 1995 and then it has been wandering around maimed for the last 30 years. And I think that's a big chunk of what happens with the end of the idea of objectivity where, I don't know, there's just so much less money in journalism now than there was 30 years ago.

01:32:54So I added a field It's fine. I do still have to finish fixing. Where did my... Are we just being inconsistent, GitHub? I filed. Yeah. So I have griped on stream that GitHub has gotten more and more flaky the last couple of months. I wrote an issue couple hours ago today and i submitted it and then i just got a blank page it just failed to load the next page and stopped there it did post the issue it didn't just throw my work away which i appreciated but github is getting so unreliable it's really frustrating me so the issue was when the comment is edited it touches the last reply at on the parent And that recursively touched the updated app for all comments going up the thread. And I went, what the heck? And I griped a little and I called it a foot gun. And for us, it's new behavior. dr3ig EDFL
In 2012, our DFL, DHH, wrote this post about the new caching strategy that Rails was adding. and said, you deal with dependency structures by tying model objects together on updates. E-DFL. I don't know E. Like you're making the same joke I was, or maybe a saltier version of it. I have heard ADSL, DFL. So if you change a to-do that belongs to a to-do list that belongs to a project, you update the updated at timestamp on every part of the chain, which will then automatically update the cache keys based on those objects. So this is the introduction of that behavior that caused that bug. And I thought I was sleeping on it. And then on Saturday, I was annoyed by it. And then on Sunday, I was annoyed by it. And I was like, what was this? pushcx https://github.com/lobsters/lob…
And I dug it up on Monday morning here. So this is when this behavior was introduced and we didn't know it because we've never used Russian doll caching, which was introduced in Rails 4 after the site was started. And the full page caching that we lean on very heavily was removed in Rails 4 and made an external project and it's no longer maintained. So we've got to think about that, but that was when, rails started considering updated at as something the framework was in charge of updating and could do so at any time rather than business logic so like the name comment on the model rails would never override me and say no i'm gonna call that model foo But it did decide, no, you have this column named updated at, and that's mine to do things with. Who do you think you are, Ofcom telling me how to manage my database? Topical. So it made me feel better about changing to this explicit timestamp feature that I did here. where I added a last edited at and said, okay, Rails, I'm just going to let you update updated at your whim because I figured that's going to be a losing battle trying to stop it, honestly. I also tidied some of this. We had a lot of callbacks and it could have just been one. And then I realized belatedly that Actually, every continued use of updatedAt should probably be replaced because in all of them, we are assuming that updatedAt is our data managed by our business logic. If it is now going to be Rails data updated when Rails believes a cache is invalidated, we're gonna have other similar bugs. But the other part of this was I have to clean up some prod data. And I started it, and then I went to the Super Bowl. Because, you know, in Australia, they have a system of mandatory voting. And in the US, we have a system of mandatory football. And so I was required to go to a Super Bowl. That's a pretty good one, actually.

01:38:20I joke, but it is a cultural touchstone. And it is actually the law in Australia. If you don't vote, you have to like show a doctor's note or another good excuse for why you didn't vote. I think you get a ticket, like a parking ticket. So we get about 200 comments a day. So there were about 400 comments between when I deployed this code and when I fixed the bug. And so I looked at all of these comments where the updated at was equal to its parent comments. Or actually, let me say it the other direction, where a parent had the same updated at as its child, which is going to be this behavior. And then I pulled all those short IDs out, which is just, you know, select the distinct PID, P short ID. i dumped it in a file and i started looking at this by grepping the production log and then i had to close that terminal i can't finish this on stream oh i knew i should have saved this damn it all right well i will at least start it off screen again so where's where's the terminal

01:39:52So that code, let's just grab these things. And let's make sure that logRotate didn't change the ground out from under me. Oh, this is going to have somebody's IP in it. And it does. And the date is 2-10 ticked over. And the week ticked over. AnakimLuke 127.0.0.1 exposed :o
Damn it, I really inconvenienced myself. All right, I'm going to take that one back off stream then because I'm going to have to hassle around. So what this code is doing is saying, yep, for each one of these 401 comments, grep the production log, which just got rotated because a new week happened, and pull those logs into a hash data structure. let's see this one yeah this one was just me checking that it worked and then this one was me finding anybody who posted to updated comment i say post patched which is the http verb and so only 75 of these comments should have the correct data on them all of the rest you know the other 326 need to have their updated app reset back to the created app. So I got all the data and then I closed the terminal because I was running around and doing other stuff and it didn't occur to me that the log was going to roll over. So I will have to uncompressed last week's log and then rerun that. AnakimLuke I didn't know you could use backticks to execute programs like that in ruby
It takes a while to run, so I'm not going to do it on stream. So let's just put that down here.

01:42:07Yeah, Ruby has a lot of conveniences like that. And if you I think it's like you automatically get returned standard out and then the exit code goes to a variable dollar question mark if you need it. And I think standard error gets thrown away. And then there's a different function or two that you can call if you want standard error or you want something else, or if you want to interact with the program while it's running, that kind of thing. AnakimLuke cool
Personally, I pretty much don't write bash shell scripts much anymore. I just write Ruby.

01:43:01So then the next thing I guess that's on my list is story merging. which I made an issue for so that I didn't have to keep track of my to-do list.

...28Oh, yeah. All right, so I've got that running.

...39And then didn't I have a merge story handy? Folks can pipe up with questions any time. Otherwise, I work on code.

...59Close these.

01:44:07And then let's grab the console.

...19dr3ig you didn't delete the touch code, do you want to update :last_reply_at for all comments in chain ?
Easier to express in SQL.

...29I didn't delete. I can see how it would update the updated app. Why would it touch the last reply at for each. So I made it specifically so the controllers update the last reply at. I'm thinking of last edited at. Yeah.

01:45:22dr3ig comment model has `belongs_to touch last_reply_at`
Comment model has belongs to touch last reply at. dr3ig `belongs_to :parent_comment, ..., touch: :last_reply_at`
Is it going to roll up that way? I think I have to see this. Yeah. So let me.

01:46:04Let's just grab some recent comments. All right, local host. Why did Firefox? Who wanted that Firefox?

...28Gosh, I'm still on the, I didn't revert that code, did I? Yeah, that's... I'm just baiting if I want to keep that. All right.

...51And I'm current master.

01:47:00Good. All right. I've got to bounce the Rails server because I put ProSupply back. All right, so here's a comment that's nested aways. So you're saying if I took this comment

...39dr3ig or post a new reply i guess
I took or post a new reply.

01:48:11So if I said again, yeah, I'm going to use the, UI to reply. Why are you thinking so long rails? Is this replying comments for you? All right.

...46All right, traditional. Oh, actually, I think there's a validation that won't let me post that. Don't bump threads, yes.

01:49:01Okay. It's now over here.

...27Hmm. You are right, that is not the behavior I expected. Yeah, and it's just because it's using touch.

01:50:01Yeah. Actually, this last comment app must be totally wrong, too, because of that. Because that's touching story. So both of these things are... I've been misleading myself based on the name that they are not doing what I want. They are doing... So this wants to just be touched true. because I do want the updated at to get touched. And I do want to do that all the way up the parent chain because I do believe that we are going to end up having to switch over to fragment caching. And I would rather do that before the page caching gem becomes incompatible with the new version of Ruby or a new version of Rails.

01:51:13dr3ig `belong_to :story` code should have been ok because there is no propagation
I don't know. It mostly works by rack. And it has a little bit of hooks into rails. So maybe they would update it if some little Ruby feature broke it. But I feel like we're very far away from the standard model. And every time I've gone dr3ig or no, other comments in the parent_comment chain would trigger the touch
Well, off of the standard model we've had some pain about it so that's where I keep leaning towards all right let's lean back into the Ruby or the rails default more. Other comments in the parent comment chain would trigger the touch. yeah.

01:52:16All right.

...59this yeah let's just grab that

01:53:46Then this one's going to be basically the same thing.

01:54:12Well, let's check that. This last reply app is going to be the created app. Actually, I have already written the exact thing I was about to write. So let's just snag that. Is it good when you have to run a migration the second time? Chat, am I cooked? All right.

...49Then is replies count correct? Should be. All right, I called it something else. Reply underscore count singular. Yes, so the counter cache should be correctly maintained. Do I want to fix the updated add on these things? No, because those can go ahead and be the latest date, and there's already the bug to stop relying on them as business data. So I think that's OK. Let's see if this runs.

01:55:44And find c.storyId from comments.c. We can just say c. All right. dr3ig does that mean that posting a deeply nested reply will trigger multiple updates to story.updated_at and comment.updated_at in the database? would you mind posting a new bump and showing logs ?
And then I can update the business logic to set the correct values.

01:56:16Does it mean posting a deeply nested will trigger? Oh, that's interesting. AnakimLuke is downloading all the president's men
I would expect that it's probably implemented with some amount of caching so that it goes up the chain of comments and only updates story once, but since each story is going to want. I don't know. Yeah, we'll check. Although.

01:57:03So let's get it into a state before I lose track of what I'm doing here. All the president's men. Yeah, I do. That one launched a generation of journalism careers.

...21would hope spotlight does the same but it wasn't a cultural touchstone in the way the nixon resignation was

...55I can't call the touch method, can I? Because if I call touch... Well, does the... I've got to check this. Where's my docs? That's my Bluetooth. It's even up. I was just looking at this. So if I touch, no validation, only the callbacks... Yeah, so I will have it touch the parent.

01:58:41It's in the right order. So I'm going to put that in update, which is the method for updating comments, and then also in create. And then about 10 seconds after I come out of this file, Greg will tell me why that's the wrong logic. I may not be able to learn Rails, but I can learn the pattern of learning that I'm wrong about Rails.

01:59:17Anything in here? No, both of these come from comments. Yeah, last reply. Last comment at, last reply. OK. And then this one, we will also say that needs an ampersand, because that's otherwise a bug. Oh, but then everything in Rails works by having invalid objects. So where's the actual save?

02:00:11So we build the comment, and then I was editing the wrong code.

...29Where's create?

...36if comment valid, and comment save. There we go. So if you save, then we will touch the story and the parent comment. Because I don't want to touch them if someone attempts an invalid edit. So comments have to be on a story. So that one shouldn't have the nil propagation operator. But they don't necessarily have a parent comment. So that looks correct. Now let's do that same logic down here in our last comment add. And this one wants to come out of here. And then if save.

02:01:34I guess I did it after the road stuff up there.

...43All right, Drake, what did I do wrong? I'm going to run the specs, but I just wanted to give you a chance to see what I did wrong before it explains something obvious I did wrong.

02:02:02And then we can run your check of, all right, I'm going to timeout and run it. dr3ig sorry, i was testing updated_at on story, and it only runs it once on story
Migrations are pending. Wrong environment, right?

...23Oh, that's great news. Thank you. Yeah, I was just kind of teasing myself that I assumed If I've written five lines of code and you're on stream, you're going to catch a bug in those five lines.

...42So it was here in comments controller line. Well, the ones with the plus next to them, 52, 53, 200. And let's see if the tests pass. Oh, looking green. Not bad. All right, let's see it in the browser and then also see it in the logs. So let's reply to myself. Definitely dump. And then, so I have the log here and I'm going to just insert some spaces.

02:03:32Where did I go? So I did not see the behavior you described. Stories updated at got touched three times.

02:04:04And I'm actually a little surprised that it didn't have to select each of these comments out.

...20Oh, it could be worse. Let's go all the way up to the actual gap. All right, so here's the gap. There's some selects. And we updated the reply count. Yeah, so the insert succeeded. It didn't update. And then here's all the extra selects. And then, man, speaking of 1 plus N, each one of these comments goes and fetches the story.

02:05:07And we insert into the votes. And then the vote stuff triggers. And it updates the story score. So that's all OK. Cached column. Maybe I just want to move this into drop the touch and manually do it in the story update cached columns. Huh, why is hotness getting set twice?

...48One from story 423. No, one from story 926. Again from story 926.

02:06:11So these are callbacks on the story model.

...35Hmm.

...49This just hammers the shit out of the database.

02:07:09The pain here is we do want these touches to happen so that we can maintain the cache key But this really could be one, and it's strange that it goes through the whole thing twice. We don't have a full trace back here, but comments controller 50, comments controller 53.

...48Oh. So drag, it's 53 here is causing the second wave of them. So because we touched the parent comment, that's gonna edit it's updated at, which is then gonna cascade upwards. So that's why there's two of them.

02:08:18dr3ig yeah, it's only one set of updates on local
And then the story gets touched twice because once it's run through all the comment updated ads, it's touching the story model.

...40dr3ig with master branch
Did you maybe make a top level comment to see that? Because here, oh, you're saying with master. Yeah, because you wouldn't have the different touch I just wrote. Yeah, I can see how that would be one instead of two.

02:09:10And instead of two sets, it could be two updates, because this could be one update of... comment.parents.mapId. twitchtd am kind of curious, how big is the lobste.rs database? maybe one proxy would be how big your backups are
So then what was the duplicate here? This one from update, why is this?

...55See what this feels like. dr3ig this duplicate is present on my machine too
dr3ig the hotness one
If this was down here, I would say I understood it, but now I don't understand why there's two updates to calculated hotness because I don't see the whole trace back.

02:10:16Yeah, this is going to be something in the voting logic, probably. So model story 923.

...44Oh, is it merged into story?

...57No, I get it. It saved the story and then the story record touched update column. I bet this happens after save, right? Yeah. afterSave calls updateCachedColumns. updateCachedColumns calls updateColumn, which is triggering the callbacks a second time, I bet. So if I bring zeal back and I pull up updateColumn,

02:11:42dr3ig no, update_column is the one that doesn't trigger callbacks
No, callbacks are skipped. Hmm, then I don't understand it. I thought I was going to say callbacks are run.

02:12:18oh there we go no that's delete i was wondering if this update cached columns is running twice and it is there it is so when you update a comment it is explicitly telling the story to update its cached columns but that's redundant somehow something else is causing it to touch the story or this is running twice no this is running once and then update score and recalculate is calling the update cached columns and then something else comment does select count star from stories No, this is repeated. So who called update score or calculate?

02:13:51Don't call it once.

02:14:37There it is. It's because this was a new comment. A new comment calls update score and recalculate, which calls update cached columns. So this call is redundant. That's the second one. Let's go double check because I've looked at a lot of code. And then that calls update cache. Yep. Okay. So there's There's that second tree through here. All right, so that's a bunch of queries that were redundant that aren't going to run a second time. That can just be a.

02:15:57And then... So one of the things I'm thinking about here is touch on the association. And I want it because there are so many code paths that can update a model that should all touch the updatedAt. And having a touch on the association is the only way to do it reliably. However, because of where it is, it can't be smart. So if touch took a block, Then I could get all of these down to one. Well, I guess two, because it would have to... Yeah, because it's so limited on recursive comment table expressions. I would either have to reimplement the parents method, or I would call parents to select, and then I would call update once with all of their IDs. But then there would still be this.

02:17:29We don't have to do all this twice. So on this one, yeah.

02:18:51All right, so that looks like an accurate comment and a clean fix. Well, minimization. Let's see if we can. So bump three, everybody, well, turtles in time, right? That was the name of the third. I can't even start with bump. Hello.

02:19:19All right, so now, There's walking the tree there is still a second one, because the controller. Right, so this one is controller create. Create because we manually touched the last comment that right so that's updated at first last comment that and this one. automatically is updating updated at so like it's it's an overwrite but we do have to do this part and because Active Record locks us into the world of callbacks. We're kind of stuck here.

02:20:22And then each one is loading its story association. And what was I just saying about press a pipe protecting me from one plus inquiries. I guess it is smart enough to recognize when they're coming out of rails. There's not a lot the user can do about them.

...52That's

02:21:18yeah and the why does short id create a transaction wait a minute why is short id starting the trend is this the damn deadlock is that the source of it have i been looking in the wrong place the whole time

...48Why did you print line 40 there?

...59That's so strange. So this begin, this is the transaction that's causing deadlocks. Because deadlocks happen when there's a race where a user is posting a comment at the same time someone is voting on a comment, and it's been getting worse over time slowly. I now get about one a month. And some of that has been site traffic has increased over the years, but some of that has also been this kind of stuff creeping up because the more of these queries there are, the longer this transaction is held open. And it really doesn't need to be because the worst thing that could happen is like someone sees a very slightly invalid story hotness where it has one fewer point because there's one fewer comment with one fewer vote. But the individual effect of that is pretty negligible.

02:23:08And we can see this is interesting. dr3ig re: the deadlock, i was thinking that most of these queries that update cached columns could be moved into `after_commit` callbacks instead of `after_create` callback where they happen in the same transaction
There's actually There was a collision attempting to generate a short ID because it happens up in Ruby. Set it down in the database as a function.

...30I think that is totally true and would help, but then also if we can minimize We don't need to be in a transaction at all, even for this stuff that's happening before we get to those callbacks. But you are right. Yes. And all of those could move to after commit. I'm just stuck on why is this happening? Why is the trace back saying that this is happening in short ID line 40? When short ID line 40 is just asking if it exists.

02:24:31Thank you.

02:25:11The painful thing is this has a race condition anyways, where two requests could generate the same ID for a comment at the same time, which is vanishingly unlikely. But that's what happens when you push the work down into an ACID transaction and make the database generate a short ID.

...42I guess what I'm frustrated by is I just don't see any kind of save method here. Is it happening in the comment?

02:26:04Yes. So that calls generate. Why does the transaction say it's inside that function?

...30I'll just delete this and leave the stub there. That was something I cleaned up yesterday.

...54So the actual short ID it used was this bb3-jan, which is this value. So I think these lines are happening out of order or something. Because when does this run? This runs before validation on save, right? Before validation on create.

02:27:27So the create is I think it's the create that starting the transaction.

...55And it's just getting reported here because these are lazy queries and they're getting evaluated eventually once it gets to this code. And this just happens to be the line of code it's at when it starts touching. Yes, when it first starts hitting the database. Why is not this one, though? because this is the one that's looking for dupes comments 37. Yeah, so it looks for the comment. Make sure there isn't breaking the speed limit. And then it calls comment save. And because these things This is what's triggering this. What we want is for this save to not, can I say that the save shouldn't create a transaction?

02:29:17Why don't you tell me the rest of the options?

...34SegFaultAX Howdy! New here. What are you working on?
Hey, SegFault. Welcome. pushcx https://lobste.rs/
We are working on lobsters, which is, that's localhost. That's also localhost. This is lobsters. It is a site where we talk about programming and this is office hours. So if you have any questions about the site, the code base, moderation, we can answer them. And when people don't have questions, SegFaultAX Nice, thanks for the intro. Been a lurker for a long time. Curious what bug you're trying to squash. :)
I break things and get confused by Rails. So what if I just call... Can I ask save?

02:30:17Oh, it's a new bug. So what if I say CZ?

...26Now, see, it did a begin. and commit anyways. I just say anything and it all right. So it's one of those methods that all right. So Greg has caught a performance issue where the way we're tracking timestamps on these things is causing rails to create a bunch of one plus n issues. And some of that is not fixable, but some of it was just Because records were getting touched twice, we were doing twice the work. So I'm at least trying to only do one part of the work, and then... So when you touch a comment, it has to walk the IDs of parents, and then after it does the save, it has to touch all of those, and then the story at the top. And it was doing this section twice, so that's fixed at least. But, you know, I could probably get rid of the selects because if I, no, the objects wouldn't be hydrated. There is another hidden layer of Rails caching where if you look something up by ID, it hangs onto that, but I can't fill the cache for each instance. SegFaultAX It has to query all the comments up to the root?
they're not looking it up by id they're looking it up as parent comment dot id yeah okay so the other thing is this transaction yeah so rails has this thing called russian doll caching where did i put it the link is here somewhere

02:32:47pushcx https://github.com/lobsters/lob…
SegFaultAX Thanks
Here you go. So Rails has this key based cache where they have the idea that you will do Russian doll caching. So like this comment would be one cache document and this would be another and this would be another and this would be another. And so if you invalidate, say this, you do have to evaluate, invalidate the parents, but not the children. And so if you invalidate the parents, it has to walk all of those associations individually, because Rails can't really know that it's a tree and definitely doesn't have the excellent support for recursive common table expressions to avoid walking that. So...

02:33:53SegFaultAX Could you query all the comments at once with WITH RECURSIVE? Is that even an issue at the moment?
Well, this is my...

02:34:13It's not the issue, it's... Well, no, it's also not the issue that I'm chasing right now. This is happening because for each comment, it has to find the parent comment, because that's an association on the comment model. And then once it's touched that comment, it goes, what associations do you have that I have to touch? Ah, you have one called parent comment. And I could delete the default rails and write my own after save callback that used a with recursive to make that one query. And it would even be a single updated at rather than a select and an update. That would be nice. But what I'm really looking at is getting rid of this begin because we have a occasional deadlock that happens in production if someone votes at the exact same time someone is leaving a comment. This guy locks comments and then tries to insert the vote, which makes sense. You have to have the comment inserted to update the vote. Vote has to do them in the other order where it updates the vote and then it updates the score on the comment. And so we get a deadlock if those two things happen at the same time in prod.

02:35:56SegFaultAX I see, thank you! I have to step away for a meeting. Will follow passively :)
You'll lose after rollback and after commit.

02:36:09All right. Have a good one, segfault. Thanks for dropping in.

...18There was another. Oh, it's linking the same comment.

...27chamlis_ I've been listening with half an ear so apologies if you've already covered this, but can you have them both lock a secret third table before they do anything else?
So this atomic create, atomic write, a lot of the Rails API is not documented. And sometimes that means don't use it, and sometimes that means be very careful.

...56Shamless probably. I can see how that would work, but I would rather just drop the transaction altogether because we don't need one. I don't think we need one.

02:38:01Yeah, so we do, drag is correct as usual, that we do want to make those other ones after commit. But if we get rid of the transaction, they won't run at all. I mean, it can manually run them. That's also dangerous.

...48Atomic Creep exists.

02:39:05Okay, so he's making a... I'm calling it Atomic Save. Let's find that one. So it's not...

...30This is not so promising. Not finding anything about the Rails codebase.

02:40:02So I guess whatever the internal API that that Stack Overflow answer was based on is using, it's no longer in existence.

...22It was making its own?

02:41:04skip transaction oh he's writing this isn't he that's that's quite a monkey patch

...34yeah manually that's a lot

02:42:08Oh, we could probably postpone the begin to the first DB execute method. So that's why someone must have come along and made this optimization. And that's why the traceback was so misleading and confusing.

...38Well, maybe I'll get rid of the deadlock if I take Dreg's point about. So we have comments. Then there's that tree. And then we insert into votes.

02:43:02That's comment model.

...26I see a matching commit. There it is.

...43Oh, this is going to make the deadlock worse because now it's going to lock comments and then votes and then stories. where story creation is going to lock stories and then votes.

02:44:05There's going to be a second one.

...15This can at least move out to recreate.

...55Let's just look at the output here. I don't have to do all my reasoning from first principles, but I am trying to build it up in my head. So I feel like I actually know what's happening. All right. So I got my browser off screen. I'm not going to tab back and forth and we are leaving another comment. Okay. So here's the commit. All the updated ads. Okay. This all happened out of order because we had a begin. Oh, because it does this in its transaction chain of the callbacks later. And since I committed earlier, that just went away. So this inserts into comments. So it's going to lock comments and then stories.

02:45:59there's still going to be a deadlock because when you repost a story you're required to write a first comment so that'll lock stories and then comments but that's going to be so much rarer i may never see it in prod and this like between these two lines there is a short window where the comment model has score one and no associated votes which is a little wrong. But what does it mean? It means if you vote on something else, well, there's already a race. And you'll immediately get it overwritten. So that actually seems fine.

02:47:09And then here's the next after commit, which this key store use could really go away. It's a counter cache with a where.

...35And then these become a series of smaller transactions, which is much better. Yeah, Drake, I think that may solve the deadlock. I'll give it a couple of weeks in prod. All right. So what all did we do? We put these in. We removed these touches, so it's just, well, instead of touching that field that was doing what I imagined, or not doing what I imagined, and then these move to after commit. dr3ig i kindof wanted to only move cached columns to after_commit, while keeping Vote creation in the same commit
Just, are there other after creates I should?

02:48:49I think I totally follow where you're going, Dreig, because that gets you full referential integrity as to what I was saying about there's that brief window where the comment has a score without a vote, but that's the deadlock. So if I insert into votes in this first commit, then that commit will lock comment and then votes. and the other one is gonna lock vote and then comment. I'm wondering if also I could address it by, let's look at vote before I commit this.

02:49:40So I wanna say this doesn't have any callbacks, no. So when I vote We call the vote thusly. That creates the vote, so in the case where You are creating a vote. That's a transaction.

02:50:17T is target, so this is the story or comment. So if this calls comment update score and recalculate,

...47Let me see the chain of, I'm gonna adjust off screen, I'm upvoting a random comment that is not voted. So here's the begin It grabs users and then stories and then inserts into votes and commits. And then calls the update and recalculate. Users and then stories. I thought it was users and then comments.

02:51:51And why does it have to touch the story here?

02:52:10Vote 164. I bet this traceback is wrong again.

...28Yeah, there's nothing here about the story. There's nothing here about touching. So that traceback is wrong. This is probably some kind of lazy query that's just getting evaluated along the way. Where could it be coming from? Well, previously it was upstream, so comments controller find comment.

02:53:33So who calls vote thusly?

...56Flag calls it and upvote calls it. Comment.storyId. This must be evaluating the association somehow, even though it's trying to avoid it by using storyId. And let's, just for humor's sake... Well... dr3ig do you need params["reason] on comment update?
I sort of suspect it's this. dr3ig comment upvote sorry
So I'm going to just pass a raw ID, which I'm going to grab out of the Donate Params Recent Account Update. One sec. Let me grab this ID before I lose my train of thought.

02:55:04no, it should be null on upvote. I see what you mean.

...19Right. So off screen, I want to watch this log come in because I'm expecting that storage lectures. I'm upvoting a comment. I haven't put it before.

...35No, it still selected the story out.

...44Hmm. See, all of these have the same number because it's where the vote save is happening. is triggering all of these because it's delayed. So who else do we have here? So we find the votes, controller 387, 393. Put that back to nil.

02:56:42This is using the story ID as well. Hmm.

02:57:09pushcx bitrate has gone to hell for a sec, I may be lagging
Because there has been more rarely a deadlock on stories, and it's clearly that right there. So whatever is triggering this is the source of that second deadlock. But I don't know what it would be.

...44dr3ig yes, for a minute or two it's been buffering
Oh, am I back? pushcx it got so bad for a sec that OBS said I was offline
pushcx am I live?
It went so bad that... Am I live? I don't know if I'm live anymore. I'm right about three hours, so if this doesn't clear up immediately, I'll just end the stream, but... No, I think I'm dead. dr3ig it's good now( here)
Oh, yeah, I'm good. All right, then I'll keep going. So I still don't know what's causing this select of stories. So it's something from comments upvote. We don't track any kind of.

02:58:58Well. Acquire logged in user show title H1. Show title H1. That's.

02:59:19OK, true.

...57funny this is One of the more optimized bits of the code base where we're passing around story ID instead of story object and such, because we're trying to avoid exactly this issue.

03:00:18But I bet that's what's happening if we say vote first or initialize. creates a new vote object, and then I bet it's looking at the association for something.

...54I want to see if I can so if I trigger it manually here before we get down into the vote transaction. If that select will go away. Because if it goes away. Then I found what's causing the select.

03:01:21Begin select user. yes so it is just the presence of that yeah that lifted it up up here so what about the story association is causing it the vote model to do that just says belongs to story optional false

03:04:33dr3ig there is a v.target below that retrieves a story sometimes(?)
dr3ig tthere
is there the target what line are you on oh wait you're right you are right but it's happening after the save so it's after the block so this is not going to be in the commit Because the issue is just that it's in the transaction at all.

03:05:17So I don't think that's our source. I mean, we can test it if you think we need to, but...

03:06:08actually. This is the same deadlock. No, it can't be because we didn't see story in there. Or we didn't see comment in there. Yeah, so comment is using the V target that's below. This is just something particular to story.

03:07:02okay so i think like i feel pretty good that that's the trick the occasional deadlock when someone is voting at the same time as story creation which is still very rare because we have an order of magnitude fewer stories each day than comments actually sometimes two orders i don't know why but it also selects these there isn't a touch or callback on The Association.

03:08:24little wordy but i would rather be a little wordy and clear than a little terse and obtuse because man that's fiddly and i've been looking for this for like two years not you know day and night burning the candle at both ends looking for it but watching for it all right

03:09:03Let's comment the other one.

...12Drake, would you like to be credited in the commit message I'm going to write in a minute? I think sometimes you don't like to be.

...39dr3ig no :) i'll be happy if it works
dr3ig but i worry it might not
Sure.

...45Yeah. And the frustrating thing about this is it happens pretty rarely. So this is going to be a, I'll commit it and then just wait a couple of months. And if we didn't get any five hundreds, then we're good.

03:10:16eliphaz_ Hi
Oh, hey, Eliphaz. Have you been here before? Somebody with a similar nick has been here before, and I'm assuming you're them. dr3ig as long as this doesn;t break other things; I worry that if after_commit fails, votes and updates will be lost
But then Twitch is highlighting you like you're new.

...42Votes and updates. eliphaz_ yeah it's my first time on your stream ^^
Let me double check that in the log. Let me check that and come back and write the comment. Oh, okay. There's somebody else. Oh, I'm thinking of Epilus, who's a regular. Well, anyways, you can become a regular. eliphaz_ yeah ^^
All right, so I voted on... dr3ig maybe some background jobs to keep cached_columns up to date would be a good idea
comment and we had the commit and then this is all the after yeah or maybe i should get recheck live in the world and we would know if these are actually problems in the real world so these are update comments and then we pull the stories out What do you want to bet this is that same one that I just did? eliphaz_ why you don't use syntax highlighting ?
No, because that happens up here, so you know it's a different object associated.

03:12:10It doesn't really help me. There is some in Vim. It's just very subtle, because this kind of angry fruit salad of lots of bright colors, I just find mostly distracting, except for this exact kind of work, which is uncommon, where I'm trying to find different kinds of things, like select versus update, and begin versus select. In the editor, though, mostly it doesn't mean anything in Ruby if something is a method versus a variable. It just doesn't tell me anything. So I highlight literals with that cursive, and then that's pretty good for me. All right. This code is pretty bulletproof, Dereg. So it's not that it's... eliphaz_ Oh okay, I see :O
The best code in the world, but it does get called several thousand times per day and will be very, very obvious if it breaks, not just because i'll hear about the 500 but. A lot of stuff is going to fall off the truck or fall off the tracks if. These aren't running like. Comments are a substantial amount of story score. And we definitely have enough power users who would notice that stories are getting ranked funny.

03:13:58Okay. I think we're okay.

03:14:15Let's go find the deadlock issue so that I can reference it and make sure this commit gets linked.

...41eliphaz_ it's very awesome code without any syntax highlighting
Yeah, this is it.

...48It's just... I think of it as very chill. eliphaz_ do you use vim or neovim ?
I was using Zed the other day because I had someone pairing, and I had to fight to find a... You know, Dreg, there's an idea. eliphaz_ like that don't need any treesitter
If I turn off page caching, the whole site will run slow for a day or two, and any race condition will get way the heck more worse. That's not proper grammar, but it's been three hours, so I'm getting tired, but... I could force that. So I'm using Vim rather than NeoVim. I have bounced off of NeoVim. I can't quite get it to install plugins correctly. Yeah, this second paragraph is wrong because it's about the vote table inserting against story. then the vote table inserting and selecting against comments and we just saw that the code path doesn't select comments so if we moved that record initial upvote out into the after commit i feel pretty good that this might get it i'm not actually going to turn off page caching and slow the whole site down though all right let me oh i didn't finish this comment

03:16:34eliphaz_ you work at Lobste ? :O
eliphaz_ I cannot make any comment on lobste
may actually be enough you know as long as i'm touching this i'd like to have these things in alphabetical order although i guess rails runs them in order work at lobsters it's worse than that i do it for free There's no job. It's not a business. So I do run the site. eliphaz_ Oh :O super admin ?
I'm the administrator and moderator and coder and janitor, ditch digger. I break servers too. So I guess that calls DevOps.

03:17:27This was me trying to find that. All right.

...38super admin it's a little bit less super admin than it is yeah there we go this is me right here why'd you take me to a random place google there we go

03:18:06eliphaz_ oh I see :D
You know, I'm playing the accordion and kicking my, oh, she's not wearing, she's got the knee symbols, but not the little jangly spurs. eliphaz_ :D :D
You can imagine me making this face too. That's even better than me having a webcam, right? All right.

...28eliphaz_ are you use window manager ?
So you're coming in at the end of the stream here. I usually run about three hours. So let's, that's the migration, that's the timestamps.

...44So Dre, thank you for kibitzing. Yeah, I'm using one called awesome.

...57pushcx https://awesomewm.org/
So Dre, you asking about that, eliphaz_ Oh okay I see. I use sway wm
issue with the wrong data is what sent us down here. And that may have fixed this really pernicious little deadlock.

03:19:35eliphaz_ how can I do to post comment or article too on lobste ?
I say it as maybe. Register for an account. If you read the About page, it'll tell you how to sign up for an account. Or actually, the Login page will link to it too. So what was that number, 1238? Yeah, so GitHub will see the words fix 1238 and then close it, and I don't want that. So I'll just say maybe 1238. All right, so, Drake, I didn't thank you there, but I thank you in my heart for that. Watch this. Let me remember to run the specs, right? Run the whole build. eliphaz_ very nice and minimalist your code workflow
God, I hope I didn't break prod again. So home stretch here on the stream. If this passes, I'm going to push it up to GitHub and deploy the site and call it a day because we're at three hours, 20 minutes. eliphaz_ you don't use LSP, Treesitter or any other thing like that
So if anybody has any last questions about the site or the code base, you got about 10 seconds till the test suite finishes. dr3ig fingers crossed :)
You got more time if I broke something. But it looks like I didn't. I don't think the Ruby Vim plugin uses TreeSitter. I think it's just doing the old syntax kind. You know, the kind of regular expression hacky version. All right. So just those two commits. Great. All right. And in my browser, the commit has popped in here. So no, I don't actually, I'm just going to end up reiterating these comments and I feel like having the commit linked is enough. And this one has been hanging out for, oh, I only reported it a year ago, but it's at least a year older. So, all right. So there's the deploy. Oh, let's go turn off the mention that I'm streaming. I don't even have that on my checklist. So there we go. So this is a new code path. Let's make sure that that actually worked because I don't want to send people here. Some logged out. Boom. Gone. Excellent. All right. There's that migration fixing the timestamps. That only takes, what, 20 seconds to run? Something like that.

03:22:47All righty, folks. Thanks for joining me this fine Monday. The next... Scheduled Lobster's office hour is Thursday morning. NoGoodNick_ thanks for the stream
That's a 9am Chicago time, different in other lesser time zones. And if you need anything else, feel free to email me or grab me on IRC or anything else. Otherwise, I'm going to wind it down. eliphaz_ bye
Have a good one, folks. Take care.