I broke the build on purpose

02:19This is the Lobster's Office Hours stream. This is Lobsters. I'm Peter. And for Office Hours, we talk about the site or the code base or anything else that's pretty useful to folks. So feel free to drop questions in any time. And when folks don't have questions, I tend to work on the code base or site issues. And I have a couple of those lined up, so empty of code.

...56Yes, of course, I almost removed this. Sometimes I am not the no fun police. Mostly I'm the no fun police.

03:11So let's see where things were over in the code base. should throw a message in the oh i didn't turn on the notification on the footer so let me pull open a terminal there and touch that file and then over here in the chat i will throw in that message of

...50pushcx This is Lobsters office hours, you can post your questions about the site or codebase anytime.
There we go. And then otherwise, this one, I wanted to merge this that's been hanging out for a minute. And I think the submitter who said they didn't want to make more big changes merged in stuff. Oh, great. Yeah, so they pulled master in, which was very nice of them to fix that merge conflict. And then I think there might have been Something pending?

04:31Right. This I'm going to end up touching with story merging UX that's been in progress for a minute. So I'm going to just go ahead and merge it.

05:11He's not my favorite editing form. Okay. So fix 1466, which is,

...55All right. So that one's nice. That one was a seemingly small visual bug in the alignment of new comments that was actually just a giant slog to fix. So let's see. This one is... Ah, yeah. So kernel 53 here made some bug fixes over on the other one.

06:30And they've copied that over here and I figured out limitation of this, since the last stream I did a week ago. But I would rather merge this mostly working code then. yeah.

...56This is great.

07:08yeah this is the test that's going to have to get improved for that limitation so the there was a bug fix that i believe actually that kernel 053 made a couple of weeks ago where the last red marker doesn't should not update until you scroll back far enough to see it in case the there are a couple of pages of stories since you last visited The issue is, if you get up to like five or six pages probably nobody's actually going to click that far back and then you can never get the last red line back because it just sinks deeper and deeper. So I think what needs to happen is a new feature so i'm going to just say this is perfect enough the test suite is happy i'm happy because.

08:41shoot, what was the fix?

...5676. Let's see. This will have the bug, right? Yeah.

09:58So let's see. Oh, wait a minute. Oh, this is, so this is not, this is the build is failing because of, let's put that on the list.

10:17That's the time bomb test. Oh, and I didn't write the, what was that request? We just looked at comment alignment.

...34pushcx https://github.com/lobsters/lob…
Then we have merged that last frame, which is 1480. Nice. Good, good. I'm going to try and share these in the chat room. So if you want quick links, you can follow along. And so this failure should just be, yes, it's just the one. We'll come back to that because I broke the build on purpose.

11:07I'm just going to rebase that on.

...31Let's put this one in here. Over in 1394. I want to reopen this. I can do it in the comment box. OK. Yeah. Keep that open. I'll come back to it. The last red marker. pushcx https://github.com/lobsters/lob…
That's fine. Just juggling everything. Everybody's got too many tabs, right? So there's that. And then. OK. So this one.

12:42soon.

13:28Fix the the issue is.

14:08So many parentheses.

15:05I need to think about this one a minute, once I realized what I had misdesigned.

16:09imagination willing to include.

17:38It's this one. Okay. So still listed as being a bug. It's no longer a feature request. So I guess I will take these off. All right. Yeah, this one. It's funny. I thought I had thought through it entirely and then just used it a couple of days and i was like oh weird i don't see the last red line because i read it through my feed reader and the irc channel and my inbox and so it slipped past for me and then i as soon as i thought about why i wasn't seeing it i realized what i had misdesigned but i had seen this pull request come in two days ago and i went oh oh i've kind of crossed streams. So Colonel, I hope you don't feel like any time was wasted. You did exactly what I asked for, but it was me misdesigning things. So thank you, Utarsh. All right. So there's that misdesign, which I think I'm going to leave that open because maybe he'll grab that. And I will just move on to My time bomb test. Yeah. So this one has an example of what I call a time bomb test over in the story spec. I've done this. I think this is maybe the second one I've done on lobsters. So we changed some story urls and i wanted anyone with an open tab to have plenty of time to either close out that tab or use the form they were looking at before i broke that link because otherwise when they reload the page their thing is just going to be gone they won't get the redirect but i didn't want to maintain the redirect forever because otherwise you end up with one million redirects that are definitely dead and it's hard to read your routes and so as a middle path to balance these I put a time bomb in the tests and at the time of implementing the redirect pick day okay you get this many weeks roughly to use the tab or you know use it or lose it And then so that we don't forget, there's a test here that just says that thing blow up. So then, you know, I am forced to come back and clear that test failure for the build to work. So now if I run this, where is it? Spec, is it requests? No spec routing.

21:12Got to run the migrate. And of course, I'll have to do it again for the test env.

...28Oh, the command I was going to back up to. justkarolis hellooo how r u
Speaking of backing up to commands, I tried to use Atuin, which is some kind of clever history manager for the shell that has a bunch of features. Hey, Just. Careless. Nice to see you again. Doing all right. Was out of sorts on Monday because my sleep got messed up, but doing all right today. And Good. Alright, so the log. Let's go. pushcx https://github.com/lobsters/lob…
Let's go pull up this PR from Don't I think I separately included this. Yeah. I don't think I asked ruffle to write this time bomb test. No. All right. So then these are the redirects.

23:00So now suggestions lives there. And if you go there, we expect to get a 302. And that's just it. Actually, I think we can just delete them. That's what I was saying. Yeah. So it is a 302 now, right? Story writing is temporary. I must be getting an error because this temporary ID doesn't exist. Oh. So whatever this comment is, it's not a complete test. I must never have. So I missed what I should have said is do the request and get the response. I think I can just go ahead and delete this entirely.

24:22Yeah. So I'm going to go ahead and delete that. Easiest way, see, I feel like I'm a LLM coding tool because I'm fixing the tests by deleting them.

...48justkarolis you messed up by wrriting tests in the first place
Oh, yeah, you know, if you never write tests, the build can never fail. Real Galaxy brain take.

25:21There we go. That's easy enough. I didn't mean to lose my scratch. Did I not write the file? Sloppy.

...47Yeah, all right, I'm going to be just sloppy and push dash F it. i wrote the wrong message if i had written a reasonable message i've removed the old test that was broken fine do that in a second thing but i would end up with what two commits whose messages were swapped that would be incoherent wrong thing in the clipboard as usual and i'm just going to grab the pull request good All right, ticking along. Feels nice. Because I missed the Monday stream, it really threw me off my routine of touching the codebase twice a week at least. And so I feel like this stuff has been hanging out too long just because I lost the rhythm of it. So let's go ahead and do the Mastodon bug. This one came in overnight. We have the ability to link your Mastodon account and somebody found a new code path bug. justkarolis why is some of the text in cursive font?
Oh, this is the, what file was it? Mastodon app. When some of the text is in cursive because it is a literal value so it's either a comment or a string or a symbol and that's pretty much all the syntax highlighting that i find i need i'm pretty comfortable with just this amount because like the most useful thing to me in syntax highlighting is i leave off a quote and half of my thing changes from cursive to Roman or reverse. Or I... Yeah. So, not a lot of need there. justkarolis ahhh i see, which vs code theme is this? lol jk
So this one... No, I tried the... editor zed i don't know last month something like that and i actually had a whole hassle finding a color scheme that was plain enough to not drive me up the wall actually i didn't find it a guest visitor on the stream joel dropper found it for me which was very nice of him So the bug that came in overnight was someone was trying to unlink their account. And when you unlink the account, we try to talk to your Mastodon server to say, because Mastodon keeps a link of API tokens, or I think they call them app tokens that are issued. And we'd go ahead and try and revoke that to not leave stuff behind. The problem is sometimes people are disconnecting OAuth accounts because their Mastodon instance is down or dead or something else is going wrong. And this specific error was that the certificate failed to verify because it was self-signed. And that one was really interesting because what it says is that the person registered with a working Mastodon instance, and then something broke in their setup, or they set up some kind of redirect to a broken server, or they replaced the let's encrypt cert with a self-signed one between the time when they signed up and when they are now trying to disconnect. So the error of handling for all of these is the same. It's like, well, we'll just go ahead and delete our copy of the token and assume that the instance has taken care of itself. And I swore there was a error that already handled things that way. And I was just going to tag on to the existing

30:38Okay, and I have a house thing I have to do. So I got to run out of the apartment to grab a package from the porch. So where's my little note that I'm stepping away? There we go. justkarolis g2g to class, have a good rrest of ur day
All right, I'll be back very

36:17Okay. Thanks for patience. I had to run the stairs a couple of times. Because it was actually more than one box. Heavy enough I couldn't do it in one. Alright.

...41So. We were just gonna delete these things. Maybe I did it up in the controller.

37:20Here we go, yeah. This is the flow I was thinking of.

...41chamlis_ heya, how's it going
So how do we want to rescue? Frici Morning 👋🏻
Hey, Shamless. I'm a little out of breath from running the stairs, but it's going well. Fixing a bug. I think we just rescue it. And let the, oh, hey, Friji. Good to see you. And we'll let the views real safe.

38:11Yeah. And no matter what, they get that. Great.

...28We won't set a flash, so regardless, the user will do that.

...49Cool. All right, so we're kind of zipping along here. issue review. Yeah.

39:08So this one is now fixed. Why do I still see it? Oh, because GitHub updates the icon, but it won't go until I reload. There we go. And then

...31So I want to get into these three. This one is a note to myself, and I have to just slip through the prod database for that. pushcx https://github.com/lobsters/lob…
So I'm not going to do that one on stream, I don't think. Yeah. So this one is an interesting one where we saw some really dumb spam. And I say dumb, I mean, untargeted and just transparently spam. And it was written in that obnoxious Q&A format that Google has encouraged people to write things in so that they become featured search snippets. And I saw someone, so one IP address with the same browser string. like logged in as one account, clicked around for a minute in a very like, it looked like a user to me because they were clicking different links that were plausibly each on the page of the previous one and taking a couple of seconds to review things. And then they clicked over to settings, they clicked log out, and then they logged in as the next account and they did that some more. They went through the second one real fast and the third one they logged into. Yeah. The third one they logged into was this Mohan Pierce and they posted spam under that one. And it was such a weird series of events and log that i booted them for spam obviously because you don't post i mean garbage about how to break captures as a spammer without getting yeeted but it just didn't sit right you know the the quality of the spam and the fact that the accounts were a couple years old it just didn't add up And so I kept reviewing the logs and we kicked it around between the mods of, what do you see here? What do you see? And we figured out this was almost certainly account takeovers. Not that a Lobster's account is worth a heck of a lot, but basically every website gets hacked eventually. And if they had unencrypted passwords or there are... stealers and stealers are mostly windows programs that you get from running pirated software and or other various hacks but that's just the primary mechanism and primary target and they just sit there and they watch people log into things and they record all the username and passwords they see and then eventually the stealer logs get sold on some tor platform and they often end up eventually leaking and then various spammers and bots take the accounts and they do what's called credential stuffing where they just try that email and password on one million different sites and eventually this is probably the first time this has happened here It's the first time I've ever seen something where we saw just crap spam and it looked so out of character for the account. So all three of the accounts that were logged into listed a particular breach I guess it's not technically a breach because it's not like there is a stealer.com that got hacked, but it's just kind of the generic term they use for when they get a whole big batch of passwords. And all three of their email addresses turned up this one event. So we're pretty sure that this was credential stuffing. So I posted a feature request that we should create a background job to download the, so have I been pwned is this great, great site by Troy Hunt, who started collecting these various logs and breaches and things. And then I was like, well, you know, all these passwords are leaked anyways. You could just download them from me. And I think it's, it's D duped or it has account something like that. And we can update our site so that if someone tries to use one of these passwords that have appeared in a breach log, they get bounced. And I feel pretty comfortable with this because as a technical website, Most of our users are going to know to use a password manager or at least to come up with a reasonably high entropy password. Lobster's is sort of highly at risk for this because it's a low important website. And I say low importance, you know, in the context of when people do bother to have different passwords. They're usually reasonably smart about it, and they do it on the really important stuff like their bank account or their job. And then, you know, they have like one password that they reuse for every social website or online commerce at a new store that they don't care about. And that's the category that Lobster's fits into. So if we block those ones that have been leaked, We should be in pretty good shape. And H. Madison, who is 355E3B on the site, gave some suggestions that maybe we should just force people to run through the forgot password flow, which revalidates their email. Because that's the thing. I don't think that these spammers took over the email accounts. I think they only, because they didn't do the reset password flow. They just logged in. And then got a question that didn't read.

46:58Oh, so Daniel Heath, I had a chat with this person on Blue Sky Remastered on, I forget which, but he created a Rails plugin, a gem called has unpublished password. And I had looked at that before I posted this, I thought it was done or inactive because it hasn't gotten a commit in a couple of years and it looked like it wanted a bloom filter but there wasn't one checked in and there wasn't a script to create one this is a lot of state

48:04I see why he's designing that way though. OK. Yeah, this. Why does he say? Reenter password. because if we get results... Oh, because you need the password in plain text. You need them to go through the login flow to recognize that you've seen it before.

49:09I can see why this complexity would be useful in a commercial app, but this is a little bigger than I want.

...33Almost needed a gigabyte just to get a decent error rate. See, that's especially frustrating because

50:22I think part of what's happening with this design is he's doing in a commercial setting with corporate users where if they are paying corporate users, they really want to not get interrupted and you can afford customer service and everybody's kind of used to jankiness where the page loads and seem to seems to work and then boots you out. So that's not surprising. Hmm. Although realistically, most of those things are using single sign-on via OAuth now.

51:08Yeah. It's an odd sense of seeing maturity happen over the decades where things like sending email Now have so much abuse that they have a lot of complexity attached that the default for sites is don't try and send your own email, you have to outsource it to a commercial service. And it really sounds like. we're getting to the point where. logins and passwords are getting enough abuse and enough complexity that the answer is fuck it, outsource it to Google and do SSO, which is kind of offensive to me as an old fart.

52:26Frici It sadly isn't wrong, Email has been so heavily abused and complex it is a serious pain to actually run and maintain your own.
Frici and large providers are super hostile to non other large provider email (and sometimes even other large provider ones)
Yeah, Richie, so we have our own mail server and directly send and receive our own mail. Frici the email landscape is absolutely nightmarish right now. NotLikeThis
And occasionally, I think there was like a year or two where iCloud wouldn't accept invite requests from lobsters. And maybe every two or three years, I have to go and dick with some obscure post-fix setting, which I really don't like. chamlis_ having to do SSO is hurdle enough to mean I haven't tried tailscale
Oh, there was just a... Where was it?

53:01pushcx https://news.ycombinator.com/it…
Somebody was just, somebody had a mail server. It was like a, yeah, here we go. This was kind of a wild thread. I think Tailscale might not have SSO anymore, because when I load it, I get a prompt of, hey, do you want to enter your email address? But it could just be that they're trying to do a friendly intro to SSO flow of, we'll take your email, and if it ends in at Gmail, we'll kick you over there. If it ends in at Outlook, we'll kick you over there. I don't know. So this was really interesting. So somebody made this mail server in a box kind of setup, and it looks like a pretty immature thing, like the version number is 0.14. It's still the top comment. So here's someone who's just like, I went through a 20-hour Byzantine nightmare of setting up Postfix and Dovecot, and I already have a deep understanding of the relevant standards. And then the very first comment is someone who has the curse of knowledge where they have worked with it so much they can't see the pain and the confusion and the misdocumentation anymore. And so they just totally dismiss that just kind of a classic programmer thing where I've ranted about this on I think Twitter years ago, where There is a particular pathology of programmers where we suffer through doing these kinds of things. And then we're proud of that enormous slog of learning all of it and doing all the debugging and getting through all of that. And then we go, ah, I'm smart because I learned all of that and stop as opposed to taking the next step, which is, wow, I should fix it. So nobody has to deal with that crap again. And that's how you end up with these comments where someone is so proud of having suffered through it that they're like, oh yeah, this is all fine. No, this is a totally reasonable amount of complexity.

55:32All right, so let's take a look at this. Oh, that's a little bug.

56:01pushcx https://github.com/lobsters/lob…
So this is the, grab this link if anybody's curious. Yeah, speaking of professionalization, somebody gave me a hard time for using the RSS builder, because I guess it's not performant.

...45chamlis_ rss feeds feel like the ideal thing to slap varnish in front of if you have performance issues
like using erb was but the hassle of erb was erb is just string soup and so we were producing invalid markup and i guess they thought it was better to have invalid markup quickly than valid markup you know minusculey less quickly it was such a speaking of classic nerd opinions anyways the boss has decided to grace us We have not had performance issues yet with RSS. chamlis_ yeah, not to imply you were
And we do have some simple rate limiting. And then I think, wait a minute, what am I doing here? The RSS, let's undo this because I'm not reading correctly.

57:56Let's put that back. Lost my scratch again.

58:39I see. So it looks like I tried to tidy this before. Let's just do the one-liner version.

59:11Easy little fix. Ding. It is really satisfying to watch the GitHub issue flip from open to closed like that. So let's go ahead and deploy that. And a couple of other small fixes I've made. And while that's running in the background, we'll jump over to the last issue where I expected to spend some time.

...52Oh yeah, this one. So I touched it yesterday, but there's a... new production bug that the site is getting an OOM kill, which means every, it looks like 14-ish hours, just like the first OOM kill bug, the kernel looks at Puma and goes, hey, you're using up all the RAM. I'm gonna turn you off for that. And that's not great. So, I'll have to do all the debugging for that. I put it on my list. We'll see if I have time for it.

01:00:44I honestly wonder if it's just that adding the bloom filter thing to earn up bloom filter the geo block for the UK is it USA online safety act. Where. I got a sneeze.

01:01:32all right so the the geoblock has a little ip database and i just had to touch that the other day to say The default out of this gem that I didn't catch in the readme is that it slowly reads the entire database into RAM, which is not exactly a great default. And it has this low memory file reader where it keeps, I don't know, a thousand or one megabyte of hits in RAM. I forget which, but you know, not all of them. But the fact that it's fact that it's doing this means that it's increasing the memory pressure because every probably puma thread not just process but thread is allocating their own one of these i can't really share this even though it's read only across all of these processes cleanly uk osa is just a gift that keeps on giving and then puma I saw something go by on Blue Sky that Nate Berkopec is starting to work on auto tuning for Puma because that'll be real welcome. Because what I would really like for Puma is to say, like the reasonable default is start processes and workers and leave me something like 10% of RAM free. And I don't really care how many workers that is, because I want to use all the RAM I have available. And then you keep an eye on things. And instead, in the finest Unix tradition, I have to infer those magic numbers and put them here, or it's also in the systemd control file. and then manually measure prod every so often and correctly multiply those numbers to leave myself with 10%. That's not a game I want to play. So mostly I don't. And so I don't know if we were just under the threshold and then having to do this bullshit for the OSA made all of our threads just enough bigger that we're running out of RAM on prod, or if we have a memory leak, or if we have memory fragmentation again, or if our bundle size has just increased over time enough, or if something else is going on. It is not my favorite game to play with Rails hosting.

01:05:07pushcx https://github.com/lobsters/lob…
chamlis_ can you push the geoblock into nginx?
So this issue, a couple of days ago, a user wrote to me, puzzled by a story they submitted had disappeared from the homepage and there was nothing in the mod log. And since they had submitted it, can I push the geoblock into Nginx? Actually, the geoblock has to be in Nginx because it has to bypass the page cache. So Nginx has to know about it. pushcx https://onlinesafetyact.co.uk/
so yes i can and i suppose i will have to in a couple of weeks if i can't remove it speaking of it there is this guy neil brown who's a lawyer in the uk who has a personal blog. So this is the website with the best available information on the OSA. And he just wrote a blog post about that apparently is not on this site. Is it over here somewhere?

01:06:44chamlis_ https://decoded.legal/blog/2025… ?
I don't know where his blog post is.

...51Yes, thank you. Ah, it's on his business site. So Decoded Legal is the name of his law firm. Thank you, Shamless. Great find. chamlis_ had it in another tab
I was going to go dig through his recent Mastodon posts to find it. So... he wrote a opinion piece about how to fix some of the destructive deficiencies in the online safety act and these are the kinds of things that if any of these defects were corrected i wouldn't have to have the geo block of the uk He says this so politely. Neil has been donating time to figuring out the OSA on behalf of small online communities because he has a big heart, I guess. And it's been incredibly generous of him. And I just want to compliment him that he can still continue to write so charitably about the OSA and Ofcom.

01:08:14So let's grab this for the scratch file.

...39Just throwing it in the scratch.

01:09:03Yeah, this one, one thing that wasn't clear to me about Neil's post was he says exclude from the scope of part three small eg 15,000 multi monthly users or fewer services where the provider has a reasonable belief that the service is of no or low risk. And I'm not sure whether he means user or visitor. he's writing in the context of the OSA which defines the word user to mean visitor which is one of its most irksome quirks because it makes it confuses every fucking discussion about this garbage law where it takes a very common standard term and redefines it to mean something different and unintuitive, and so people hear things like, oh, if you have users in the UK, but no, it means readers.

01:10:21So not knowing whether Neil means users in the normal sense or users in the abusive OSA sense, it's not actually clear if his proposed solution is would exempt Bobster specifically but I generally yeah I can't tell whether he's writing normally or in that context anyway that whole thing is let's not spend a bunch of time sad and mad Let's improve things instead of losing more time to UK politics. So a user submitted a story. It disappeared from the homepage for them. They were very surprised because there was nothing in the mod log. And it was in their history so they could pull it up and it looked fine, although grayed out. What had happened was they had accidentally clicked hide on it. And if you haven't seen a feature like that, it's actually not obvious what's happening or what state the story is in, especially if you load it up. schmonzie missed the email landscape nightmare discussion, but (1) yes, and (2) I'm a developer of notqmail and happy to talk more about email challenges with anyone anytime (but not here unless it's what we're talking about already)
So I just wanted to review the feature a bit and then work through some improvements. Cause I realized there was, Oh, you're a developer of not QML. Hey, neat. I can't remember if... I know I've used that for something, but I don't remember what.

01:12:16Because was... Yeah, so not QML was like a repackaging of QML with... some ongoing development, right?

...46Anyways, yeah, we've kind of moved on from talking about the frustrations of email, but that is a,

01:13:04Ah, actually, that's why I'm thinking of it. schmonzie (if you're not already running qmail, probably you don't want notqmail. someday I hope that changes)
Our chatbot looks like it used to use not Qmail, which actually is odd. I don't know why it needed Qmail at all.

...26Yeah, we've run... So it's odd. I've pulled up the... All of our system config is in a repo called lobsters Ansible. And I was like thinking that I knew I'd seen not QML in there and mock turtle is our IRC bot. And I think this is just some kind of default stuff that slipped into here because our bot doesn't actually. Like the IRC bot doesn't send or receive email. So this is probably some kind of. default stuff that accidentally got copied into this Ansible repo. Or maybe Hunter was thinking of doing something different with it. I don't know what his plans were five years ago for that. Yeah, doesn't look like he's got a site up for it.

01:14:31To sort of answer your implicit question we use post fix in production.

...43schmonzie ha. yes, one of the erstwhile lobste.rs admins is also an erstwhile notqmail developer. maybe it's still around :-)
I have used Q mail professionally but it's been a long time like 20 years.

01:15:05yeah easy to lose track of things over the years i was just online in a discussion of picking dependencies because lobsters has outlived several of our dependencies before and expects to do so in the future so One of the mental criteria I have for, am I willing to take a dependency on something is if this became unmaintained, how quickly would it break? And if it broke, how likely is it that I could fix it and maintain that effective fork while deciding a strategy and executing a strategy to realistically, to migrate off of it, or to pick another version, or, or, or, right?

01:16:25All right. Well, oh, and I didn't say it, but Shmanzi, welcome to the chat. Hello. You can feel free to pipe up with other kinds of infrastructure questions or site questions or coding questions or anything, whenever. I usually put stuff on the to talk about topics list unless I can immediately address it. But you know, the point of office hours is the door is open, wander in and ask questions anytime, so. All right, so let's work on, brainwane (Peter, hi, just saying hi and thanking you for streaming. I'm only like 1/6 listening/viewing, but it's a pleasant way to feel like I have a bit more company as I plug away at my own tasks, and to hear you talk shop.)
story hiding there's a couple of things to do here and nobody has responded so i think the first thing to do is document the effects of story hiding because one of the major effects of story hiding is hiding comments and it makes sense when you think about it but It's not always obvious. Oh, hey, Brain Wayne. I recognize you, Nick. Nice to see you. Thanks for dropping in.

01:17:52Yeah, I try and just talk about what I'm doing. And I think one of the values of the stream is, There is very conspicuously no magic here. You can just watch me making typos and trying to read docs or not read docs.

01:18:16I've had a couple of people over the years say, like, oh, I didn't want to bother you, but... And then they mention something incredibly important. And hopefully streaming kind of takes down any perceived Wall of in approachability I am just a coder. Just kind of keeps nudging things along here alright so. let's find a place to describe I think story hiding is probably small enough, it can fit in a bullet point.

01:19:04If anybody doesn't recognize the Nick Brainwain, by the way, she's got a wonderful blog with lots of smart stuff around running open source projects.

...19So let's grab this. Stickers should move on down to trivia because it's not actually a technical feature.

...38brainwane Indeed. Back in like 2011, when I was helping Wikimedia Foundation grow how many contributors felt comfortable providing non-binding code reviews for MediaWiki patches, one good thing I did was set up a hackathon session in which our lead dev, live, on stage, reviewed a submitted patch. One newer volunteer said afterwards that it really demystified the process: "Oh, he does what I would do, just, more."
And then what else we got. Oh, this funny to run into this. brainwane Thank you so much for the kind words, Peter! https://harihareswara.net/ is my site
I mentioned the online safety act that has me wanting to resurrect our tour service.

01:20:09Yeah, Brainwind, I hope that is one of the values of, yeah, thanks. brainwane [smile] I get it and yes, and thanks!
I didn't want to link your site if you, I don't know, there's the whole norm around how much personal info does somebody want to share, but since you dropped by with a recognizable Nick, I figured it was okay to at least mention in passing, but yes, there's the link to her great blog. You know, I've actually been trying to get in touch with the Wikimedia Foundation. If you know anybody who's there, currently could you drop me an email because i've been trying to figure out where to follow wikipedia dealing with the uk online safety act if you haven't run into it the uk has a new law that says they get to censor the entire internet even the parts that are not based in the uk And it's been a big frustration and time suck for me. I would really love to read over their shoulder if they're doing any of that work in public. And I emailed them and I've searched around their site and I haven't had any luck. So if you know someone who currently does stuff with the foundation there, I would love a connection. Or Shamless, if you want to do your usual incredibly resourceful Googling in the background thing, I wouldn't mind. Because otherwise, I am kind of stalled out on anything there. So let's go up to the other story feature I just saw merging.

01:21:58I'm going to call it story hiding.

01:22:46brainwane I'll have a think and a check on whether I know anyone at WMF who would be knowledgable about how WMF is dealing with that. I found https://meta.wikimedia.org/wiki… on a first search but that seems several months old
Oh, well, this is, let me grab this link and drag it over here. Let's jump back into, oh, there isn't a way to jump into dark mode. Okay. So there's no discussion. Is there a, yeah, it is a few months old. the other thing is there's okay i can try reaching out to this person right so i saw this and i saw this blog post about the censorship and i know they so there was a like ah that's what i'm thinking of There was a open comment period where the regulator invited sites that they were about to censor to talk about it and then ignored all those comments and did nothing with them. And I knew that Wikimedia had included a public response. Yeah. All right, well, in any case, You could probably tell by the way I talk about it that I'm pretty frustrated by all of this. But thank you very much, Sumina, because it gives me another pointer for reaching out to them and asking if they have any kind of current material or current movement, because I really don't want to geoblock the UK. The whole thing sucks. And for anybody who doesn't have context, Speaking of history, why does Firefox do this? If we search for, there have been two lobster stories about this. pushcx https://lobste.rs/search?q=comm… https://lobste.rs/search?q=comm…
And so if you read this and this, you can watch, well, it's funny, I would have to add, chamlis_ there's this where they say they're evaluating it but not much more: https://meta.wikimedia.org/wiki…
sort by oldest but if you go through the two threads the lfgfs one and then the online safety act one that i posted a couple of days later after researching it and if you were able to read these oldest first instead of newest first you could just kind of watch me trying various things and doing various research and trying to contact politicians in both countries and then just getting more and more frustrated nothing working okay oh shameless let's see what you've found all right no dark mode so kind of jump to light what do we got hmm yeah this looks like they so this is a little incomplete, but a lot of the political rhetoric around the online safety act said that all of this is for the children and to save the children, because obviously, you know, sites like Wikipedia and a programming forum, when we're not giving children eating disorders, we're recruiting them into ISIS. And so it's appropriate to destroy them. And so a lot of this, brainwane https://meta.wikimedia.org/wiki… (the Liability Protections section) and https://meta.wikimedia.org/wiki… - if you search for "online safety" you'll find a few papers:
brainwane Online Safety Act in the UK (2023). Read our initial analysis of the legislation, our campaign with Wikimedia UK, and our final review when the Bill became law.
earlier stuff especially talks about children but the law is much more broad and destructive than that yeah this must be a couple of years old if it talks about it as an act huh only last year all right

01:27:16Okay, it's a year and a half. RNO. Can't do math. 7-8. Hmm. Oh. More links. Let's grab these in. I was trying not to lose another office hours to the Online Safety Act, but this is a thing that I've been trying to find for a while. So we've got Yeah, I tried to reach out to this team, but I didn't find this page.

...56Yeah, so Australia has its own Online Safety Act, and I really hope we won't have to go through this again, but... Initial analysis, our campaign, and our final review. This final review, I think, is maybe the one I'm thinking of.

01:28:28All right. Well, thank you. I'm going to save these for my later review. Parliament just had a chamlis_ I'm blown away that all this is dated 2023, that was three Prime Ministers ago over here
don't know the right uk politics term a debate a discussion some kind of discussion of it in parliament itself led by or started by you'll have to excuse me if i'm getting these terms wrong but by starting by one of the bill's proponents and one person in response pointed out that the osa was set to destroy small communities but it was just sort of Like one person said it, it was one sentence that went by. It didn't really get any response. Yeah, politics has been pretty interesting the last couple of years. Here too, Seamus. That's maybe getting a little spicier than I need to for the stream now. So...

01:29:52chamlis_ may you live in interesting times I guess
Does this page describe filters? Yes, and pass. brainwane Peter, I think checking out https://lists.wikimedia.org/hyp… may be of aid to you
Oh, a public policy mailing list. That's great. Thank you. So I didn't have any insight into how Wikipedia and media, Wikimedia were structured and like what was the groups doing it, but this has been very useful. Thank you, Sumida.

01:30:59I will check these things out. In my last comment about the OSA on lobsters, I said I was kind of out of things to try, but the whole situation is so awful and disastrous that I keep trying more things even since then.

01:31:40chamlis_ I thought last stream you were pretty hopeful and even talking about removing the geoblock warning; did something change?
How did I put it in the bug? No, I think I need my email.

...53Sexually.

01:32:16I just want to put this link later in the

01:33:02I kind of want to shift into second person here, but the rest of the page is in third, so I'm just going to leave that, I think. All right.

...32brainwane Cool, great, glad to provide some info. How I found that mailing list: I went to wikimediafoundation.org, chose the Advocacy section, then noticed the "Stay up-to-date" paragraph that included the mailing list link. If you end up not getting info you need, please feel free to let me know and I'll try to escalate with people I know who are at WMF?
went to Media Foundation Advocacy. Well, thank you. I appreciate that. And we'll get in touch if I need it. brainwane yay!
You've given me a lot of leads to go on that I was not finding on my own. So thank you.

...56I've reached out to a whole bunch of people about the OSA and A lot of it I did two months ago before I posted the lobsters meta thread and nobody had heard of it, even though the deadline was. had just been enacted recently. And I think. The lobsters plan has started raising awareness in tech circles like it's shown up on hacker news, a couple of times in the last few weeks and people cited and say oh. here's what it means and here's what's going to happen. And there are fewer of the really common misunderstandings, but I think still most people have not heard of the law.

01:34:51Just need to be parenthetical. All right, and then let me just kind of double check the features that touch hidden stories.

01:35:23Having a lot of irrelevant stuff.

...30Tied. Yeah, that really is it. Okay. So there isn't much more. I thought it was going to need a longer description for the about page, but that's fine. So what else was it going to do? The single story view under the list. All right.

01:36:00brainwane (Peter, not to absolutely thoroughly derail this to be ALL about OSA, but are you already chatting with Denise Paolucci at Dreamwidth about it?)
So when you view a story that you have hidden, There's the standard list detail, but there should be just some kind of explanation of why the thing is grayed out. And how do you put it?

...21This, we almost never show this.

...29All right. So we've got the markdown description. Let's go in here, yeah.

...44It's funny how there isn't a method to check if a user has hidden a story.

01:37:25just do the bear form like this. I'm getting the hang of this in Ruby 3.3, where if you just give a symbol like this, it's like, okay, this pattern of using a local variable with the same name is common enough. We'll use it. I'll have to test this, but...

01:38:28OK, yeah, I got exactly the query I expected. Good.

...41And that will work even if I pass a nil, right? Yeah. I mean, it should just return false, but... It's a little funny that Rails even ran a query. Doesn't it know that that column can't be nulled?

01:39:30And now I'm checking that it can't be null. Yeah, it can't be null. But Braille doesn't know that and work with it. That's unfortunate.

01:40:04All that story hiding, right?

...32See, I call it the single story view, but what I'm trying to make the distinction of is between a list of stories like the homepage. And when you view one individual story and the code calls it single story view, but none of the users call it. And so I'm trying to say.

01:41:54I'm going to say that it's all of those lists. Oh, I missed your comment there, Tsumana. I don't know Denise Paluki. I do follow somebody at Dreamwidth on Blue Sky, but I don't know this name. brainwane Yes, rahaeli is Denise.
And... to look her up.

01:42:39If you have a specific thing she's written about the Online Safety Act, I would really appreciate it. Oh, that's Rahaley. Okay, so I'm already following her. I didn't recognize the username. So yes, I follow her on Blue Sky, but I don't know that I've seen her write much about the Online Safety Act there. I will have to ping her, I guess. Are you mentioning it just because you're she's someone who's very thoughtful around moderation issues or do you know if she's done something specifically osa related.

01:43:41Strange language, the idea is you don't really see links into it, but this individual page works and so i'm trying to explain. why you can view it when you go to it directly.

01:44:33There we go. brainwane I don't know whether she has already written/done anything specifically about the UK OSA, but given stuff like https://dw-news.dreamwidth.org/… I find it very likely that she is plugged in to the world of people who run smaller social media platforms and have legal opinions about these kinds of things
If I say it as you've hidden links to this story and its comments from the site, from your view of the site, click on hide. It's full again.

...51Make sure, why am I explaining where the link is?

01:45:03update you i should clearly be putting her blog in my rss reader ah yeah well i see the link in her second sentence about the wave of terrible and unconstitutional social media bills yeah she's written a lot about the kids online safety act that is a so That is from the same people they literally have an organization and have given interviews about how they propose these censorship bills across the anglosphere so like. In case anyone is puzzled by the similarities between the UK online safety act and the Australian online safety act and the US kids online safety act it's because they have the same authors and the same advocates. and I feel more comfortable talking about American politics confidently. And there's in part because I just read more coverage as an American. There have been stories in TechDirt and elsewhere where the bill's proponents have been real candid about what the actual goals of these bills are, which is part of why I'm very comfortable calling them censorship bills.

01:46:32Let's just grab the link.

01:47:17yeah i have a so i really enjoy reading rayleigh's public notes on explaining why mods do weird things that they do because she does a really nice job of explaining some of the odd behaviors especially the things that look tyrannical but are actually very responsible and brainwane and further discussion on the UK OSA in the comments in https://dw-news.dreamwidth.org/… with some links to the European and UK groups she's following for more info on it
I have daydreamed about reaching out to her to ask if there is a cool kids club for mods because it would be nice to talk to other folks sometime.

01:48:02brainwane although it's from 2 years ago
Oh, some discussion in the comments of another post.

...15Ah, so she is way ahead of me. More than two years ago, when the Online Safety Act was still the online safety bill, she said, hey, please contact your MP, which is basically the exact thing I wrote. Yeah. Yeah. I had a heartbreaking experience a couple of weeks ago because I saw a blog post that did an incredibly good job of explaining what the Online Safety Act was and all of the problems with it. But it kept calling it the Online Safety Bill because it was written like two and a half, three years ago when the thing was first proposed. And the author tried to meet with a bunch of MPs to talk to them about it, and they all blew her off and none of the problems were fixed. And it is exactly as disastrous as she predicted it. And it was so i mean it was nice to read because it was like yes i'm not foolish i'm not jumping at shadows i actually understand what this means but then at the same time i read it and i was like oh this would have saved me hours of work if i had found this at the beginning because as disheartening as it is it didn't actually change at all meaningfully since she wrote that post i can dig that up if you want or it's in the stream archives

01:49:48So there's that. And then I was going to go in here.

01:50:00so you only see this just tidying these at some point some of these templates got line wrapped and it's just not helpful so only when i touch them i go ahead and join the lines up so it'll be approximately 20 years but eventually the templates will be all cleaned up so

01:51:15There we go. There was actually a lot of things that could have used polish for this user. I think, honestly, if any one of these things would have been done, they would have been set. That's funny. One of the things I've been thinking a lot, especially with the Online Safety Act, actually, it does tie in thematically here, is just how hard explaining stuff is where even after months of effort, I am still doing one on one explaining stuff about the Online Safety Act in the lobsters chat room this morning, you just have to just hit these things over and over and over and over. And so for hidden stories, which is what the whole feature is about making something invisible, which is Potentially very surprising, especially because you can one-click it and not get any confirmation, which I don't want to change. I thought about it, but I don't want to change. So I want to make a whole bunch of entry points to explain what's happening, like what's happening here. chamlis_ should that "- 1" be inside the pluralize?
And this is going to hit a 1 plus n on slash hidden, isn't it? All right. Yeah, let's deal with that.

01:52:40Yes, you're right. Good catch. Thank you, Shamless.

...51Yeah, can't subtract things off of a string, and what I'm trying to say is the n plus 1 of Heider count, so yes, good catch. Must be how you got to be a VIP. Ah. So there is a channel practice that's maybe worth explaining for anybody. If you catch a bug before I do, you get to be a channel VIP. Because I make plenty of bugs, so you have lots of opportunity to become a VIP.

01:53:27All right, so as long as I'm on the single story view, going to go ahead and add the mod feature and then come back for the one plus n so let's go to slash hidden because then i'll close out the tab and i'll catch it oh i didn't it's the silliest thing but you know i wrote the update to the about page and then it helps me a lot editing to just see it again on the actual page there's something about shifting from mono space to the output where I read it again anew. Tag page, that's a typo.

01:54:27You know, I say I won't appear, but over here I talked about it's about hiding links to it.

...49And then these cannot hide from their list of hidden stories good. it's sort of like how one of the best ways to edit your own writing is to print it out and grab a pen. Seeing it in a different medium off the screen is enough to shake you up so that you read it again and you catch stuff like those typos can't do all of it, but some. So let's see.

01:55:57In the model. Yeah, I can find the hightings. So let's go. Airbnb is so noisy. Let's go story, hightings. Reload. brainwane Peter, you already know about the Indie and Community Web Compliance Mailing List ? https://buttondown.com/indie-an…
Yeah, that's up in the controller.

01:57:10Indian community web mailing list. So many tabs open, I immediately lost that link. brainwane found via https://connect.iftas.org/libra… from IFTAS - Independent Federated Trust & Safety
Is this the promising trouble folks? Yes, I am on this mailing list. And I didn't watch the webinars they did with off calm, because I knew I would just get mad and I actually at that point I had talked to off calm enough that I had very low expectations from them and, unfortunately. Like off comes go to example of a small gaming service was something with 5000 simultaneous users which. they're just not. talking to small communities and it was especially frustrating that one the webinar that they did with promising trouble or that promising trouble organized or prompted them shamed them into doing i don't know what the actual sequence of events was because A thing they said multiple times in that webinar was, if you run a service, you can email us with questions and we will answer your questions. And I have emailed them, I don't know, five or six times. brainwane (I'm remembering when MetaFilter and Lobste.rs answered those research questions about trust and safety techniques alongside, like, WhatsApp. Just .... the techniques are going to be different at those different scales https://metatalk.metafilter.com… )
And one time I got a bullshit response where they sort of answered, they gave me the PR answer to the question they wished I had asked instead. And then all the other times they've ignored my questions.

01:59:11Oh, Metafilter. I know you're active there. Speaking of active indie communities, could you double check that link? It seems to be down for me. brainwane oh that "adjacent answer" pattern is so annoying, yeah
Or I'm having an odd DNS issue, but... Hmm.

...47brainwane The MeFi link is up for me
davidofterra FWIW, the link works for me.
calls load user votes It works for you. I don't think I have any kind of VPN or anything enabled.

02:00:24I'm just double checking because I have played with a VPN.

...40I've seen one or two. There's something odd that happens every once in a while in this browser that I set up for streaming because it's a standalone Firefox profile. And there have been a couple of websites I haven't been able to load. So it's got something funky about it, but I don't know what. So I only want to preload these usernames for moderators, and so I just end up shotgunning the logic everywhere. brainwane Stepping away but thanks for the stream and the conversation and the kind words [smile]
And so I've made scopes for some of these, but I hate pushing down this controller knowledge into the models. Yeah, well, nice to see you, Brainwind. Thank you very much. You've been really generous with your time finding those links. chamlis_ thanks for all the links
Thank you.

02:02:35Got a little competition, Shamless? I'm smiling because for anybody who doesn't know, Shamless has a very neat trick where I mention something obscure and suddenly 30 seconds later, there is a link in the chatroom from Shamless explaining exactly the thing that I was puzzled by, which is a very wonderful and appreciated magic trick.

02:03:12So I think I can just do that. And then let's double check that mod preload happens. Nothing? Nothing calls that? Oh, the story repo. Yeah. So these hidings should go to. This is going to do it on. chamlis_ I just love links, links links links, too many bookmarks, not enough tabs
So this mod preload is actually intended for lists, not the single view. Rails. And it's overkill because I don't want to actually do this on the single view.

02:04:19I only want to do this on a single view.

...29I'm willing to be sloppier in moderator-related code, but this is getting not great. I could actually want all of this. Because it would be nice if it appeared on the page.

02:05:06So I've got to go Hidings to User.

...16I really only want to pluck their username.

...24But I'm going to end up grabbing the whole model, because then I'm going to use the user. Rails.

...37This is me wishing I wasn't over-selecting, but that's not a thing I can really express succinctly in Active Record. What's the here, right?

02:06:19There's a helper here for usernames.

...49styled user link so here's a little bit of my syntax highlighting just careless if you're still present where when there's an actual syntax error i get a little red arrow and i get Other stuff. Like that brace was yellow. Expected token. Oh, because I'm going to make this a list, I think. Actually, no, I could just make it a scalar. I don't actually have to wrap that up. Because I only want to walk the association over there. All right.

02:07:52And then what else?

02:08:06I got a pig butchering text right there.

...25All right, so that should be good.

...43And then I had another... I mean, always with Rails, I have another one plus N. Too many bookmarks, not enough tabs. Shamless, since you do seem to be a big... reader do you use one second excuse me frog in my throat do you use any of those clever i think the old term for them is personal information management systems I guess I'm thinking here of stuff like Obsidian and Roam and all that other kind of stuff. I've tried a few of these kinds of things, but they're always more complex than I want, or they have no privacy and they're syncing. And I often, I mean, as someone can guess, I read a heck of a lot online and I like saving some of that stuff. And what I've just slowly settled on over the years is just file, print to PDF, because I can shove it into with all of my books and find it again. I don't know. If you use anything along those lines, to hear it.

02:10:44chamlis_ the closest thing I have is a load of org-mode files. bookmarks are just in firefox with no archiving for me at the moment
OK. Thanks for explaining. Yeah, I'm not an Emacs guy, but I have heard the values of org mode and seen very nice demos.

02:11:10So over on

...22I bet this is over on home controller, isn't it? Yeah.

02:12:22I'm going to open this story's paginator.

...34Or the story repo. So that wants to preload. Let's break this up a little.

...55Is this the only use of hidden by? I already had a hidden by current user method.

02:13:16I'm going to have to go check on that.

...27Thank you.

02:14:20to replace those, probably.

...27So if this is the only use of the scope, then it's safe for me to say that that should have the preload.

...52Except it's used...

02:15:05It's used to do that join.

...22See if that wants to work or if it's just going to make things worse. Well, syntax errors don't help. What did I break?

...46Something in here.

02:16:05I mean, it's not that, right?

...18Is it unhappy because the first item is a hash? And yeah, that's what it is. Yeah, I think that's it.

...35Yep. I think I could probably have done that if I passed explicit curly braces around that. Association named hidden story. Oh, yeah. It's hidings.

02:17:00I'm really just loading this page to see if I get the 1 plus n, and I don't.

...10actually i don't think i'm gonna get a one plus n unless yeah even if i'm a moderator right that's fine then

...44Let's put this a little back, because I'm not editing it. And then what else?

02:18:01What was that called? IsHiddenBy. IsHiddenByCurryUser. So it must be a helper, right?

...15Some kind of decorator?

...23Where is that getting introduced?

...31In the story paginator, and there is a matching thing on the story model. OK.

...44So I just added this. Where did I call it from? I'll have the story model in scope, won't I? Yeah. So I don't need that.

02:19:33Why am I not seeing the comments link? Oh, because it must be dropped off. That's really frustrating. All right, so let's unhide this. Figure out what I blew up. I don't know if I started typing there. Just throw that away.

02:20:50I'm having a hassle because I want to link to the single story view of a page, and because the story is hidden, it's not showing me the link.

02:21:08So let's just, I can grab it off of that, yeah. So if on this one I click Mod Edit, I can just Create the URL to see the view. Good. And then finish updating these to say,

02:22:28Yeah, there are extra copies of this because of heinous inline partial. I was thinking like, wait, why would I have it in four different places? That'll get overwritten, but that's fine. You can unhide.

02:23:00should be at story.

...10There we go. Nice little description.

...17All right.

...30I think that's everything. Let's bring that list back up. That's everything from the to do list, right? Yeah.

...42All right.

...57So we have that preload for mods.

02:24:12And then we have the about.

...20Yeah, and I moved the stickers down.

...28That's just the line wrap. That's the new thing. New thing. Repeated. Great. And then just because I know I touched something that's a hand-less inline partial, I'm going to bounce the server to make sure everything is rendered out.

...58What's the number here? 1482. All right. So with that up and committed, great. I didn't use the magic word close or fix.

02:25:41Alright, so the stream is at about two and a half hours and i've run through. Well, all of these topics of the alarm kill. Right. kaycebasques KEEP FIGHTING THE GOOD FIGHT PUSHCX
I guess I can.

02:26:04Thinking about what I want to do here I guess the answer is that I should.

...16take that heap dump like I did when Byroot came and paired with me. Oh, you could see me considering ending the stream, huh? Yeah, I was considering ending a little early. kaycebasques thanks for keeping lobsters great
But the thing with the heap dump is what I really want is to get one on a fresh server and then one when the server is up for a dozen hours. You're welcome. Thanks for visiting the site. And I guess dropping in on office hours. kaycebasques good idea to post link to the livestream in the footer
If you have any questions about the site of the codebase, feel free to fire away. Oh, thanks. Yeah, I just added that maybe two weeks ago. And it's manual, but... I've thought about scripting OBS so that it automatically gets turned on and off when I stream.

02:27:23Adjusting my little standing desk. I don't know how much of that whirring came through. I should make it a, you know, I'll embed the Twitch live player there, but only for the hot tub streams. That's the real content.

...54so let's go over to my own stream archives and i'm suddenly very glad i made these stream archives because if i didn't i wouldn't actually have any notes on this so oh no i didn't put it in scratch did i

02:28:26I might actually have to watch my own stream to remember what command I ran. I'm digging back. pushcx https://push.cx/stream/2024-08-… https://github.com/Shopify/ruby…
So this was, for anybody who didn't see it, this was the stream and this was the issue over in the Shopify repo tracking where I talked with, I grabbed Tenderlob at a Ruby conference and I asked him for help because it looked like a, memory leak out of widget at first.

02:29:05Yes.

...11And then all the interesting stuff is hidden because GitHub doesn't want you to read your comments where you really get into the depths of figuring something out. It doesn't hide the early ones where you have no idea what's happening. It hides the ones let's see all right we wrote this whole we casper wrote this script and then i had to tweak it i think that's the commit i want to rescue this code from

02:30:00OK, yeah, so this was just stats. That's not actually useful. What I wanted was the full heap dump. And then I had this strip heap command that, actually, I got to download that. I should save that one. Because if this user, Casper, should come along and delete their gist,

...34Put that in my blog folder and I'll sort that out later.

02:31:13chamlis_ by the way, I used https://twitchtokengenerator.co… to generate a token so I can use twitch chat over irc. obviously you're trusting this third-party, but you can scope the token to only allow reading chat, which may be enough to let you do some integrations
I still, you know, especially looking at these graphs reminds me, I still think this is probably the, what do you call it? I still think this is probably the IP lookup for the OSA geoblock because I've had to just the timeline lines up very neatly. And if their thing is accidentally got a small leak or a bug, we would end up seeing a chart like this.

02:32:12So let's.

...32I kind of want to bring back these stats so that I could make a chart again.

...45Yeah, let's do that. So that's config puma, and we will just dig back in the history.

02:33:03Oh, I left it just commented out. And wait, this stuff is still hanging out in prod?

...18Looks like I've never turned off that debugging code. So I already have the heaps that I want. OK. Wow. Well, let's grab that strip heap script, which is not sitting on prod. chamlis_ here's one I prepared earlier
Is that still on my clipboard? Yep. So let's grab that.

...54Oh, and Shameless, I did see your link for the token generator.

02:34:03That's real interesting. I had not heard of this site. mjiig I'm not sure if that reference crosses the Atlantic
And I really thought you had to do them in the...

...26chamlis_ basically they're the ones who registered an application that you're piggy-backing off of
in the Twitch console.

...35Ah, they registered an application. Yeah, I don't know. Maybe that's enough.

...48I'll check them out. Do you know off the top of your head if they've been around for a few years? Frici you do need to register your own app pretty sure.
Because that would be nice if they have a reputation of being nice and stable and boring. As opposed to, you know, if they're six weeks old and a lot of people are like, hey, a weird thing happened.

02:35:11Frici as for swifty they have been around since like justin.tv days one way or another pretty sure. very solid person
Well, we can ask Shamless, did you have to register your own app to do that?

...26chamlis_ no app nor phone number needed
Oh, okay. That's really promising. Thank you, Fritchie. It's great to hear that reputation.

02:36:02chamlis_ I got there from https://efertone.me/posts/2023/… via a deprecated equivalent service that recommended them
Ah, neat. Yeah, fun fact, I actually use WeChat for my IRC client. Frici oh actually they are still employed on twitch... my pessimism thought they were caught in recent layoffs.
And man, I hit a frustrating limitation in it the other day, where I would really like to be able to optionally connect through a bouncer. So like if the Bouncer goes down, just connect directly until the bouncer comes back up. But in WeChat, you have to see that as two different chat networks. And so you can't really toggle between them.

...53Let's remember the interface. Okay, so it just does put so.

02:37:09chamlis_ I guess you could use nginx or something as a tcp proxy with fallover
chamlis_ chat is really simple, a solved problem
Or did I just call it?

...49I should probably have just copied these back to my home machine, because as I recall, I think I... Yeah.

02:38:11Nginx as a... You can use Nginx as a generic TCP proxy? That's interesting. You're talking about WeChat?

...26So the issue is I have a shell box. And I've had one for years and years. And I use IRC by SSH-ing to the box, and then I have a TMUX session that WeChat runs in, which is very simple and bulletproof. And that's been fine for, you know, I mean, it was IRSSI before it was WeChat, but that's been basically the same for like 20 years. chamlis_ yeah it can proxy tcp, but thinking about it I'm not sure how you'd wrangle all the details so it was transparent to weechat
And nothing broke with it. but Tailscale kind of broke the setup. Because the big, the only reason I needed like a $5 a month shell server is for all the odds and ends I would do when I was working out on my laptop of proxying connections around and all this kind of stuff. Or it would be the SSH host that I would tunnel things over to my home network on, that kind of stuff. And with Tailscale, all of those problems disappeared. And so now I have a $5 a month shell server and literally the only thing I use it for is WeChat. And IRC... It's hard to say that IRC is worth $5 a month. I would like to turn that server off. But if I move WeChat to my home machine, that goes to sleep at night. And I do want to see the scroll back on stuff. And if I get a bouncer, well, that would need to be on all the time. And so I'm back to $5 a month because the bouncers charge for it. or there is someone that I'm trusting with my DMs, which is because IRC doesn't... I would say its security model is maybe a little outdated.

02:40:46And so I was thinking like, well, I could throw a, you know, a bouncer up on Lobster's production. There's a shell server that's on all the time, except sometimes I deploy to that and the times that I most want the bouncer, or when I'm logged into IRC so that I can explain why prod is having an outage.

02:41:08IRC just doesn't fit my happening modern lifestyle.

...39Frici IRC doesn't fit most people's modern lifestyle tbf.
Yeah, Fritchie, you saw the Notes blog post about that? Because that was just collected from years of... We've been kicking around IRC in IRC. chamlis_ I'd offer my bouncer but I'm not aiming for maximum uptime and it is increasingly patched while I wait on upstream to review things. plus I'd understand if I'm within your threat model ;)
Oh, I guess there's a development for that. mjiig With tailscale, could you replace the bouncer with a raspberry pi on your home network?
I get to feel prescient. So...

02:42:17Frici https://push.cx/discord-vs-irc-… this one ? I have it on my links to throw at people notefile
MJ, that's not a terrible idea. Yeah. Preachy, that's the one I was talking about. MJ, I'm going to come back to your idea in a second. I'm going to let that percolate.

...40So one of the things I said as a final bullet point was Discord is a single service run by a VC-funded business. B2C venture capital orients to growth metrics, so quality will nosedive hard when growth plateaus and the established process can't adapt to that failure, in scare quotes. chamlis_ discord moved towards IPO recently I believe
And along those lines, the New York Times and Bloomberg just reported yesterday that discord is starting to talk to bankers about the ipo process which is you know bc funded businesses are either going to ipo or get acquired and this is when a lot of this stuff really kicks in because now they need to make the next quarter's numbers yeah just in the last day or two fritchie so you have you said something really that caught my eye there the You have it on your links to throw at people. Is that a, cause I was kind of like, I collected a whole bunch of rebuttals and stuff because you hear the same stuff over and over. Like, honestly, this whole thing is like, what does discord have that IRC doesn't? And the answer is like 400 features. And I linked, you know, I described the ones that we want most. But a big chunk of that is if you are having a discussion, you mentioned one thing and you mentioned message editing. And if you only mention one thing, the IRC fan is like, oh, you can get a weird bot to do regular expressions to do that because people love regular expressions. It's only when you see like 30 or 40 of them in a row that you're like, yeah, maybe you can't really paper over that with bots like the mobile app. So if you are using it in that way of being a comprehensive rebuttal, I'm really glad to hear that. That's part of the reason I collected up these notes.

02:45:16Broken activity UI. chamlis_ time for the lobsters signal group chat
It's so funny because the Discord fans are like, I never see this. And meanwhile, on the app, just for the last two weeks, a server that's like some friends who live here in my city, so not a very active one, had a bug where the server always shows as active in one of my discord clients like one install I think it was just on my phone or it was just on my iPad and nowhere else and so like I would click into the server and choose like mark all red it would be like I'm totally red and click out of the server and then the activity notification would just turn back on it's I understand that activity UI is a surprisingly complicated problem of distributed state but it's just frustratingly unreliable.

02:46:24Signal group chat, yeah. It is maybe worth mentioning that we have like 700 people on IRC and I think Signal chats are limited to 256 and they have basically no mod tools. It's not a criticism of Signal. Frici signal group chats are different scope of chat and usage as well after all
Signal is intended for small group chats with a small list of trusted participants. It's not a meet strangers and chat online kind of thing.

02:47:04chamlis_ yeah, wasn't a serious suggestion
Yeah, just it's different use cases in the same way that one of the things I said in those notes is Discord is about playing video games with your friends and watching other people play video games it's not about having discussions yeah yeah I knew you were being silly as long as we're being silly about signal actually it's not silly the one feature I wish it would add would be location sharing because I have slowly come around to like when I am catching up with friends it would really be nice to be able to Take a signal group chat and say here's my location for the next 30 minutes or until I meet up with you all or. chamlis_ I think it has that: hit the plus to the left of the text box, then hit location
You know, for the rest of the evening, because the like are you there yet i'm five minutes away who's in a cab I caught traffic like all of those things I would like to throw that away and. signal is pretty much the only APP that I would actually trust with location sharing data.

02:48:17It has a location. What? Hold on. Frici On the same line of thought (loosely) how is the zulip experiment going?
I have to pull up my phone. What it has. Oh my gosh. I see location. I want a feature that it actually has. Oh no, I think this is, so I clicked on location on the signal app and it okay so i can tell it me personally or i can like share a link on a map oh this is pretty neat all right i'm not going to actually play with this more on stream but you just nerd sniped the heck out of me that's the exact thing i want thank you here i'll put it on my list

02:49:22Guess that says how often I attach files to signal messages. So for Zulip, slowly. I've seen a little bit of activity, but I haven't started posting and I haven't run down the owners of sister sites to try to drag them in yet. And MJig, as long as your name is actually visible on stream here, because you were chatting on the heap dump stream, To your plan. I don't actually have a Raspberry Pi. That is the exact thing I would want of a low powered, always on kind of device. And I don't think I have hardware around that fits the bill. So it's funny if my gripe is that I'm paying five bucks a month for basically IRC bouncing. I don't know what the current Raspberry Pis are. That would probably be 25 or 50 bucks so it had a little earn back but that'd be okay

02:51:38Hmm. Right. We had this heap profiler or reap.

02:52:10chamlis_ I think SDF will let you use IRC for $36 one-time
Who's SDF in this context?

...25Oh, I've heard of these folks. SDF.org, you're thinking of, right?

...45I've seen blogs on this site, I want to say.

02:53:10chamlis_ yeah, under join they have their "plans"
twtfmb Well that was a short-lived plan
We had an experiment in 96 and it didn't work out. So here we are 30 years later. Under join, they have their plans.

...33Man, DSL, I haven't thought about that in a minute.

...57there's a little difference here between being allowed to connect to irc servers and being allowed to leave like a client open indefinitely which choose up ram or run a bouncer indefinitely

02:54:39chamlis_ good point
Oh, sounds like they have a bouncer installed. Well, if it's installed and recently, you're probably allowed to use it.

02:55:00I'll have to look at that. Let's put it in my notes.

...10Shameless if you can turn up like maybe an experience report of someone mentioning that they've done that there. It sounds like they're pretty well organized on plans. And the reason I mentioned that is I don't want to pay $36 to find out that it won't work.

...54This is probably going to, I don't even know if I have a Rust installed. Yep. OK. OK. Why do you want a cargo.homemold? I don't recall asking for one. Oh, it says something on dash C3. pushcx https://github.com/oxidize-rb/r…
It's not going to change this error, no. All right, who's the Rust expert in the chat room to explain why this tool doesn't want to install and run?

02:56:35chamlis_ you need to be in the reap directory for that command
But I remember we used this to dump out flame graphs, which were pretty darn useful for immediately spotting things. I need to be in the reap directory. chamlis_ or `cargo install reap` or is it in arch?
Oh, so I have to clone this source. Or I could probably run cargo install reap. Yes. It might be an arch, but looks like this is running. So now we just wait an hour for it to compile. ghost_user_1984 It's in arch, but it might not be in your path.
No, got a binary. All right. ghost_user_1984 Also push can I make a Feature Request?
So I said, all right. So this dash dash is separating arguments from cargo run. ghost_user_1984 Can we have some way for other admins to do TOTP resets?
ghost_user_1984 I have someone on IRC right now asking.
ghost_user_1984 <3
yeah what's your feature request burt yes and i think we even have that one github issue to track this

02:58:24Have you verified in some way that they own the account?

...49ghost_user_1984 I'm asking them to email you because I'm pretty sure I need DB access to fully reset it.
The production Rails console or database access, yeah. But if you could, if you have a way of confirming that they own the account, like it's linked to a GitHub that they can demonstrate they own or something, then I can just, it's a one-liner and I can do it on stream. ghost_user_1984 Sounds good.
Well, I would probably put their username off stream.

02:59:22Oh, I see.

...58So I was just... Cameron away into IRC anyways Thank you ghost user for. ghost_user_1984 That's what I'm here for.
prompting that so let's go.

03:00:28ghost_user_1984 Well that and helping with MariaDB when it gets upset.
around. I'll push the X code lobsters. Let me drag this off screen and take a look at it in a second. What did I typo?

...51Oh, it's the. All right. It was the dot in the directory. All right. MariaDB, yeah. I was so, until we realized it was the Ubuntu updates, I was real worried about the prospect of having MariaDB restarting because, you know, core data. So let's see. So you'll have to watch the URL, but we got the boot flame graph and the bloated flame graph. So what's new? ghost_user_1984 I still like the time it didn't apply a migration and we spent a few hours on that.
We got like a thread hanging out. This new Ruby VM object is 20% of memory. I'm not seeing anything that's leaping out at me. It's an array. I18n. Yeah. Mail parser. chamlis_ is jemalloc definitely still doing it's thing?
So who's this that's so deep? Let's check these other two deep ones. Who's this? chamlis_ its*
More I18N. Rake. Oh, so this SVG stuff is for the stats page. That's fine. All right, so what's big and chunky down here? More threads. There's nothing in here that is jumping out at me. Obviously, I don't have a Beirut's experienced eye. Is Gemalec still definitely doing its thing? Yes.

03:02:47GitHub? Is GitHub having an outage? Am I having an outage? This page is taking a surprising amount of time to load. All right.

03:03:03So yes, this is still here and it's still in the service worker. So that hasn't gone anywhere. That was my first thought too, because that was what the original cause ended up being.

...22So this is bigger, but I don't know that that's bad. So it's gone from 680K to... 20, call it 22 megs. I mean, multiplied out by the number of workers, that is substantial, but it doesn't immediately mean anything to me. taimouraaa hi how did you got this svg.
I really did expect that we were going to see something related to the MaxMind database in here, chewing up a bunch of space.

03:04:12pushcx https://github.com/oxidize-rb/r…
But it doesn't look like it's misbehaving. Oh, hey, Timura. I got it out of Reap. pushcx https://github.com/lobsters/lob…
Made this SVG out of our heap dump, which is coming out of our config Puma. So it's configured to dump once at startup and once after looks like a Some time has passed I don't remember what the exact definition of this was. Because it I thought it was like. After a half a million requests, but I don't remember what the word slots means in this context.

03:05:04These are going alphabetic. taimouraaa looks interesting.
Yeah, so if MaxMind is in here, it's so tiny I can't even manage to put the mouse cursor on it.

...27Well, I have looked at the heap dump graph, and I have no idea what I'm looking at. The only thing that pops out as conspicuous to me is that the Ruby VM is bigger, but that... is maybe the YJIT's memory usage. So that's probably fine. Boy, I don't know. I don't know much about Ruby internals. And honestly, everything I learned about Ruby internals is whatever stuck around from talking with Bayreuth on the previous stream.

03:06:31Yeah, I don't know what to investigate next on this one.

...49I guess I could uncomment the... Let's go ahead and do that. I'll uncomment the stats.

03:07:09And that'll get me some amount of introspection. I can generate one of those memory graphs like appeared on the previous thread, but that does take a dozen hours. And if I look at that, and it has, where'd it go? It has, I don't know, that's the slots. If it has one of these same profiles, maybe something did break with GemAlloc. Because if there's nothing conspicuous in the flame graph, and it really does look like what I saw with Byrick previously. Yeah, so I'll run that. We'll turn this back on.

03:08:32But it didn't push. So let's. Actually, you know, I'd rather regenerate these because it would be nice to have them from the exact run as I'm generating graphs from, so yeah. So if I drop those and then chicken amend, and then I will deploy that and I'm going to have to manually log into prod cause it doesn't. Yeah. So generally it tries to be smart about reloading the app when possible. and then only doing a hard restart of Puma when needed, but it's not completely clever, and so it doesn't recognize that... It knows if anything in the bundle or the Ruby version changes, it has to do a restart instead of a reload, but it's not quite so clever that it knows that if any of the one-time files like these things in config change, That's reload instead of restart. Just didn't want too many moving pieces in there. All right, so with that pushed, I'm going to get that deploy started. And if anybody has last questions about the site or the code base, I'm happy to take them. But otherwise, I'm going to wind up this stream because we're just a little over three hours, which is my usual target. And not too much else going on.

03:10:38yeah and then if you are listening person who was waiting for their totp reset i did see you say that you were going to drop by the stream so if you are here hello and you know give me five minutes to knock around and then i will come and do your totp reset the most useful thing you could do is email me from the email address you used on your account so that i know you're you Or if you linked a GitHub account and you can post a gist, then I know you're you. That's about that. Yeah, see, it's doing the phased restart instead of actually the full restart. So I'll let that finish. I could have Ctrl-C'd it, but... So we're going to take... Probably 60 to 90 seconds of downtime there, which is not my favorite thing, but that'll be okay. Alrighty. chamlis_ thanks!
Well, thanks for hanging out with me, folks. taimouraaa bye
This has been the Lobster's Office Hour stream. The next scheduled one is Monday at 2 p.m. Chicago time. There is a... twtfmb This was unexpectedly soothing, thank you.
taimouraaa too many options hehe
schedule on the twitch page that you can click on if you want like an ical kind of thing or you can click some kind of follow or subscribe button if you're logged into twitch and it will give you a notification when i go live or you can just email me or message me on the site or on blue sky or on irc or or happy to talk about this stuff anytime there really is no magic take care folks