to hell with random acts of kindness, I want pączki

02:59pushcx This is Lobsters office hours, ask questions anytime!
One for Anubis, there's one for SQLite. There's gotta be issues.

03:09Oh, I haven't gotten the... I don't know, what is it? 16.02.09 stream archive. Didn't get that done this week. Ah, busy, busy. My whole morning got derailed by the cat, because he was in a big mood. He is currently chilling on my lap after a lot of very bad behavior. We'll see how this goes. graefchen Cats do be cats. limesNodders
But if I suddenly start cursing, it's because he, decided to take a bite out of me. Because sometimes his feelings are so big, he can only explain them with his teeth. Which, you know, hashtag mood. Yeah. He, We adopted him when he was about three years old and his previous owners had not been good to him. There were some real clear signs of both neglect and abuse, but he is in a better place now, but every once in a while he has a bad day.

04:25All right. Where's my, okay. Getting my windows all set up here. All right, so when folks don't have questions to ask, I work on maintenance on the site. And the big thing is a new pull request from Hunter, who is one of the site moderators. His username over on the site is 355E3B. And then... graefchen I am happy that we got some pretty chill Romanian Street cats that are cudly and fairly easy to work with. limesSit
yeah we've got some other nice stuff from last stream to continue hopefully thomas will be able to drop in and we can pick up the database migration work ah great that's nice yeah we don't have street cats here in chicago but you know i see them Most everywhere outside of the States.

05:38This guy, you know, we are one of those households that gives the cats a hundred nicknames. One of his nicknames is unfortunately Beast because sometimes he is just beastly.

...56So let's see where we're at. Where do I want to start? Let's start with something small here. Because I think these two are probably ready to go if their authors have checked in since the last stream. And then we can get into Anubis and the other thing.

06:18So let's see.

...27Oh, opacity. This is a CSS issue that will never die. And somehow I haven't seen this PR.

...48graefchen Nicknames are always funny. We got Queeny for one and Cleo for the other. limesNodders
When sites have negative stories, Queenie for one and Cleo for the other, those are, Some pretty classic cat names. And it doesn't escape me that both of them are royalty related. Is Cleo short for Cleopatra?

07:20graefchen Correct.
Yeah.

...29Save button looks lighter than the rest. And the hidden is not affected. I see the difference. Man, this opacity for stories with negative scores is kind of a gift that keeps on giving.

08:03in that it never really stops generating minor bugs. The only thing this adds is form. Yeah, OK, this is one line.

...32Oh, because we changed how these buttons work. Yeah, so these were links that we hooked JavaScript onto, and they have been becoming buttons. This is just white space, right? I can't see the... I'll have to look at the main view.

09:10Why not just delete this?

...20I got to look at the PR, but this feels like a work in progress kind of thing. GitHub is not good about PRs where people revise their work. And I know FedAmp likes to do that. So I'm guessing I am just seeing work in progress. pushcx https://bsky.app/profile/push.c…
oh man speaking of github there's one thing to bring up i just posted on blue sky and mastodon looking for suggestions the other day but did not get any responses to this unfortunately So I'll throw the link in here, but the short version is GitHub's uptime has not been super great lately. And I've been talking for, you know, the last maybe six months that the endless parade of front end breakage is pretty frustrating. And as a free user has me questioning like, hmm, is it time to start, you know, worrying that one day github will go away or stop functioning well this is a lot more concerning and these ones that classify as minor are things like it's just pull requests that are broken it's not the whole site and that's It's like, OK, so it's only the core workflow. It's not the whole site. That's minor. So that's been a little rough.

11:14So if anybody happens to look at that and think, oh, I have an option. I've been thinking about that git dash bug tool I've mentioned before. And I played with a random Python script. I'm going to take another look at Gitbug. And I don't know. If Gitbug doesn't want to do what I would really love is working bidirectional sync to some kind of human readable files in the repo, maybe I will vibe code something. Okay, yeah. So third commit. He removed it. I got to see the full PR diff just to make sure I get it. But this looks great. Vote counts color.

12:15Ah, yes, it should. OK. I really appreciate his attention to detail.

...42Is. Can I use dot com? Is. Is this widely available? Because we make an exception for has.

13:01Baseline widely available. Not bad.

...15All right, so comment bad is voters, is byline not drop down parent, is the flagger is the comment text.

...35easeout sounds tricky to map the problem to git when your average bug tracker doesn't have a branching timeline
I wonder why he didn't just write star.

...46Oh, because it's just the it's just the header. It's not the comment text. It's as well.

...57Ease out. Can you give me a little more context? Because I think maybe you're a second behind and I'm not catching what. Oh, you're talking about the graefchen I like css and what you can do with it. So many good changes in the last years. limesSit
syncing issues into the repo? Yeah. easeout yeah that topic
Bidirectional sync is a lot nicer for if we really wanted to move from GitHub to another forge of some kind. We would really want to have bidirectional sync between GitHub, our repo and, you know, the new forge. That would be lovely. Because then, you know, folks can migrate away at their own pace. At this point, there are no alternative forges that look like what I want to see to take on the pain of moving forges. And so I'm sort of willing to take on just, just sink me down one way to the repo so that we don't lose everything if GitHub goes down or puts up a paywall or rate limits the shit out of everything. I mean, like GitHub can barely render 10 comments now That's the web UI. It hasn't happened to the API yet, but it feels like the clock is ticking.

15:56And that doesn't, to be clear, that doesn't super freak me out because we only have, what are we at? 1889. We only have like 2000 issues in PR. So even if we had to fetch them, easeout mhm GitHub's become pretty sad
With a really strict rate limiting on we could fetch them in a day or two with you know just really conservative rate limiting. I just don't want to lose them entirely because there is so much good stuff so much absolutely vital stuff of planning and decisions and everything yeah I don't know if you joined in you know it's in the chat scroll back that link to the graph. That one is. that little uptime chart is just painful to look at. And they have said, you know, their two big priorities are rewrite everything as a single page React app. And then that was their only priority for a long time, at least public priority. And then they said, okay, well, we're also going to add marcoroth_ I feel like using the `gh` cli you can pretty much get everything you need
moving into azure because you know of course we should be eating our own dog food and azure is the best cloud and it's just like okay so that one the only user visible change from them moving to azure is going to be the inevitable chaos of moving the ghcli well that's an interesting idea i hadn't thought about it So what I really want that a lot of the tools don't do is if I go leave a comment in the middle of this PR, like I almost did, the tool has to get this comment included, which means it has to get something about this diff and just anything. Because we've had a lot of important discussions in those inline review comments and whether or not, you know, whether I do the add single comment or open a review. I'm going to rebase. Oh, man, maybe it's finally spring. I'm hearing birds outside. All right.

18:38bsandro VoHiYo greetings cyberpals
Oh, hippy Sandra. Bring this merge. Great. So let's get that in. And then, all right. So this one reviewed on stream on Monday. And I want to say there was one minor thing I asked to have changed. Yes, a little bit of duplication. Let's take a look, because I don't remember the exact line of code. Yes, that was it. I wanted a constant.

19:25Great. Yeah, this is ready to go.

...42Yeah, speaking of why I really want review comments, there's one example here of, or actually maybe it's just, no, it's in the main PR body, but like, this is why I want all the PRs is like, these links are valuable. And if I'm looking back through the get blamed to figure stuff out, Nogweii kitty tax
don't bite me why was i talking too loud do you not like that i gesture with my hands when i talk you have to be good sir sir this is going on your permanent record sir ow god damn it okay sir you are getting evicted hey you've got to be good

20:30marcoroth_ yeah, you should be able to get this using: `gh pr view 1895 --comments` and you can also get the raw JSON output
Nogweii LUL is it still a kitten?
cat chaos i swear i don't even know what prompted that one cat tax i should just put up a picture of the poop emoji i only didn't break skin so i don't have to stop and go like find an alcohol white that's just annoying so one of the things that's tough with this cat is that he's smart So he can, if he is offended by something, he can stay mad for, I don't know, five or 10 minutes. Like remember that he's mad and who he's mad at and come back into the room to start shit again.

21:24Marco, that sounds perfect. Let me put that into my notes. Thank you. Let me just grab this whole thing. bsandro haha one of my cats is the same, very moody and remembers stuff
Come here, Twitch. Let me select.

...45So Nagwe, no, he's not still a kitten. He's now, well, he's technically a senior cat now, actually. I want to say he's nine now. hono5kami heyya
there's a little bit of vagueness because we weren't sure how old he was they told us about three when we got him and the vet kind of was like hmm so maybe he was a little younger I don't know yeah and I like this guy and I'm glad we're experienced cat owners so that we can give him a supportive environment we've done If nobody has had problem cats before, you can do things, especially when they're smart, like train them to do tricks for attention instead of do bad behavior for attention, because it is true of all animals, including two-legged hairless apes, that at a certain point of neediness for inclusion and attention easeout there is a repo-scoped event history endpoint too. the schema looks detailed https://docs.github.com/en/rest…
that at some point all animals become unable to tell good attention from bad and in a cat that has been neglected it shows up like well if the cat really wants to play and god kittens need to play so much but if people can't keep up with them eventually they turn into bad behavior things like you know if i bite this person i will always get attention Oh, thank you. He's out. Let me grab that for my notes too.

23:41Maybe we'll look at that on screen. Probably not. I have plenty of PRs to keep me busy. Anyways.

...53So with cats you can train them into tricks of hey here's this good thing I can do to get attention. And something in the last week he's sort of forgotten all his tricks and he's been testing boundaries. I would rather just squash and merge these.

24:48Sir, sir, he's coming around for affection. Sir, you got to be good. You got to stay good. To flop, sir. He has a whole trick where he flops over and plays dead and rolls back and forth. I don't know if he's too moody to do it. All right, to flop. To roll. To roll. roll all the way sometimes he just kind of puts his belly up and shows bunny paws he has to actually roll all the way over to roll i'm watching roll back roll back sir to roll again yeah as many times as i say to roll again What a good boy to get a treat for it. See, he did the, this is his primary trick. And it's complicated and it's ridiculous, but it takes up his attention and it gives him something good that he can do for attention. Yeah.

26:22bsandro will there me a cat camera? :P
bsandro *be
Bessandro, there is a cat camera, so I have a filing cabinet behind me. And if he goes up on the filing cabinet, I turn on the cat camera so folks can see him. He doesn't tend to go up there in the winter because it's near a window that has a little bit of a draft. But if you, you know, just hang around the channel a few months, he'll show up. All right.

27:38Pardon all the on-screen typing. Someone I know is watching a... It's the fucking bags.fm coin scam. One of their friends engaged in... All right, so... Let's get this merged. Merged.

28:27Yeah.

29:10It's this one.

30:15Sorry. NoGoodNick_ I just came in here, what is this diff?
It's a little bit urgent that I not let people get scammed. I should be done in about 10 seconds.

...58NoGoodNick_ oh letter transpose, nm
NoGoodNick_ couldn't see it as I was staring at it
yeah very small diff and no good nick for context fed ampere finds every bug in our front end so many bugs okay big sir oh you want to play fetch he brought his favorite toy because he wants to play fetch that's

31:58You know, I was just talking about this to Hunter, so I should add it to the contributing dock. marcoroth_ I'm pretty sure every other JS linter could actually catch this too
Yeah. Let's fetch.

32:23Let's bump up to main.

...37hono5kami looking at you talking about typescript reminds me of how svelte team decided to ditch typescript in favor of plain js + jsdoc. iirc. cmiiw
I am not on main. Let's make a new on main GitHub. Divergent. Oh, good.

...56Nogweii @marcoroth_ problem is that there aren't any decent Javascript linting tool that doesn't pull in a thousand JS dependencies
Marco has a good point. I'm really reluctant to add any JavaScript to the developer or developer dependencies or the build pipeline. hono5kami @Nogweii how about biome? i haven't tried it yet
marcoroth_ @Nogweii oxc is starting to look promising tbh
And that does include TypeScript, but FedEmp is making, you know, kind of the best case one can for that in that he keeps catching bugs and making improvements and his initial stuff caught like some unused arguments and yeah, making a,

33:39making a bunch of contributions and pointing out the value of it is very convincing. Oh, Marco, yeah, oxy. I have an open issue about that if you haven't seen it. pushcx https://github.com/lobsters/lob…
That's not it. Yeah. So if you want an excuse to play with the ox format and the linter, Hey, hey, here's an open feature request. So where was it? Making your change, sharing your work. OK.

34:41Whether the code is a draft or is ready to merge.

36:33NoGoodNick_ for a linter to catch typos in symbols it needs to have the symbol table, right? ideally with types on each variable to know what symbols are available on it. if I'm not mistaken? then might as well just go for TypeScript
marcoroth_ oxc actually has type-aware linting now
You know, no good, Nick. I don't know that you actually, well, a symbol table.

...42marcoroth_ but for variables it should be fine, since you have `const` and they are local to the scope
NoGoodNick_ what if the typo was on the right side of the dot
You know, for small ones, like the one that FedAmp just caught, I think you could get away without having the full symbol table because that kind of, you type out a local variable, doesn't require it. Well, But I could have been referring to a global object. Yeah, so I don't know. Yeah. I mean, you could do a real sloppy version with, like, are any of these identifiers a really low Levenstein distance to any others? And probably catch some, but you'd get false positives. But that's just me trying to be clever and finding a way to produce anything useful without the symbol table. I think you're right that You'd probably need it.

38:02marcoroth_ standardjs assumes you cannot have globals, unless you explicitly declare them in the linter config
Oh, that's interesting.

...09pushcx https://gleam.run/
I've been playing with Gleam, which is if anybody hasn't heard of Gleam. I don't want to totally sideline, but it's coming up on every stream because I've been playing with it. NoGoodNick_ there's no types in Ruby, right? and no "TypeScript" equivalent for Ruby?
And it does compile to JavaScript. So maybe if I really get into it, I'll rewrite the front end of Gleam, but probably not because I think it pulls a fairly large amount of dependencies in and I really liked that we only ship like a couple hundred lines of JS in one file and then the one vendor dependency of Tom select.

...58what's the way to.

40:14marcoroth_ @NoGoodNick_ There is. RBS is the official solution. There are Typecheckers like Sorbet / Steep. And there also have been a few "typed ruby" implementations, but they aren't really popular
NoGoodNick_ ah, good
Nogweii note that type checking as a compile-time thing hasn't really been adopted by the community, sadly
Yeah. Yeah. I've used, it's rebate. Nogweii I would love to see RBS and Sorbet more widely deployed
NoGoodNick_ ah ok
i was working at stripe and it was quite nice for especially for a large code base where i was constantly depending on and integrating with other people's code played with it once for lobsters but it's a lot of work to add it to a rails app and it would catch bugs but i'm not sure it would catch enough bugs to justify the time we put into it

41:12marcoroth_ @Nogweii I think it's slowly coming along. At least for libraries
Yeah, you usually in Ruby, you see the type check as a build step, which actually is an open pull request, a build step here, not one of those. Let's take a look at that because Felipe has probably got that ready to merge.

...56Drop the commit that added bundler link. Yes. Remove bundler link. Yes. Reorder the task steps. Ooh, wow. Oh, Marco, this one might be a fun one for you, actually. Because it's clever Ruby programming. It's probably up your alley. We wanted to add a proper rake task for running the build. So it's just one command instead of three. Because if it's going to expand to four, it really has to be one and not four. That's kind of the point of ridiculousness. marcoroth_ dumb questions, the new Rails `CI` feature is not a fit here?
But Felipe has had to add monkey patches because Breakman wants to exit when it finishes its run, and RSpec works by catching at exit, and that's when it runs its test suite. And so both of those things want to own process exiting. They are not really made for this is a library you call into. I don't know what the new Rails CI feature is. Do you have a link?

43:15marcoroth_ https://rubyonrails.org/2025/9/…
Like I'm gonna merge this, but you know, if we wanna immediately replace it with something that Rails does.

...32marcoroth_ => Local CI
Great. Ah, see, I was, all right, light mode warning, folks. I was thinking... I had missed something. If it's in a beta, I don't feel too bad.

...58marcoroth_ 8.1 final is out already
Ah, well this one... Oh yeah, I see, this is months old. I guess I just missed this release. What are we running? NoGoodNick_ RBS reminds me of header files
We must not be on 8.1, right? Yeah, we're on 8-0 still. I had wondered why Steven Margime was less active on his acid job stuff. For durable execution and rails doing their own implementation of it without even saying the words durable execution that would do it.

45:05Porter provides a unified interface for producing structured events.

...15Nogweii @NoGoodNick_ yeah I'm not enthused by that similarity
marcoroth_ @NoGoodNick_ The trick for that is RBS Inline now, so you can actually have the siguatures right above your methods. Sorbet does it similar, but it's syntax is not the best, imho
OK. The reason that RBS kind of looks like header files is so that you can add types to libraries you don't control. other gems. And because the, and I think they went this way because it was implausible that the entire Ruby community would rally around adding types to Ruby and then, you know, take on the job of doing it in every single library. Nogweii yeah I know the history, along with Matz's reluctance to add a new special syntax for types
So you kind of have to be able to add types to other people's code. And so it has to be able to work in something else. And so then the default became, these header files. Oh, great. Nogweii all good :D
Sorry to explain something you knew.

46:40NoGoodNick_ I didn't know, so the explanation helps
Yeah, this is a little cleaner. Does it...

...55I'm going to kill you to link to docs. Marco, do you know offhand if these steps run in series or parallel? That doesn't change anything for our adoption, but it's... marcoroth_ For Local CI, you don't have to use their signoff stuff, you can just run the bin/ci script in CI.
I guess they must be serial if the first one is set up. marcoroth_ I think they are serial
Nogweii looking at the example I believe it's serial
Yeah. Well, that's a thing to consider. I want to get the build thing merged so I can use it. But maybe after we bump to 8.1, we do this. It is a shame I didn't know that this existed or apparently Felipe know that this existed.

47:56Actually be really big shame if he knew it existed, but you know thought I wasn't open to adopting it because I usually try to. it's mentioned in the read me or the contributing docs of I keep trying to toss our custom code and use built in rails things.

48:26Oh, good, there's another way to pass config options.

...36This is my one. No. No, I have a lot more than one. marcoroth_ oh, 8.2 on main has a another new way LUL
This is a pet peeve I have about Rails that there are so many ways to pass configuration variables and credentials. Oh, A2 has a new new way. Yeah, I haven't even stopped to count, but I can see like five of them active in lobsters right now.

49:32Nogweii 8.1's new way is just a way to fetch the config rather than passing it it appears
Let's get that tab back open. Is there, what do they call it? Because they give everything a name.

...47Is there a doc or a guide I can link to?

...58No. Of course not. We'll go where else is pretty good, right? There was test.

50:21You know, it would be really attractive if this also worked with GitHub actions. Yeah, I don't want to just... This is the opposite of what I want, because I still want to... Maybe we'll wait till 8.3 or something, because the ability to run these things in parallel on a pull request is very valuable, because you don't want the first failure, you want all the failures. And for the release notes talking about how wonderful and powerful multi-core modern machines are, it's running these things in serial.

51:11Yeah, all right. I'm not even going to. That's not the one I want.

...29Yeah. marcoroth_ yeah, I guess the whole point for that was to ditch CI/CD and to just run it locally anyway. And so that it's fast enough that it runs in a few seconds
Yeah, he tends to fix up PR. So let's push new commits. Great.

...51Yeah, Marco, I see the benefit there, but the transition would be easier. And the tool would be more useful if it ran in parallel.

53:12marcoroth_ on the PR introducing it, it says "Got a plan cooking for parallelizing steps as well! "
marcoroth_ https://github.com/rails/rails/…
Oh, good. Yeah, that's why I joked about maybe on 8-3, because it's such an obvious next thing that, you know, it can't be just me considering it.

...41Ah, and here's John, who immediately points out the thing that I like about the feedback.

...54We'll see what comes of it.

54:06dzwdz if you're between PRs do you want to see a lil userscript that i made
I don't know what this one is. Oh. That's great. I must have missed this. Why are these canceled?

...25DZ, I think I saw this go by in the chat room. Yeah, you want to share the link? Because this is going to be a fast merge, I think.

...49dzwdz https://tilde.town/~dzwdz/tmp/l…
KV, MP4, sure. Maybe 3-Way of Ockflag. It is tempting to nitpick and ask for these to be sorted, but I'm not that nitpicky a mood, I guess.

55:09Yeah, there's not going to be any fights. Alright, alright. Well, actually, you know what? Let's just go ahead.

...43Bring that on. dzwdz just open some large thread (e.g. the hotness one) and pop it into the console, you can click the comment preview thingies
And then let's look at your user script. I'm going to reload this so it's in a state. Yeah. And then what are we looking at?

56:06dzwdz oh the code is shit, it's just a PoC
Yeah. Speaking of the hotness one, it is, flaming red highlighted item on my to-do list to spend a couple hours going through all the recent meta threads about vibe coding and give a proper response my days have been a little swamped the last few weeks unfortunately all right so

57:00Does it have to be a big one? Can I just do this one?

...29dzwdz go for it, but it's kinda pointless on small ones
dzwdz try clicking the summaries!
ooh that's interesting what i don't understand what i'm seeing here oh is it was it collapsed because that was a one-liner oh i'm sort of alternating which comment is open

58:11dzwdz comments are arranged chronologically - and they're not collapsed, but backlinks / links to children
That's kind of a fun tick. I could see this being nice on mobile. Let's turn that on. dzwdz if i'm catching up on a long thread i can just scroll down to the first comment i haven't seen
We've talked about how, especially in deeply nested threads on mobile, when you have limited, what on earth is this? Let's change to a popular phone. dzwdz that's the apple watch i think
Why is the most recent phone, like, years old? That's weird. Is there not a... I must have customized this at some point and then it stopped updating to, you know, whatever new things are. No, these are all old. There's nothing newer than the 13, like... The 12 is so old that I have it. All right, so let's say that size. That is nice.

59:31Nogweii @pushcx I think mozilla has simply stopped updating the device database in Firefox, I see the same list
Comments are arranged chronologically. Oh, I see what you've done. marcoroth_ I guess tap targets are a bit small
dzwdz this is a bit more like a traditional forum
so is this so if i click just don't do the thing oh yeah i'm jumping halfway down the page oh that's a really interesting take ah nagwesham i i bet there is a bugzilla for it somewhere Yeah, so DZ, I've played with having a selector at the top of threads to toggle from nested to a flat chronological, and then also from nested to flat sorted by score. Because especially if you come in four years after a conversation was posted, the natural thing to want to do is say, oh, show me the top comments. easeout strikes me as twitter like
dzwdz top comments without context can be a bit useless
Yeah.

01:00:52dzwdz but i suppose this could help
DZ, they can be, yes. That was part of the reason I never shipped it. But this is very nice. graefchen dzdwdz is also on tilde. limesNoted
One of the ways I can tell you're a programmer is programmers love putting boxes around things, where designers would put lines between the comments instead of a box around. Programmers always love a box, and they love a color that goes all the way left to right.

01:01:26Yeah, there's some styling tweaks. easeout lol yes that is a true trend
This is a really interesting presentation.

...41dzwdz the boxes are still better than what it looked in the original screenshot i sent lmao
pushcx https://github.com/lobsters/lob…
DZ, could I talk you into, we have an open issue about, what is it? dzwdz actually lemme find it
Deeply, yeah, here it is.

...56graefchen *tilde.town, stupid graef limesHeck
yeah oh i'm not trying to insult the boxes or anything it's just it's one of two or three things that i see and i instantly clock something as a programmer made that ui could i ask you to please paste a link to your user script here and a couple of screenshots and describe it a little because i think that's a really interesting take especially for small screens because this is what we're doing now which is Politely pretty bad.

01:02:34dzwdz sure! i'll go make it a proper userscript first, then
And yours?

...40So it's funny. Where did I? Oh, I closed the tab, didn't I? No, I got it. I had thought from this top one with two children that what you were doing was collapsing the three children and their threads underneath. DZ, I think, I don't think you have to do that. dzwdz i want to
I think what I would really like is just, because if you, if you read this issue, we're just kind of like throwing ideas out there and kicking stuff around. It's very informal. I'm trying to make sure the barrier to contribution is low. And while we're just throwing ideas around without a really clear plan, you don't have to put any work into it, you know? So I had thought that you were collapsing the subtrees and that if I clicked this, it was expanding the wand.

01:03:38But this is real interesting.

01:04:05dzwdz i was initially planning to use icons that would be more suggestive of what this does
dzwdz instead of this box drawing character thing
dzwdz but uh, that didn't work out
Yeah. This is really creative. Thank you for sharing this. Yeah, that's okay. I mean, that's so much of visual design is you try nine things and only one of them is visually coherent and then you put it in front of actual users and it makes no sense to them because they didn't see the other eight that didn't work. And then you go back to the drawing board and you do it nine times more. But anyways, this is really clever. I like it. Thank you. dzwdz oh it would definitely work if someone actually designed an icon, i was just pasting in unicode characters that seemed similar to what i had in mind
dzwdz shit, even i could design that svg
dzwdz that i do agree with, i don't like this in shorter threads
yeah it's hard to i feel like i've lost the structure of the conversation because it can be useful to see like is there one big thread that the author is of a blog post is replying a bunch in versus are there nine small threads but On the other hand, on mobile, with a small screen, you lose everything anyways. Yeah, it's just that the boxes are visually heavy. Here, actually, wait. I can just come here, inspect. Where's the styling? marcoroth_ do the ├ actually have a reason to be there now? Isn't it full de-nested because it's underneath each other now?
Did you put it on the comment item or the details now you put it on the comment.

01:05:45So if we change this to order dash bottom.

...54See now there's you know and like I would bring in the border a little from the edges, but now there's just a lot less visual noise. dzwdz i think [-] should still work
dzwdz so you can hide super long comments
it's a little clearer that you know each one of these is a comment yeah and i can see why you would reach for icons like it would be nice to have an icon that means parent and an icon that implies child although this this one is good

01:06:31Yeah, I usually see like a little arrow that's turned upwards or something. This is clever. Yeah.

...46And you know, at some point, like you would have like a collapsible four more replies or something in here.

01:07:01dzwdz if you give me a second to find the original screenshot i'll show you some real visual noise
Or honestly, you could collapse it and show like you know there's more than one say there's two applies.

...22that's fun that's a lot of fun let's I didn't even let's grab this link. Which, here, let me save this. Where on earth am I saving this? Home. Log.

...51dzwdz on an unrelated note, can i ask for the schema in a format i can just dump into mariadb? rather than the ruby thing
dzwdz assuming you can do that easily
Just saving a copy of this for the stream archive, because when I see temp in a URL, I can figure you're not going to keep it around forever. But that was a fun one. Yeah.

01:08:18Yes. Yes, there is a way to get Rails to dump that real fast. If you are asking because you are thinking of writing queries, one of the benefits of hash link through getting a Ruby setup is There is a fake data task that will create a whole bunch of stories and comments, which is nice for running queries. But rake.

01:09:11dzwdz you know what, fine, lemme finally set it up
marcoroth_ I feel like situations like these are great reason for devcontainers
Does the dump command? porridge3463 There should be a tag like "I am the author of this link" but for "I believe the content of this link is LLM generate" that's also triggered by flags.
yeah okay so i can change that don't primary and then marco we actually do have a dev container now it was contributed by chale codes so yeah we got that

...47There we go. marcoroth_ yeah, that's what I'm saying that
All right, so let me make a gist over in a personal browser.

01:10:03marcoroth_ why*
Can I just drag this file in? dzwdz are devcontainers the github thing?
Code, lobsters, structure. twitchtd oh hi, I didn't realize you were streaming today :)
Yoink. That was easy enough. Dizzy, no. Dev containers are a VS Code thing. dzwdz even worse :D
pushcx @dzwdz https://gist.github.com/pushcx/…
Hey, Thomas. Yeah. marcoroth_ you can also run them outside of vscode actually
I started late. Started very late, actually.

...42marcoroth_ using the devcontainers CLI, also Zed.dev supports it now
We also have a regular container, not a dev container. I don't know what you'd call that.

...54Oh, neat.

01:11:08porridge. Yeah, that is probably a thing to add. And I guess the problem with it is how much more work it is. This new hassle of veqqio Did you ever add on mods back when?
dzwdz on the topic of userscripts, i feel like that'd work great as an "unofficial" feature
mods have to read every article and make a complicated judgment on whether it looks like its llm output is more work than we can take on right now but in a practical sense is important to the continued viability of the site because slop is just not worth reading

01:12:08marcoroth_ @porridge3463 would the author really tick that checkbox though?
Short version is no. dzwdz Lobsters Enhancement Suite
Long version is too long. Marco, they wouldn't. I think Porridge is suggesting that it be a lag rather than something on the submission form.

01:13:09This one is we added a feature for better story media cards online, and then immediately ran into a hassle with how we overlay a graphic on the image, and we can't predict what especially Mastodon is going to do. And yeah, at that point, it's just like, all right, we're not going to sink 100 hours into this one. Let's just punt.

...58So let's throw this in notes, quick PRS today, and then a couple of giant ones. All right. So I'm red indicator appears over new top comments. Oh, this one. Yeah.

01:14:32If comments is comments one, it is the top level comment. Insert it deeper. OK, that's clear. Great. Yeah. So this one is a bug where the UI is unclear because if you add a comment from a top level comment, This thing is underneath, which is kind of weird. It should be above. I'm OK just YOLOing in a JavaScript fix. Yeah.

01:15:55They didn't say the magic word fix so yeah.

01:16:07let's let's grab this.

...40Yeah, peek at something off stream here real quick.

01:17:51Alrighty. DZ, thanks for pointing out that comment that was over the line. dzwdz oh, also, random graphs are disappearing from /stats
dzwdz is that a known bug
I just pulled it.

01:18:36dzwdz yesterday the two bottom ones were missing, now it's the active users one
Random graphs are disappearing from stats. It is, you know, I saw it, but I thought I fixed it. It's, I'm pretty sure it's 1814. So I've looked at the jobs control dashboard and they're getting Sig killed. Because of memory pressure on the box and it's mostly this active users by month, because this is a big join because it joins users to stories comments and votes, which is okay so it's our four biggest tables. and It just takes more time to run. That's my hunch. dzwdz you'll be happy to hear that i wanted to use this enormous join and join it with another table
Now that I say that, I haven't double-checked it, so I don't want to make stuff up.

01:19:41So I'm pretty sure... That's totally fine. dzwdz the query i wanted to write was checking the tags filtered by active users
You can write as big a query as you want, because what I'll do is... don't prod to my local machine and run it on my local machine. So it can be as complicated as you want. Not only is my beefier than prod, it does not have, you know, hundreds of thousands of scraper hits a day. So what I think is happening with these graphs is they're getting OOM killed because we are so swamped by bots the last couple of weeks that the site is running sluggishly and there's more memory pressure on the box. And so the Puma worker that is running the longest or busiest is the one that's catching the SIG kill.

01:20:41So I don't know if it's worth... You know what? pushcx https://github.com/lobsters/lob…
Actually, if you want to, yeah, you want to file an issue about how these are missing, but I think this is 1814. Like, I think if I ship Anubis, this bug stops happening. 1814 has become the roundup post for all of the aggressive scraping we've been getting the last couple months.

01:21:20Yeah. Oh, you know, the other bug there, maybe that's something I should look at now. is failed jobs are not going to the bug tracker.

...39Yeah, so I'm looking at the mission control for our background jobs. Mission control is the nice Rails background for, yeah. All right, light mode warning. dzwdz does the faker thing also generate plausible activity for the active users stats?
I'm going to just bring this up. marcoroth_ isn't mission control just the web UI?
See, all of these are catching sig kills. So here's all of the graphs that you were pointing out, DZ. DZ, no, it's not plausible because it posts all of the comments within a few seconds of each other rather than trying to spread them out over some time. If you wanted to add that, I would love that kind of improvement because we improve the fake data dzwdz been there
as we need it rather than preemptively because there's you know an infinite amount of trying to make synthetic data look plausible we could get into but i understand if that's bigger than something you want to take on so marco yes this is the web ui but what i'm saying is the bug is these failed jobs are not showing up in our bug tracker which i didn't look at off screen to know isn't going to throw up pii yeah yeah so here's the bug tracker in the last seven days it has only got one error report and in the last seven days we have had five of these jobs fail so whatever this wiring is that's supposed to be sending these failures to the queue is not working i swear it i've tried to configure it once or twice and i thought i had it working but let's take a second so we got telebugs that set stuff up i'm just around yeah

01:23:50marcoroth_ actually, this might be another good use for the new Structured Events in Rails 8.1
user enrichment oh it's mail errors that's what i'm thinking of is we had a we fixed it so that mail errors would show up in telebugs and i think that's still working but we haven't had any for a minute so no okay this is a new feature i guess

01:24:23marcoroth_ they have events for all sorts of things, and I assume they also have active_job.* events
marco could you say some more because i only just skimmed past that in the thing a minute ago and didn't read it closely because it is hard to actually read and comprehend things while i'm also keeping up patter for streaming so i tend to focus

01:25:07I don't know if these will even get logged because the whole process is getting gunned.

...40Do the individual jobs get run in their own process? They must. Or there would be nothing in the queue or nothing in the mission control because, you know, when something catches an OOM kill, it is dead dead. All right, let's see if we can find a human talking about this. So what's our API around it? Yeah, we don't wrap capture exception. But we could. We could retry on sig kill. No, that's real dangerous. I don't want to do that. I want to be able to actually stop jobs if I really need a job to stop. So let's find some actual documentation.

01:27:10AtreidesNuts oh that doc lookup is sic
AtreidesNuts sick*
AtreidesNuts ironic typo
I don't love the rando Google summary of overriding perform.

...27pushcx https://zealdocs.org/
It's an app called ZealDocs. There you go.

...52Is there not a, yeah, there's a basics.

01:28:09There isn't a callback for my job failed.

...28And it does ship up the guides too, which is nice. Because I often find myself kind of ping-ponging between the guides and the actual API docs. Oh look, error reporting on jobs. Rescue from. AtreidesNuts yeah that totally beats 13 browser tabs and sifting AI gen snippets
See, I was expecting something like that. All right. Rails error report, and then re-raise. Great. That's just what I want. Yeah. And you know, this little AI snippet was OK for pointing made application job, but this is nonsense. I don't have to override. And it just didn't feel Rails-y, because Rails is so big on having callbacks for things. So let's say rescue from exception.

01:29:34What are we mad about? There we go. Standard was confused for a second. Hello, bugs. What did I call it? Report. Context message. Was there not a? Capture.

01:30:06Because if I re-erase, I'll also get the failure and mission control, which I want. All right. So then here, let's make a capture.

...22Quargs, such a good word.

...37So according to Gemini, it's sentry dot capture exception with arms, but I would like to actually see that in the sentry docs. I do wish Zeo packaged up marcoroth_ typo in `Sentry`
gems so that I could search their docs as well, that would be very nice. Type one sentry? Yeah, thank you.

01:31:27This is probably the ad blocker. No? Nogweii ublock blocks sentry.io
Hmm. Yeah. Sentry didn't use a different domain for their... Even when toggled off, apparently. Didn't use a different domain for their docs than their reporting, so of course they get blocked. dzwdz lmao
marcoroth_ my pi hole usually blocks trackers too lol
Okay, so capture exception. All right. So rubydoc.info, which scrapes all the gem docs, claims that Gemini didn't hallucinate that one. So thanks. Where am I at? Oh, I don't need that structure. Oh, you know what I need to do? Nogweii from sentry docs: begin 1 / 0 rescue ZeroDivisionError => exception Sentry.capture_exception(exception) end
When it's fetch main and rebase on top of it, we should get a divergence.

01:32:38Rebase.

...46Rebase two commits. Oh, I didn't push up the other thing. Okay. Thanks, Nagwe.

01:33:09So this is report job failures to telebugs. So I know they happen. Let's bump main and push it up. marcoroth_ typo still present at telebugs.rb:39
See, I finally made aliases for those things, as opposed to Control-R replacing everything. Let's get that deployed. So I've actually merged a whole bunch of stuff.

...44Ah, shit. Thanks, Marco. Let's force push domain. I bet my alias will throw an error now. Yeah. And then JJ up? No. Yeah. What is it?

01:34:25marcoroth_ 🙈
That's not plausible.

...34What do I want? I want J.J. Squash. I know it's immutable, but I'm a bad person. What is it? Ignore immutable?

01:35:01Take that, GitHub. I didn't run my new Rails task. Why did it say there was an exception? Why am I getting a million pieces? Did I merge a migration? You have 156 pending migrations. I don't believe you. I wouldn't believe five. What weird state am I in?

01:36:10marcoroth_ maybe earlier when you dumped the schema?
Yeah, I'm now coming off of the SQLite. Yeah, but I removed the file, and all I did was dump the one thing. I didn't actually update the config file for what format to dump in. And I've been working on the SQLite branch, but I'm not downstream of that. Oh, I am. It's config database YAML. Speaking of how many ways there are to config things, this file is gitignored. Because, you know, it used to have odd credentials in it. So now it's just here to foot gun me when I switch between branches.

01:37:21chamlis_ mockturtle-gleam 👀
That's concerning that heinous inline partial fired twice. Why did my initializer run twice? Okay, so yeah. Ah, chamless. I wondered if anybody would be eagle-eyed. Yes. The space build. So I mentioned earlier I've been playing with gleam. chamlis_ I was going to ask if you had the exception trace but sounds like that might not matter for long
mock turtle has broken in a weird way so for anybody who doesn't know mock turtle is our irc bot that sits in the chat room and it has a bunch of features and the core one of announcing new stories on the site has been broken for a couple of days for no effing reason i can determine it throws an exception but it is not because it's async it doesn't even give a line number chamlis_ too late 40% of the economy is riding on this
oh no no no oh don't say for long that was just me playing around because i am learning gleam so that was me just smashing into very basic like syntax errors this is you know i've written hello world in gleam that's literally it until now so that is me playing around with well 40% of the economy. The economy is in shambles, I thought. Nogweii shambles, yet still precarious :D
So I thought, hey, you know, what's good at long running background processes? The Erlang VM.

01:39:08So Oh, camel is also it has a reputation for being excellent to write parsers in. So I was kind of playing around with that.

...25I have Yeah, so on that branch, I have written literally one function, a smart constructor for channel names that knows the rules for valid channel names. They have a maximum length of 50 characters. There are three characters they're not allowed to contain. And what's the other part of it? Oh, everybody starts a channel name with a, dzwdz they have some special meanings iirc
dzwdz one of them means that the channel is local to a particular server i think?
a hash but they're apparently like five other sigils you can use for no i have never seen those in 30 years of irc 30. one no yeah more like 35 years of irc and now i have a smart constructor for my type That's the functional programmer thing to do is you don't export your type constructors because then you don't have this awful Ruby pain of having to ask channel.valid. Oh, neat. I had no idea. Not surprising that someone else who has gray hair immediately knows the answer to that puzzle. Not to call you out, DZ. It's just such a specific thing to know about an old school format. So yeah, that's an experiment. I would rather get Mock Turtle working again. Let's look at that, actually, because we're going to.

01:41:35chamlis_ have you tried curling from that server? on the chance it ended up in a scraper blocklist
Yeah.

...50dzwdz "grey hair" you've been on irc longer than i've been alive
That's a really obvious thing to test. And no, I have not tested that. Let's test it.

01:42:13dzwdz by a substantial amount
Nogweii @dzwdz maybe you're just an old soul :P
By a substantial amount, okay. I apologize for attributing too much maturity to you. I won't do it again. It's in a, yeah, it's in a funny directory. And then I don't actually need to be in here because I could just get it out of internal control.

...42yeah so here's here's our exception you know if i had dropped the prod ip of mock turtle that would cause it and i have been blocking so many bots lately but i wouldn't intentionally block our own god that would be ridiculous so let's

01:43:12No, curl works. Okay, I did not block our own bot. chamlis_ phew
Unless... So we block bots at a couple of different levels. And I'm kind of thinking through them. So iptables is the one that could produce an error like this because it just drops packets. And so I could see getting a vague timeout out of something, right? But... dzwdz i'll also mention i'm having some issues importing structure.sql into mariadb? probably PEBKAC though
It's the user agent stuff is in caddy. So it would at least, it serves a like 400 or a 401 with a silly message so that people can find it in the log. DC, I've got nothing for you on that one. marcoroth_ @dzwdz I wonder if it was the sqlite dump?
dzwdz yeah i don't expect anything
I think that might be the first time I have dumped our structure as SQL.

01:44:15Oh, yeah, Marco, no, you're... Yeah, no, Marco's got it. Let's dump that again. That command is back in the history of some other terminal. So I get to do it again. No, it's here, okay.

...45dzwdz i don't have the brainpower for this
All right, so let me pull up the gist. View my gists. This one. Now, DZ, Marco got it. dzwdz i know, but i also should've noticed
I accidentally gave you a dump from SQLite because of what it just carried over between branches. And then, so let me grab that and then code lobsters, DB structure. Let's bring that up. And this one is Maria DB. pushcx https://gist.github.com/pushcx/…
Yeah, this is definitely there. All right. dzwdz thanks
So there you go. Strikes me as a very easy thing to not catch. Marco, thank you for figuring it out. dzwdz but also i think i'm off, i gotta eat something
dzwdz bye
I still can't add more VIPs, damn it. That was a great bug catch. Later, Deezy. Thanks again for dropping by with your user script. That's a lot of fun.

01:46:12All right, so back to this. I'm going to throw this up on the screen, and I'm going to step away from the keyboard for a second. chamlis_ I tried setting these headers from my curl earlier but that still worked for me https://github.com/rbren/rss-pa…
So Chambliss, if you want to do your usual thing of being incredibly insightful and figuring bugs out, this is all the debugging info we have. I'll be back in 90 seconds.

01:48:28I tried setting headers and that still worked for you. Oh, yeah. Well, you know, I still think your idea about is this server getting blocked is not a terrible idea. Let's double check. So if I want to do this, I want to know the prod IP, but let's grab Let's grab this exception so I don't have to log in again. Put it in Scratch.

01:49:07That is not what I want to paste. There we go. And then disconnect. How convenient. I got MockTurtle's IP.

...43Nogweii I guess you're back :)
Oh, yes. Thank you.

...51That wasn't very visible on the green, was it? Can I? Are visible on the background. I can make it that obnoxious lime color, right? Yeah, there we go. All right, so now it's still not visible. Where's my, aha, there we go. See, I am so good at OBS that I can even show a color on the fourth try.

01:50:28which is tailed action log. Grab the last 10,000 lines, because it'll be in there.

...43Are we really getting hammered? OK, so this is, oh, but this one might get handled by Caddy, because we do render newest.rss. So yeah, I'm looking in the wrong place. I should be looking in. Caddy.

01:51:05Really?

...22Nogweii LUL
Why did I think...

...30That was the search bot. What the shit? marcoroth_ LUL
chamlis_ did the curl hit it over ipv6 or something
Nogweii probably
My lack of taking notes is going to bite me because I didn't grab representative lines for all of these. Let's fix that. So, Chanless catches another bug, which is, you know, news at 11 levels of surprising.

01:52:3226 yeah i remember this one not like this line but this block of them was we were really struggling

01:53:07And why does the curl? Oh, I bet the curl worked because these all have multiple IPs and it must have gone across like the internal IP or something. chamlis_ quickly bet if mockturtle will post every story in the last few days
Nogweii `curl -v`?
Hmm. Okay, mock turtle is immediately working. chamlis_ ahh nice
chamlist no it has a reasonable amount of coding so that well it might not actually it might be about to flood itself off the network that's happened before although actually it grabs newest.rss which only has what 10 stories in it 20 i think it's 25 to match the page

01:54:18Yeah, 25. The extra one there is the feed has a title tag too. Okay. So, you know, what did we learn? If you want your bot to work, don't ban it from your own site.

...40I thought we knew that. God, is that remix still online? There's a deep cut.

...58Yeah, no, this is forever ago. There was a lawyer named Lawrence Lessig who did a bunch of work around network neutrality and related issues. Nogweii https://archive.org/details/Gar…
And somebody made like a dance track that was remixing his stuff because, you know, that was automatically interesting and funny back in the day. That's probably it. Let's see if the audio wants to behave. So if I turn on this...

01:55:58Nogweii can't hear it from the stream though :(
marcoroth_ I think it was just quiet
yep that's it that's it i have this exact file somewhere you know i have like a lot of mp3s march 2001 yeah yeah that's going back a minute i thought we knew that oh no all right well i don't know why we can't hear that and the last thing i want to do is debug linux audio issues on stream so if you want to know why I thought we knew that, is a cute meme to someone who was online forever ago. All right, let's try it one more time.

01:56:46chamlis_ yeah that audio file is older than me
What if I crank this up a little?

...56twitchtd for the sqlite work, were you able to narrow down the issue? I don't have the exception during the sqlite load. I was thinking that maybe there's an invalid utf8 byte sequence in mariadb and that's getting somehow propagated into sqlite?
Yeah, I don't see. marcoroth_ maybe lol
meter bouncing for it i think if you could have heard that at all marco it might have been playing out of my speakers and bouncing back off of me into the microphone because i am not in any kind of properly insulated sound booth thomas anyways nagwe good good find on that link by the way thank you that's It's a ridiculous deep cut to have turned up. The other one is the Ted Stevens series of tubes mashup for ridiculous mashups from the era. And anyways, yeah. So Chamlus, you mentioned that the audio file is older than you. We have a bunch of, like, jokey meme references in the code base so like speaking of things that are older than you this meme reference like wow a blue car i added this comment linking to the simpsons clip because that came out in like god what like 94 or something somewhere way back there and i i have kind of wondered at what point Do the memes become so old that we should recycle them?

01:58:35And then some of them, like, you know, you have replied too greedily and too deep. Come on, Lord of the Rings is not falling out of favor anytime. But, like, a random sentence from a Simpsons episode, like, yes, oh god, you know, I heard this one hundred times a day back when it was new because you know we didn't have as much media you'd get like one funny simpsons episode that would have to tide you over for a month these did not post their user script so thomas I have not taken a look at the pull request anymore because I have spent my time fighting bad bots. But since you are here, let's take a look.

01:59:47marcoroth_ bad bots and your own bots Kappa
And just to set expectations, I usually Oh yeah, I'll fight anybody. I usually stream for about three hours and I started late so I'm gonna run I guess an hour later than usual because we're right about two hours now so yeah I'll run for about another hour so my best guess on why this refused to load was that confidence order path value

02:00:31And I know Unicode does stuff with 255 bytes, so I could see that making it mad.

...53Let me take a shot at something then. Let's get back into a working state.

02:01:16Where was it? I had started a commit on top of yours, hadn't I?

...31This is the old one, I think. Yes. So here's this. There we go. So let's rebase. that commit after your new one. Because even if it's just linting, I want to be in the right place. And now I'm in the right place. Well, I have to go over there. So let's go edit my own commit and take a look at it so I know where the heck I am.

02:02:20Windows are all out of order.

02:03:06Oh.

...49chamlis_ speaking as chief youth, I'd rather learn about some older memes than roll my eyes at lobsters saying "67" or whatever
We could add a meme so that somebody who posts the 67th comments to a thread gets an extra notify, like a little something in their flash message besides just comment posted. Or to the search engine, if there are exactly 67 results.

02:04:24highlight comments that have exactly a score of 67? Or a karma? I don't actually know the meme. I know the gist of the meme is it's just automatically considered a funny number, like 69 or 42, but I don't know what you're supposed to do with that one. Maybe you do the... chamlis_ that one is past my time too
The Fortnite flossing dance or something else youthful.

02:05:04See, even my memes are too old when I'm trying to talk about the youths. chamlis_ you can't handle my memes, they're too old for you?
What did I call it? Let's bring that back.

...28There's too political, but that was a good setup for a joke.

...57graefchen All your base are belong to us. limesGiggle (that meme is apparently younger than me and that surprises me limesO )
Oh, yeah. I don't think we might actually have an all your base meme. Well, it's shown up in dump.yaml, of course. Not in the code base, though.

02:06:18What's the name of the rake task?

...26Migrate.

...43Is there, yeah, right up here. I could just read, I'd have to tweak the database YAML.

02:07:20I don't know if it's going to be a string or an integer. Or a symbol, rather.

...49So. Put everybody back. What's it called? Dump? Just dump?

02:08:10Let's delete that just so I don't have to fight it.

...36I don't know why it's not showing up. Oh, because it doesn't have a description.

...53Hmm.

02:09:12All righty.

...35twitchtd how about them apples
Yeah, I don't remember how long this took on the last stream. A couple of minutes?

...47And I want to say it got to two or three gigs.

...55So let's take it along. Pretty good clip, but yeah, not much to say. You know what I should do, though, is I should pull this up. Because there might be enough of it dumped that I can see that the confidence order path thing worked. Oh, vim does not like giant files. I must have broken my config.

02:11:14It doesn't look like it's made it to the comments table yet.

...29Let's get rid of that page so I can actually see progress.

...52Now it's made it to the notifications.

02:12:13Thrilling work of software maintenance. Watching number go up.

...29I guess I could put this back.

...38Not really anything to do.

...54AtreidesNuts what was that calendar date message on the other window?
Oh, it's D-Date. It's... There is an old Unix utility called ddate. AtreidesNuts oh interesting i'll go look that up
It is the date in the Discordian calendar, which I believe has a Wikipedia page. Let me keep this up so I can actually see it ticking. pushcx https://en.wikipedia.org/wiki/D…
Yep. Yeah. I'm just... Put it in my Bash profile or my Bash RC, whichever of those is for interactive shells. I never remember. When did I do that? Maybe 1995. olexsmir no one knows difference between bash_profile and bashrc
I want to say this came out in 63, but I learned of it from the Illuminati trilogy. I just like it. AtreidesNuts idk looks fun. feels like playing nethack when youre hopping in a terminal session
Got to have a little whimsy, right?

02:14:10I have NetHack uninstalled after it whacked some of my college grades.

...22Didn't fail anything, but there were a couple of classes I could have done better on one semester. graefchen I have written a small program that does that with the 24 japanese solar terms. limesGiggle
All right, so this is done. Nogweii speaking of whimsy, I made a website to make every day a celebration: https://everyday.holiday/ (self-promo)
And my grep must be wrong because I don't see confidence order path. Well, let's just try and load it and see what happens.

...52Everyday holiday? Come here. Oh, that's fun. Are these like pulled out of Wikipedia? Yeah. How do you pick when there's multiple?

02:15:21Nogweii then it shows multiple :)
You show multiple. Nice. That's cute. I like that. And you have a bot to announce them?

...37Nogweii the bot is broken lol
Excellent. graefchen That is so smort. Gonna bookmark it. limesGiggle
Oh, no. Did you maybe ban the bot by IP address from part of your service?

02:16:02Nogweii I'll have to check :D
Nogweii @graefchen thanks!
Well, we have a different bug, Thomas.

...11Because I attempted to load before, and I have the file sitting around on desk. So it just tried to load into a partial table, I bet.

...30So let's blow that away. Yeah, actually, there's a task for it. Rails db drop primary. Rails db create primary. twitchtd rails db:drop:primary db:create:primary db:schema:load:primary
Schema load primary. And now let's load db. Ah, yep, I just typed it a little faster than you. I have Control-R to thank, though.

02:17:17It's funny that it failed immediately on votes. I guess it really is not loading things in alphabetical order like I sort of naively assumed.

...38When I dump and load the MariaDB database, it does do tables in alphabetical order. So I've gotten kind of used to that.

...53All right, well, we're zipping along.

02:18:37twitchtd btw happy fat thursday :)
There's a, oh, you know what for memes? Oh, I was thinking about that. Let's see how fast we can code it. The real one is, let's find the Wikipedia page for this.

02:19:08I don't think it was until the 17th.

...16Oh, I'm confused because it's Fat Thursday versus Fat Tuesday. So while that's running... wrong clipboard there still wrong clipboard come here so the title of this wikipedia page is wrong but next tuesday is gonna be punchki day pushcx https://en.wikipedia.org/wiki/S…
I don't know what all this Shrove stuff is. I do, obviously. Where's Chicago? Because that's the important part of every Wikipedia article. Chicago famously has a very large Polish immigrant community. And we celebrate Punschki Day. So that'll be on Tuesday. And actually over on my instant messenger, We were just talking about putting in our order for punchki day. Cause you gotta, you can't just like wing this and drop in and pick up punchki. So punchki are, they're like a donut. I mean, you know, it's not a donut, but it's, it's a cousin to a donut, let's say. pushcx https://en.wikipedia.org/wiki/P…
And they're excellent. rotated8 Mardi Gras
So this Shrove Tuesday is also kind of the progenitor of that big, what's the big one down in New Orleans? Mardi Gras? They do a big party in front of, yeah, you got it there. graefchen We here in Cologne, Germnay, celebrate a carnival from today until next tuesday, it do be interesting. limesSit
do a big party right in front of lent which is supposed to be a very sober holiday kind of about you should ask wikipedia what's it what it's about i'll probably commit heresy if i try and summarize it but the idea of mardi gras and the more important holiday punchki day is that right before this very sober restrained holiday You have to use up all of your, like, butter and all of your good tasty things. Ah, yeah. olexsmir yay Easter Europe traditions
So... I wonder if it's in here. marcoroth_ huh yeah, I was just reading that and was wondering how does it relate to carnival?
Most famous is the... Oh, here you go, Grafchen. Is your holiday known as one of these?

02:22:33graefchen Yeah. limesD
marcoroth_ I would say Fastnachtsdienstag is something completely different
yeah so marco the where is your color why are you naked sir oh cat well it's a good thing that this is not a cam on stream because you would have just gotten me banned for nudity on the channel sir the cat has lost his collar it's one of those collars that pops off and it's a couple years old so the it pops off very easily now he must have gone under the couch So anyways, I've been looking forward to Tuesday cause I'm not normally a big donut guy and, I do celebrate punch key day. So I was thinking, Ooh, it looks like we had some kind of failure or no, I bet it completed. And if it completed, it wouldn't need its wall. Well, that counts as completed. This is the same comment. No, this is the same story text. So all right. So let's throw this in the topics.

02:23:57Where's did you seriously only gruetzhaxe just learned my birthday will be pasta AND opera day. looking forward.
Copy the part of this I can see.

02:24:10graefchen It is the last day before lent, so it fits okayish limesSit
And because I don't want to figure out how to type the BRDC, pasta and upper day. Oh, yeah. We got to check. Hold on. We got to check Nogways. Everyday holiday. So let's go ahead to next Tuesday. Random acts. Oh, no. Punchki day. I'm gonna have to PR this thing. chamlis_ pushcx says no random acts of kindness
Nogweii @pushcx holidays.sql
Do you have a list of holidays in here or? Hmm.

02:25:02espartapalma add an entry in an SQL file
Yeah, no. To hell with random acts of kindness. I want donuts.

...28Nogweii I'll have to remember to add it :P
Too bad I didn't say I want poochki. You know what? It can be funnier. It could be an Easter egg that I said the wrong thing. twitchtd I wonder if it's an invalid bytesequence in the story_text
So it looks like Thomas isn't active, but this is actually where we saw the same failure before is this one story text was scraped from a site that was talking about binary debugging and a random sequence of bytes that's in the middle of this story text is not getting wrapped somehow. I mean, yeah, there are so many layers it could be, right? Like, is it wrong in MariaDB? Is it something about the collation on the table? Is it something about the dump to YAML? Nogweii does the mariadb use utf8mb4?
Is it something about the load? Is it something about the table in SQLite? This is one of the fun things about character issues is they give you a lot of things to check.

02:26:56Oh, the whole side thing about punch key day is if I was going to be waiting on the database for a long time, I would have been tempted to try and add an Easter egg for that Chicago holiday, because I especially like putting Chicago Easter eggs in the code base.

02:27:34Is it not show table?

...45twitchtd @Nogweii yes https://github.com/lobsters/lob…
It is show table.

...54Oh, yeah. You know what? If we really want to be. I just dumped the sequel, right? And I accidentally dumped the SQLite version so we can compare them. So yes, MariaDB says UTF-8 MB4. olexsmir github gists for db dumps, smart...
And SQLite says nothing explicit that I can see. twitchtd I have to go but I'll watch the capture later today sorry can't stay
olexsmir sad..
Alexamir, this is just the definition of the table. pushcx https://gist.github.com/pushcx/…
Someone who was on stream an hour or so ago asked... Here, I'll throw you the link if you want. So...

02:29:13olexsmir is it one generated by rails?
So it kind of think of where this could be coming in. Nogweii is sqlite's TEXT type utf8 safe by default?
Alexa mirror yet both of those are generated by rails. And the the in progress PR from Thomas twitch TD here is.

...40one to migrate us from MariaDB to SQLite. So that's why we're running into some kind of encoding issue between the two. chamlis_ they say they'll accept as much as they can: https://sqlite.org/invalidutf.h…
And Nagwe, I couldn't tell you. Let's find out. Did I reseal? I got docs. Oh, yeah. garbage in garbage out is exactly what we want.

02:30:17Nogweii from sqlite docs: TEXT. The value is a text string, stored using the database encoding (UTF-8, UTF-16BE or UTF-16LE).
Oh, no. No, it might give us something different. If you put garbage in, then you may not complain if you get different garbage out. But I have garbage. Invalid get served pair, invalid multiply sequences, which is what we think we're seeing. They considered that. That seems dangerous. Yeah, null characters and strings.

02:31:15Generally speaking, if you insert invalid, you will get the exact same byte sequence out. Okay.

...32You know, I should... Where was TD's comments? So let's... The thing we haven't done is actually see the exception. We see the traceback, but there are so many lines of exception that we can't see it. So let's try loaddb head-n 30. That'll take a couple of minutes to run, but it might give us even more debugging info, because this sounds like the expected behavior is that SQLite would take the invalid data. And you know, there's a lot of stuff here about we'll try and roughly preserve it and you might get back nonsense. But it doesn't say we're going to throw an exception crash. Yeah, not crash. It's not its own process. You all know what I mean. So the behavior we're seeing doesn't quite match the behavior we're seeing out of the database. And it could be that what's not specified here is it's indicating some kind of error condition back. And this is active record going, hey, you are so likely to have just destroyed data that I'm gonna throw an exception and force you to deal with it.

02:33:18I'm going to pick up the mic here. You might hear a little rustling, but you might also hear the cat purr.

...34Nogweii purr!
AtreidesNuts aw hell yea
olexsmir hi cat
graefchen I sounded a purr limesYay
It's not a particularly loud purring cat, but he came up on my lap and he settled down and he wants to be a good boy. No, no, he just bit me, sir. Why are you so bad? How can I tell the difference between happy purrs and agitated purrs? Why do you use purrs for both? This is a thing cats do. Sometimes they purr when they're overstimulated.

02:34:02graefchen spounded, what, I ment heard. limesGiggle
Yeah, well, it was a lot sweeter and a lot cuter until he put his teeth on me. And it was just one of his little warning ones, but if he does that, he's working himself up. why are you such a sinner sir don't you know it's shrove day coming up you're not going to get any punch key before lent if you don't be good yeah yes sir all right all right good boy so me clicking my tongue i'm glad you could hear the purr that's good me clicking my tongue is that is the easiest trick to teach a cat if they are sitting on your lap and you have to get up like number one you're a monster you just don't do that right like you live on the couch now you die on the couch but every once in a while you have to get up and so i click my tongue twice and then i start moving to get up and even the dumbest cats i've had have learned that trick because hear an unusual noise and you wind up on the floor two seconds later is memorable enough that they learn it god why are you so high strung today

02:35:42I wonder, yeah, we'll see the exception.

02:36:07So I could change this body to be a blob, and we'll see if it goes away, right?

...26Not that running these over and over makes for a thrilling stream. Where are we? About 20 minutes from the end?

...53marcoroth_ are you letting ActiveRecord read from maria and insert into sqlite?
Marco Roth, it's yes. If you look at the PR, it's the migrate task that he added. Come here. So here's the whole thing on screen.

02:37:29you failed again and you didn't print anything different because you're printing to standard error instead of standard out well it only took 276 seconds so we can do it again two greater than we have one

02:38:00So that's 276. That's what, four minutes? Four and a half, change.

...16The other thing to try would be to try making this one blob.

...25Honestly, if it's this one record,

...35It's binary data of... We looked up the blog post on the last stream. It's literally someone talking about running a hex dump on things. So if I deleted that out of the middle of the one record, that would be fine. I would be willing to accept that loss on prod. Because it's not like you could search for that and get anything. So... If it is literally one out of our, what, 120,000 stories? It's not so bad. I'm willing to be pragmatic on that one. Oh, this, I should just delete that. We don't know what order it did that in. Yeah, hang on. So let me... If I look at dump.yaml, OK, so we've got a All right, so. Yeah, we're not getting as much feedback as I had kind of hoped. We're getting back story text before we get to comment. So I don't know if Confidence Order Path did anything or not.

02:40:42Yeah, we're not getting much feedback because story text is, well, I guess story title has arbitrary user text input, but comment is the big one.

02:41:28marcoroth_ could it be that Ruby doesn't handle it properly?
marcoroth_ and screws up the encoding somehow?
Yes, it could entirely be that Ruby doesn't handle it properly. So we have this much traceback. We don't know what the actual top thing is. Yeah, we don't know because there are several transformations happening here. We don't know what layer it's happening at. And honestly, the fact that it's happening at all is kind of concerning because My one fear with this process has explicitly been that we take on, we have some kind of mild incompatibility in character encoding and we don't see that we have like turned all the curly quotes into something or, you know, five comments include say the Beardy C that's in Punchki and then We don't know it until we've corrupted it. I don't know how to catch those things. Here we go. So here's our exception. So let's grab the top of this. Active record statement invalid.

02:42:52marcoroth_ but it's always failing on the same row/record, no? So it might be worth to add a breakpoint before that one, if possible
Unrecognized token.

02:43:08It's not an individual record or row. They're in batches. So when you look at this, there are not 4,500 records in the vote database. There are just over a million. There are not 20 records in the user database. There are like, I don't know, something in the tens of thousands. There's not one tag. There are 140. So they're in chunks of some kind. I don't know that a breakpoint gets us anything.

02:44:34marcoroth_ and that batching behavior comes from where?
chamlis_ https://github.com/rails/rails/… looks potentially relevant
Is there a better way for us to dump from MariaDB and load into SQLite?

...59What if we didn't dump? What if we didn't attempt to store in YAML, an intermediate format, or SQL, an intermediate format? Rails talks to multiple databases. If instead of toggling database.yaml back and forth between the two, I named one MariaDB and one primary, We could access both from one process. That saves us encoding to a file and reading back.

02:45:50What do you have, Chamlus? Try to insert something that contains a null byte. We might have a null byte. Unrecognized token error. Was that the exception we had when we saw Yes, we saw unrecognized token. I guess all zeros counts as a token. Try to reopen 947. Also closed.

02:46:39Why did you close that lighthouse? 2013. Oof. Deep cut.

02:47:08Fix adds an explicit argument error. I guess we're getting one, but that's not... I mean, if I had a blob, why wouldn't I be able to put in a null byte, right? It's not valid text.

...37chamlis_ oh my bad, was that latest run with a blob column
but won't fix that.

...44Sir, do you want to play fetch?

02:48:08No, I didn't attempt to what the blob column yet.

...17And honestly, the way folks are describing this.

...30You can still send them in blobs if you're

...42Your quote string is used for blobs. This is exactly what I was thinking of. It's a text first blob. Does notice accept. Add the check and quote string. All the interesting discussion is hidden behind load more. So for blobs, we call it quoted binary. We use the default. So we could... Sir, do you need to play fetch? Binary blob, SQLite correctly inserts. Okay, so this implies that if I recreated that column as blob, we would be fine.

02:49:39But then this implies it doesn't. No way to know but test, especially, well, these are only six months old. Eight months old.

02:50:03Hmm. Oh, hey. So I know this username. espartapalma terlebug
Kirilo is the developer of telebugs that we use for bug tracking. I bet his bug is about storing arbitrary data on exceptions. I don't know why he put it on. chamlis_ it is the everything app, after all
x maybe he's storing images generated by grok it's hard to respect people posting on this site anymore

02:51:21chamlis_ does rails stop people submitting text with null bytes in it? ie is this only a concern for values we're externally fetching like the story text?
So that back went at SQL light, change this to a blob, recreate the database.

...49chamos that's a good question i i cannot guess the answer to that i mean at some point someone has to have pasted a null byte into a comment right there's no way we get 600 000 comments from developers and nobody does that at some point even you know just pasting the output of something else let alone experimenting on prod because people occasionally you know poke prod with a stick to see if it does something funny which because i mentioned please don't poke broad with a stick to see if it does something funny olexsmir but it's interesting
If you think you're doing something that might crash prod or cause a security issue, please test locally or please just tell me. veqqio Wait you want me to turn off the bot nets? I've been writing a paper to see...
It is at least a little bit rude to test if you can crash prod. And when I see people who are doing basically malicious things like running vulnerability scanner on prods, oh god in fact if you started a master's at the university of chicago and you decided to experiment on us if anybody doesn't know that one it's mentioned on the about page but university of chicago did a experiment on me chamlis_ wait it was actually your local university? that's pretty funny
Because apparently at the University of Chicago, it is okay to put people in the treatment arm of an experiment without them consulting to be experimented on, you know, just like how you can walk down the street and randomly give drugs to people. And just to see what they do to people like the University of Chicago considers that ethical behavior. And yes, they were actually local. There are like seven universities in Chicago, so I guess we had good odds. University is not my local one, though. They're Southsiders.

02:54:19There is not a... The only the only bit of like north side versus south side that I will partake in is the. The cross town classic. As long as we're waiting on the database to crash does this. Have a wikipedia page. Okay, I guess here. So Chicago has two baseball teams, one on the north side that's good and one on the south side that's evil. And... Once a year, they play... I think it's a series of games, right? Hi, baby. Don't have big feelings, sir. Be good. Because, I mean, you know... Your local sports team is going to have a rivalry with whatever team is geographically closest. That's going to be the most evil, perfidious team to ever bribe a ref into defeating local sports team. Sir, please don't sleep on the mouse and get big son Felix. I don't know if Chicago is the only city with two baseball teams. But we definitely have to. And so, of course, the biggest rivalry has to be that perfidious team that plays, you know, what is it, 10 miles away? Ah, yeah, there's a little map even.

02:56:29The rivalry turned physical on May 22,006. Yeah, that... I remember that one.

...45chamlis_ in UK football it's a "derby" when two teams from the same place play
Anyways. So that's got a couple of minutes to run. espartapalma LA Houston Philadelphia also has 2 MLB teams
A derby won two teams. Ah. I didn't know that... There were places that had two sports teams, or two soccer teams. Not to open the can of worms about football versus soccer, I'm just being an American talking about it. It occurred to me, I should pull up this, LA, Houston, and Philly also have two

02:57:30Big baby.

...36espartapalma Mexico City used to have 4 professional futbol teams
You're a good boy, sir. God, you're in such a mood today. All right, well, I would like this run to crash or pass the size that we were at before I finished the stream, but I am coming up on the end of the stream, so if I'm allowed to touch the mouse without getting the champ, I'm going to turn on the last call banner. Wow, four teams. There's some intercity rivalry.

02:58:20Intercity rivalry. Yeah, wrong prefix on that one.

...35all right didn't get to anubis and i don't want to bum rush it but i really do want that today especially since it'll probably let me unban a whole bunch of ips the the hassle with banning ips is there are always a whole bunch of vpns that have legitimate users in them and i don't want to ban those users espartapalma America Futbol club, UNAM (Pumas), Cruz Azul & Necaxa only the later doesn't play in CDMX nowadays
The IP bands happen up at the Linux IP table layer, so they can't check, hey, are you a logged in user? Because if it happened at the caddy layer, I'd be happy to let them in. Hmm.

02:59:33So I wanted to say we got to like 2.9 before. So if this gets to, call it three gigs, we can feel like we passed the point of the crash.

03:00:06Hmm. I could, no, actually I can't. I was going to say run a query that counts the number of records in each table, but that's not a trivial thing to write off the top of my head. It's a little easier in SQLite than some databases though.

...52Nogweii are you considering migrating to SQLite for prod?
Nogwa, yes, that's issue 534. 539. pushcx https://github.com/lobsters/lob…
I actually knew that one cold at one point, so I must be getting tired, because we are coming up on... Well, we are at three hours on the stream. So...

03:01:23To summarize a fairly long issue, because we've got a second while this thing takes a long. There are. Now, I guess I don't have to summarize because it crashed. On a different record.

...57But it's the same exception.

03:02:07But those are comments rather than story texts. chamlis_ guess that's one mystery solved
So that tells me, yes, somewhere in a comment, someone put a null byte. Maybe they were copying something. Nogweii so BLOBs for everything?
And these random comments that got dumped are just from the block of 1,000 comments.

...55Let's think about it the other direction.

03:03:12Sir, you're not happening on the mouse.

...32Why am I not save?

...43Why am I getting? Let's stay to my end. I didn't undo far enough. Where's development? I did these out of order, that's why. Gotta love Rails configuration. I'm gonna come up with some better method of toggling those as much as I don't want to. How am I still fucking getting a SQL like? Did I not save? I was in the test section.

03:04:36But we know there's some in there. marcoroth_ don't you need % signs?
Oh, boy.

03:05:06So this is... Oh, I do. You're right, I do. Thank you. I'm getting tired and I'm making bugs.

...26There is literally one. Okay, so that's not so bad.

...36There are literally two comments.

...47marcoroth_ strip them!
chamlis_ guards, seize those commenters
I want to see these comments. I'm going to finish the stream here. VYH6M. look at that look at that beautiful question mark sir the mouse is the cat the cat is on the mouse that's why i'm retyping these comment ids what's this one n zero c zero y five the linked paper says poof and they pulled it out of a pdf

03:06:36You know, the one table we didn't check here is a short ID from stories where description, none, usernames from users where about, username. Okay. None there. Let's see. chamlis_ user bios?
Let's switch this back to account from messages, which is where message like. Is it body? Okay, so none there. UserBios is that previous one. It's called about in the database. So. chamlis_ oops, misread
I'm not going to bother checking story titles, because we would have seen that when story texts failed, right? I'm going to bother checking.

03:07:48It's okay. I don't expect everybody to know all the random... Yeah. I mean, there could be subject on messages, but that's less likely if nobody was... Yeah. That's pretty much... whole of the text field the users can type in. I mean, I could have mis-pasted into the moderations, right? There's action. Yeah. There's reason. That's pretty much it.

03:08:48marcoroth_ but I guess even if there were some more NUL bytes, they are most likely not intentional LUL
There's a lot of confidence order, though. But that's the one that we can drop and recreate. Actually, no, we can't. We just have to make sure it's a blob, which it should be, right?

03:09:09Yeah, it's binary.

...16Well, it looks like none of these are intentional, it looks like the the two comments, the one that's on screen and this other one are both. pasted out of other material. So. At least they don't look like our bugs.

...45marcoroth_ and the one title?
marcoroth_ or the subject or what it was
So if I delete those two null bytes from prod, we should be good title. There were no titles. There were no subjects. I can see the subject here is zero. Which one are you thinking of? Nogweii there was a post
We got the two comments and we got nobody else.

03:10:21Nogweii were you thinking of that one @marcoroth_ ?
Oh, you're saying the one story text, yes. Yes. So story text is the... marcoroth_ the story text one yeah
we grab the copy of link contents in case they go down so yeah that one okay all right

03:11:06I guess that's my to-do for the next stream is to just remove those zero bytes. Honestly, if I just literally delete the character. Yeah. Yeah, I'll do it with like stir replace or something. Because if I do it with a MariaDB query, I don't have to worry about there being layers of encoding or any other fun jazz. All right. Very exciting. Let me leave a comment on Thomas's.

03:12:13Do we scroll back up to where channels found that link yeah.

03:13:19Nogweii further question: What will you do when someone posts with a NULL? Right now it'd be a database exception which doesn't seem all that clean from a UX perspective
Nagrae, that is an excellent question.

03:14:13You know, do we need to catch it or do we just want to strip it like story text?

...44marcoroth_ is this only happening in sqlite? I almost sounds like this would also happen in ActiveRecord with mariadb
Nogweii @marcoroth_ only in SQLite yeah. MariaDB is happy enough with NULL in text fields
I mean, Marco, it obviously didn't happen with MariaDB, or we wouldn't have them in the database, right? This seems to be a SQLite-only thing. MariaDB is like, you can take anything, and I will consider it valid input.

03:16:20Nogweii wonder if postgres would complain?
marcoroth_ unless that changes in-between versions or something
Changes in between versions. Oh, man, that makes it even more fun. All right. chamlis_ thanks for the stream!
Well, on that fun note, I'm going to roll out and end the stream here. I hope everybody has a nice carnival or... Facts knocked something. Nogweii have a nice day!
I didn't catch the German term. whatever your local holiday is. I will see you next Monday for the next stream. Take care.