Interesting in Boring Ways Unfortunately

02:42pushcx This is the Lobsters office hours stream, ask questions anytime!
You know, this morning, somebody had a just, I don't know, I think a half an hour ago, maybe an hour ago, somebody had a bot drop in the lobsters chat room and some more. And it was a really strong reminder that those things only see one continuous stream of text. There isn't really a concept of prompt injection because there isn't really a difference between the prompt and the activity. Because the LLM chatbot started posting messages to the Lobster's channel that implied that it was read-only to the Lobster's channel and it was spitting out control plane kind of information like i am allowed read access to this and not write access to that and i don't know where these messages are coming from and it was just yeah llms the way they fundamentally work is one stream of text it's not not even separated a little oh boy all right i'm a little froggy this morning we'll see how it goes Ah, so let's pull up, let's pull up code. Get over the base. I'm in the right place, right? Yeah. All right, so topics. Where else was I? Usually I have a title idea, a post stream, to-do list. The big thing is 1871 Skullite migration attempt, which I should pull up that gist. in case folks are curious about it we've talked about the attempt to move to sqlite a bunch on this stream and attempted it on saturday like five days ago whatever that was six days and it didn't work out there wasn't any big crash but we worked a whole checklist so this is pushcx https://bookshop.org/p/books/th…
I know I have mentioned it many times on stream, but this is one more... Yeah, one more plus one for the checklist manifesto, because even though we had a big unsuccessful database migration, because we use checklists, which are a developer superpower, we were in read-only mode for... God, I wasn't watching the clock. maybe two hours, maybe three on Saturday. And we were down for maybe 15, 20 minutes. But there was no disaster. There was no outage. More than that, there was no data loss. And most importantly, we weren't afraid of any data loss or serious outage the whole time because we had a a nice checklist and we had thought through possibilities. So I'll talk more about that, but let's go find that checklist. Put this in the notes. The basic gist of it was... All my gists. That's every gist in the world. That's... There we go. Didn't give it a title.

06:48So let's take a look. What else is going on? Oh, I did just recently merge a small bug fix, fixing the route for origin editing. That was nice. So we'll put that in the PRs.

07:20And in case anyone is like, hey, wait, this bug says that mods can't access the form for editing and banning origins. That is correct. So I did those that I had to block recently through the prod console because I can just call the origin finder create by or whatever exactly the same way as a controller does in the production rails console. this one was nice when did i do this yeah i did this one on stream all right so this one is caches ui i saw i got a con comment from fedamp who is a regular contributor now who does a lot of front-end accessibility stuff so i take graefchen Heya limesHi
his concerns pretty seriously.

08:35Hey, Grafjan. New user. Oh, invitee. That's great. So James has picked up a couple of content things recently. marcoroth_ Hey HeyGuys
This one, I don't know if you're present, DZ, but This was your feature request from, I think, last Thursday or last Monday. Oh, hey Marco. All right, so let's just go in the order they are in. Because then we can start with something small. olexsmir HeyGuys
And then I can work back up to talking more about SQLite, because folks weren't immediately jumping in with questions. So I would like to do the regular maintenance stuff of moving things along. This is a new Twitch chat emoji on me, this hey, guys. Hi.

10:16Hmm, maybe this one is more complicated. Yeah, I didn't know this stuff on UserLink Helper. Let's take a look.

...44Oh, and in case anyone's wondering, I had thought about streaming on Saturday when we were trying to deploy the SQLite stuff because I was already on a call with Thomas, the author of that PR. But one of the things I have mentioned on stream is that maintaining this running pattern takes like 25% off of the top of my intelligence and my attention. And it seemed like a bad idea to be logged into prod as root and doing complicated things and, you know, trying to not destroy data while also, you know, having one arm tied behind my back. Let's make things a little easier on me. All right. So why bold? I guess. And then safe join. I don't know. Safe join off the top of my head. It's got to be in the docs, right? Let's bring that up.

11:57HTML safe string. It's a little odd to be doing escaping. I don't know. graefchen Yeah. Streaming a migration does not sound that smart to me. limesD
Rails is so hit or miss about where escaping happens. Sometimes you do it in helpers, sometimes you do it in the templates. This is one of those like, I wish types could solve this. Yeah, and you know, if someone is a very avid watcher, you can watch me goof up regular deploys of the site all the time on stream because i'm distracted by doing the streaming and you know the risk on those is fairly low but it does feel like about once a quarter i get a little confused about what is deployed and what isn't and that's just running my regular deploy script where everything is automated so twitch is buggy

13:21The invitee is recently invited by the inviter.

...59What did I say in that issue? Did I explicitly say the word mentorship? No. But I think I've been thinking about this since doing this last stream.

14:32Definitely not putting in a baby emoji. It's a little too cute.

...49I've been thinking about the slop contributions we've gotten, and getting random emoji is One of those little signs, it's a little funny, because I've started using some emoji in the admin surfaces. But I started well before the chat GPT blew the lid off, what, two and a half years ago?

15:33olexsmir i started using emojies right after llm became a thing
Okay, so I have gripes about the exact message and the class name. Do I have gripes about the structure? Like, this is good.

16:58graefchen I honestly like some emoji and some emoji I absolutely dread. limesLurk
marcoroth_ isn't that .after? comparison kinda backwards?
No, because of the ago. marcoroth_ I mean more in the sense of how it reads
Wait, created ad is actually, that does seem backwards.

17:16Yeah, it's, that could be clearer.

...35marcoroth_ I have just been thinking on how to improve it, but I'm blanking LUL
If you were created. Wait. Yeah, it's. It's correct, but I had to read it twice to and think about it to say that it was correct, which means it could be clear.

18:07I don't have a great idea there. I do prefer the after and before helpers to greater and less than. Not that I'm super consistent, but.

...27All right.

...34I don't think this needs to be a separate function.

20:12hovsater Good afternoon. :)
Hey, HuffSider. What is that icon next to your name? Twitch recap. Okay.

...25There's tests of new styled user link, right? There's no tests of styled user link. Whoops.

22:52hovsater I didn't know Lobsters was written in Ruby using RoR. That's awesome. :) I've been working professionally as a Ruby developer for the last 15 years xD
bsandro V.v.V
Oh, hey, Bissandro. Oh, Hubsider, yeah, neat. We've been using Rails for, what, 13 and a half, 14 years? 13 and a half years.

23:40hovsater With that said, I went into a rabbit hole a year or so ago, so all my free time is spent in Nim, building my own text editor, haha.
Oh, neat. Yeah, I've been tempted to go into the rabbit hole and make my own language and web framework. What's special about your text editor? Sounds like a fun project.

24:02Alright, so there's that. Yeah, here's another one from James. So this one, James talked about this in the issue. Okay, it's already in the list. But Federico noted that you can't click on tags in the submit form because of complicated interaction with float layout and the Tom select replacing controls. And James spent a minute playing with it and figured out a way to patch this one line to use Flexbox. hovsater So, I've been using https://craigbarnes.gitlab.io/d… for the last 10 years probably (before that I was using Vim). I love dte, but there's a few fundmental things I would do differently, hence writing my own.
And honestly, the whole form could, but I would rather just fix this now and come back for the whole form later.

...59hovsater It's highly inspired by dte.
Great. So you've been using DTE. I don't know DTE. Hold on, I'll have to pull that up here. Let me bring this in.

25:40So let's finish that thought. And then this one I know has a little conversation I got to think about. So where's, okay, so it's in light mode, but small and easy to use text editor. That's, you know, easy to use is almost a negative to me. And I say this as a Vim user, but I am happy to have graefchen I got into more or less a try to get into the rabbit hole about how the djot markup language works to then be able to use the way the markup language is parsed into an ast to copy it for my own small lang. limesSit
an editor with a learning curve if it means I have a higher power cap. Nano is really easy to get started with, but you can't really become a nano expert. Multiple buffers, macro recording, key bindings, command language. That is one place where Vim has quite a bit of historical cruft.

26:45Macro, Unicode, 10 millisecond startup, nice.

27:05Is there a, what would be really interesting to me is twitchtd good morning
A description of what's interesting about DTE compared to other editors. graefchen morning limesHi
What is it better than Vim or Emacs, just because it's terminal-based, or VS Code? Ah, hey, Thomas. I was hoping you'd drop in. I've got the SQLite stuff on my to-do list. I wanted to run through the PRs and issues first. Anyway, sounds like a competent editor. marcoroth_ C99 <3
Oh, man, it's written in C99. That's an interesting challenge nowadays. So let's look at this one. Oh, Marco, did you? No, you wrote. I'm trying to think what you wrote. marcoroth_ yeah, also C99
hovsater So, before dte, I was using Vim. I loved Vim. But I also felt it did too much of things I don't really care for. I've always been rather minimalistic, so eventually I just gave dte a try.
all your stuff in i don't remember which which c standard did you target was it 99 isn't there a more recent one i don't know that i've written c since 99. oh okay

28:47marcoroth_ there is C21, I think? but C99 seems the most portable, and C21 doesn't really offer much more
Desktop logged out, desktop logged in. This is nice. I think I commented that I'd like it right aligned. Oh. Oh, yeah, that's pretty bad. Fit with content and margin auto. hovsater What I like about dte is that it's highly extensible even though it doesn't have a plugin system. It interacts well with other UNIX tools and you can seamlessly spawn other programs from within dte and have dte parse the output of those programs to do stuff and so on.
That's pretty good.

29:15This one's allowed. hovsater So even those it's rather small editor, it's very powerful.
He opened this PR before I approved his other one, so it's not auto-running. There's C21. Okay. Yeah, I haven't paid attention to writing C for a long time.

...39Ah, neat. So, Hope Sater, what is it you don't like or you feel like is a missed opportunity for the editor that you're tempted to write your own? Or I guess not tempted, but starting to write your own. It's like the other issue, so caches at the same time. I'm okay with having multiple open. bsandro c99 for life, but lately I've quite enjoyed c23 with gnu extensions (gnu23), raw string literals are too nice to ignore
vinitkme Hello, Hello!
and the current implementation in prod doesn't work with orca nor for keyboard navigation that's fine with the current appearance that other issue mentioned in the internet tracker that allows every caches to that one's fine yeah it's kind of a ui thing so i'll say that let's go look at this code again

30:36hovsater The syntax highlighter I would do differently. Not because I want more highlighting (I don't, haha), but I would make the grammar more flexible. dte also has a concept of compile commands that allows you to parse the output of a program such as make and then navigate errors. I want that, but not in the form dte does it.
I haven't it all right so all right so we're cleaning up the yeah this is one of those places where the css classes and the text on the screen have slipped out of sync because people didn't recognize that archive the verb actually meant archives plural and So caches plural is a lot clearer. And I think that's on me that I didn't go all the way through and do all these CSS classes. They didn't change it to. hovsater I would rather have the output written to a "special buffer" that allows me to navigate the text within the buffer. Kind of like quickfix for vim.
That's OK. Labels for archive underscore for. Okay, that's fine, the screen reader's not gonna jump on that. Syntax highlighting is different.

31:53Ah, yeah, it's one of those lessons that we keep learning in programs where it's like, Somebody starts something and they solve a problem like make output. And they tend to solve it in a specialized way. And then a couple of years later, someone comes along and collapses it down and goes, oh, all of these specializations are actually one core feature. And jujutsu is doing that to get in spades. hovsater I also want acme-like behaviour as in text in buffers can have special meaning.
And I can see how there's a lot of opportunity for like DTE to do that to Vim or if DTE has been around for a couple of years for you to do that to DTE. Any particular reason you are starting your own editor rather than contribute that to DTE? Acme-like behavior. Acme is an editor, right? hovsater It is, yeah, from Plan9.
I don't think I know it. I just have heard the name. So if it's a label for an archive, stay as a pointer. I think we have a button class that can get reused.

33:22Yeah, a.button.

...35Where's that the opposite, where I want the link to look like a button? Or does it just act like a button? Dang it, I don't know my own semantics off the top of my head.

...55hovsater I started of contributing, but I also don't want to force my direction upon the project. Craig (the author) is great, and I think the current vision of the project is what's going to stay there for the foreseeable future.
Here we go. OK, so this one is styled like a button. Where's the settings index?

34:12There's link post.

...20Link post has the class name. That's not useful.

...51hovsater off*
Oh, this is the opposite, styling a button to look like a link. Maybe that's just what I'm thinking of.

35:05Yeah, maybe this is OK if it stays independent.

...21So then this becomes just caches. How does this work without a... Oh, it's the start. I'm thinking it should end with a wildcard character. I forgot to have the character to mean starts with. I would rather have a class than hang on those.

36:18So there's a checkbox, the dropdown, the divider. Oh, well. Seems reasonable then. That was to have said her about your contribution plan. So these are the repeats stories show. All right. bsandro @hovsater huh dte looks really nice, just tried it out a bit. I'm a heavy nano user :D jumped off vim several years ago
Okay. Caches. Yeah, this is.

39:18hovsater @bsandro It is. It's a shame it's pretty much just me and Craig that use it, haha. :D
hovsater @bsandro If you're open to give it a try, I'm available in #dte on Libera on IRC to help out with any questions. :)
You know, it's funny that you say it's got a small user base. That's a pretty comprehensive set of features. bsandro thanks! i'll try it for some time, the default stuff looks very good to me (i had to tweak some bits and parts of nano)
I would have guessed that it had a fairly active community from that pretty big feature list.

40:09hovsater I've seen 5 people come and go over the last 5-6 years. Craig and I are the only daily drivers as far as I'm aware.
hovsater Could be lurkers though
It's just

41:02Let's go back.

...36Okay.

42:12Yeah, so this is nice. I'm being a little nitpicky, I guess, this morning about those front end things, but both are real nice little fixes. All right, so Thomas, if you're still here, probably, let's talk about how the SQLite migration went, right? Why didn't, did I look at issues? I don't know that there was much. Yeah. We pay open one related to his full request. Maybe that can close. And then otherwise, there hasn't been activity unless I close something in the last day or two. Those fixes that we already talked about. Oh, I closed out a feature request that's me looking for designing stuff. I still love type ID. It's just really I like it more for an API than a URL because they're so long. And I would rather not take on the pain of changing every site URL unless there's like really a clear benefit in every way. Yeah, these are all done. All right, so let's go into this. Right, so there's this pull request that Felipe did. And there were a couple left. Yes. I, okay. So this is good. This is just giant thing. And I just need to throw on a label to confirm that this is above. Ewo_On_Sight Sup
So this is, we do soft delete on pretty much everything and Standard and database consistency both are like hey you know how you have foreign keys you should use foreign keys to destroy as much data as possible. Ewo_On_Sight lcolonShine
hi EWO on site. And since we do soft delete rather than trying to delete as much data as possible, we should be throwing as many exceptions as possible.

44:46Ewo_On_Sight On the bright side your DB will get a lot lighter. LUL
So that's pretty minor. All right. So going back to Saturday, this was our SQL migration checklist. So Thomas and I were on a call and we wrote all this stuff out. I wrote the first draft of this checklist on a stream, what, maybe Monday of last week? And then pulled it out on Saturday morning, and we fleshed it out some more, and then we just started working through it. And the best part about writing the checklist is you think of things you never thought of before, and then you have some structure when you realize things day of. so like day of we realized or you know in progress we realized what was it there was a hassle with creating the database on prod and we got that solved but it really helped us keep track of our work So as you can see from the Xs, we checked off most of this. And then can I just edit this? Can I get you to render? Didn't even think about it at the time. No, you don't want to? Did I not click Save?

46:34Wow, GitHub, that was special. That sure looked like a full page load when I hit save, and then it showed me the old version of it, and then I did a shift reload, and then it changed to what I actually saved. Well, there's this streams GitHub UI bug. marcoroth_ this has been around for the last week or so, it's quite annoying haha
And I'm counting that one because it definitely felt like single page app jankiness. I've started seeing these on Blue Sky now, where every couple of days on Blue Sky, somebody posts a GIF of the GitHub front end doing ridiculous stuff. Yeah. Ewo_On_Sight Githubs days are over gotta go somewhere else.
How's that East River source control coming? Ewo_On_Sight I'm also moving away
Can I just pin all of my hopes on that, that it will have every feature I want and no downsize whatsoever, and then I won't be disappointed when it launches and is imperfect? What could go wrong with that plant, right? Yeah.

47:55Ewo_On_Sight Going to gitlab
So the first crossed out item is if prod doesn't immediately melt down. And prod did immediately start melting down, which was especially concerning because the site was in read-only mode. So to be a little more specific about meltdown, I mean, prod load is typically, it's a four-core machine, and prod load is typically around 1.5, which tells you the box is like 30% utilized, which is... a comfortable place to run prod. I'm happy with it for capacity planning. I wouldn't with a commercial service. With a commercial service, I would have already upgraded us to be more conservative because traffic spikes for business are usually when good things are happening like, you know, somebody famous mentioned you and you got a lot of inbound traffic. Well, you really want to be up for that. Ewo_On_Sight Money lcolonShine
or have something like judo scale to handle spikes. Ewo_On_Sight Customers lcolonShine
For us, I'm just kind of like eyeballing and being like, if we're at 20, 25% utilized, we're fine. Our load immediately jumped up to five or six. And the rule of thumb is if your load number is higher than your number of CPUs, you are having a bad day. And especially without doing any significant that's what was missing from this checklist was stopping yeah PG jobs and made it into the revert checklist that's what we had to do in prod. Especially without any significant number of rights go into the database, the fact that load was. jumped up so high was very concerning. Because the big fear with SQLite is even with the wall, you can only have a single writer at a time ish you know with the wall they at least get in line properly as opposed to rollback mode they just throw exceptions so so that was a big problem and even without taking more detailed timing info we knew it was unusable in prod And we spent more time in read-only mode tinkering because we wanted to try two or three obvious things. Chamlus, who's usually on these streams, had an idea. We tried that and it didn't work. I kicked a couple of things and it didn't work. We talked about, well, should we just throw RAM at it? But the problem, I don't know, it was kind of a hunch, but the problem seemed big enough. I didn't want to just try and throw money at it. because it didn't seem likely to solve it. The problem seemed larger. twitchtd we also investigated the sqlite options that were set by rails
Ewo_On_Sight You know how expensive RAM is???
Ewo_On_Sight LUL LUL
And we kibitzed a little in the Lobster's chat room, too, because one or two people there know SQLite and prod usage. And none of that seemed to immediately fit. pushcx https://gist.github.com/pushcx/…
And yeah, we talked about all the pragmas. Those ended up down here. Oh, I should just share the link to the checklist. I put it in the topic list. Yeah, and all of these are the Rails defaults.

51:46And Thomas, I don't think I discussed it with you, but I did get in the Lobster's chat afterwards, somebody tossed in a message about one of these. Probably cache size because everybody looked at cache size, which I thought this was negative 2000 and was a count in pages rather than positive 2000, which was a count in like bytes or kilobytes. twitchtd positive = # of pages
There was something very odd about the format of this number where negative numbers meant something quite different. And a couple of people caught attention on it. Positive is number of pages. Okay. twitchtd negative = # of kilobytes
And then like negative was number of bytes or something. There was someone, and I want to say it was this 2000 value that was funny. And then negative was number of kilobytes. Okay. Thank you. The other one was mmap-size. So I pulled up my note off screen here of what someone told me. And they said, this was somebody named Sorceray, S-R-C-R-E-I-G-H. If you're around, you can put your hand up. They guessed that the SQLite performance problem was due to mmap-size being significantly less than the database size. twitchtd it's really weird, never seen a sign change the meaning of the units before
And the number of pages in the typical work set being larger than the connection page cache, which I would guess is this one. Mmap size being 128 megs, meaning the first 128 megs gets mapped and anything past that is loaded via read. Normal client server has a large buffer pool, but SQLite can only use Mmap for the shared cache. That makes sense.

54:04But then they did caveat that that's just a guess. Yeah, Twitch, you know, or TD, we were talking about C99, and it strikes me as a very C approach. I have seen that kind of thing before, and it's the C type system is, and I'm not trying to be mean here, but it's anemic. It's very oriented around davidofterra PDP-11
the machine types that were available on... What was that? The PDP? Rather than... So it's hard to even say that its type system is oriented around machine types. It's oriented around PDP-11. Thank you. I knew someone would know. It's oriented around the machine types that were available on a machine that hasn't been made since, what, the late 70s? davidofterra Early 80s
So that thing of... positive numbers are in one unit and negative numbers in another is one of those things that leap off the page to me as this thing is written in C. Because if you think about it, C doesn't allow you to express something like this is an integer and a thing you would have to make a second type, right? So there'd be pragma cache size and pragma cache units. And then yeah, just C is not good at compound types. Oh, hello, sir. The boss has entered my lap rather than his nice sitting spot.

56:04All right. hoistbypetard the last pdp-11 models were introduced in 1990. https://web.archive.org/web/200…
So then when our first couple of tries didn't work, and we didn't have a high confidence answer for what was causing the performance problem, and knowing that Saturday traffic is half or less of Monday traffic, we started a revert process. And if I had been really smart, we would have written this revert checklist. Wow, that is a lot later than I would have guessed. Good trivia, hoist by petard, thank you. I should have written this revert checklist beforehand, but it was not too hard to just go through the previous checklist and figure out which steps we needed to do in reverse order and then do them. So once again, I will say, like, if you are a developer, you should read The Checklist Manifesto. It's up there with the thesaurus in terms of books I hit developers with. God, I should make that a blog post at some point of like, the books I most throw at programmers is like The Pragmatic Programmer, Badass by Kathy Sierra, Rodgett's Thesaurus, The Checklist Manifesto, Oh, and the first Edward Tufte book, which... Which one was that? Was it... The Visual Display of Quantitative Information, I think. Yeah. hoistbypetard I still throw Brooks at people
That's my programmer's bookshelf. All right. There's got to be more, but that's just off the top of my head and turning to look at my actual bookshelf, which is in my line of sight. hoistbypetard yep
Brooks. Oh, you're thinking Mythical Man Month? Yeah. At some point, we're going to start saying that adding LLMs to a late project makes it later, right? At the same time, I don't know. Some of the maintenance stuff they can do is pretty handy, especially if instead of trying to one-shot a problem, you just set up with one of those things that can browse around and read and compare all the config. try and diagnose a bug even because they kind of they're iffy. A lot of the times they want to do like the junior dev thing of making the worst possible bug fix where they fix the bug at the last layer with janky one-off code, as opposed to run it all the way back to where the bug was introduced and really fix it. So getting them to investigate and figure out any part of the sequence and then have an experienced human makes the fix is better. I don't know. I also saw a cute blue sky post that was like, in the format of that meme about the torment nexus, it was like, at long last, we have invented the silver bullet from Brooks's famous article, No Silver Bullet.

59:37So where does that leave SQLite? Thomas and I have had an email chain going with, I kind of threw up the bat signal. Did I throw it in 1871? Or maybe I just did it in the chat room. Or maybe I did it in 539. We're getting spread out. unify our work.

01:00:08Trying to stay coherent. At the same time, especially with this fucking thing, it is hard for people to realize what's relevant context. The GitHub issue structure is just breaking down under a change that took a couple of years to figure out and then a couple of months to work through and then had problems in prod and needs to go back like this is not adequate tooling so yeah they must have seen me talking about it in the chat room Somebody emailed me and I cc'd Thomas into the discussion. They did some performance analysis and had kind of confirmed a hunch I had about what was wrong, which was almost certainly, and I'm like, I don't know. 80% confident on this rather than 100% but. Our hot path view is rendering all the comments on a story. and A smoking gun be suspicious of this thing is well. When you view a story and the comments you're going to call this function story threads to load the comment trees. And there is ConfidenceOrderPath, which is a giant performance hack that I wrote for MariaDB. And the two suspicious things in there are it's a performance hack for a different database and I wrote it. And so with those reasons to be suspicious of this code, I'm fairly confident that we had high load because this performance hack did not translate to sequel light and the the person who emailed and i haven't i'm kind of talking around their name because i haven't i've only talked to them a little bit about what kind of public credit they want for it and so i don't want to throw their name out if they're like no i would rather just be behind the scenes that's fine twitchtd the translated sqlite query does a full table scan of the comments table
They did some analysis and they said that they think that the way SQLite handles this recursive query, it's not using indexes effectively. One of the things that's happening here, yeah, is In the innermost base case query, we say we only care about comments on this story ID. And MariaDB is correctly saying, well, if you only care here, that also means you only care in the recursive and you also care in the external. And I'm not going to do the relational algebra, but it does, it is a valid performance optimization. And apparently SQLite doesn't recognize it. And as one pass, they said, well, you can like copy this stories ID thing out here. And then SQLite will quit doing full table scans. And they showed up in prod and not in testing because by default, the fake data task only makes, what, a couple hundred comments, maybe a thousand comments, and we have more like 675,000 in prod. And then generally when you are testing one off and seeing if pages load, it's just you clicking around loading pages one at a time with a human level break of hundreds of milliseconds or even thousands of milliseconds between them. Whereas prod is more like 10 queries a second. And so then you get into contention and all those other good things. So that's why this didn't immediately turn up. So Thomas and I are not sure how to summarize the thread, because I'm not sure I've read the last comment or two. I was offline the last couple of days. We are doing more performance testing, especially that, well, if we create a dev database with a prod-like number of comments, and compare two versions of this query number one the straightforward porting of this to sqlite versus the second version of putting the story id out here does that solve our performance problem and that might be good enough to go to prod but there is also the like what if we got rid of all of this because this This whole thing with having a recursive query is it's two things. We want the comments back in a tree structure and we don't want to do multiple round trips to MariaDB because there's a significant overhead to each trip. And so, you know, There's the classic one plus N problem where you do one query and then that prompts you to do more. And that's not expensive in SQLite, especially if you are going off of primary keys or indexed columns, which are basically the same as primary keys for this context. So the naive thing in SQLite would probably actually perform better than all of this nonsense. And I call it nonsense It's a hack for MariaDB. It's not a hack for SQLite. SQLite is going to have different performance characteristics, like this whole cop length thing. And this being three bytes is, yeah, three is oriented around internal MariaDB row level limitations so like if i changed three to four we passed some structure size and performance fell off a cliff and a very funny thing about sqlite is if we do the most naive possible one plus n thing of give me all the comments that have that are on this story and parent comment ID is null. So that's all the thread routes. And then we loop over those and we go for each one of those, give me all the comments where parent comment ID is this comment. And we do that loop in Ruby. We would blow the performance of this out of the water. And we may even beat the straightforward fix of copy this to the outside. to hint better. So that's basically where we're testing. I'm going to throw up my little break thing and step away for a second. I'll be back in, I don't know, two minutes. Thomas, if you have any other ideas about the current status of the SQLite migration or anything else you want to talk about or do on stream, I'll give you a couple of minutes to type that and I'll be right back.

01:10:55okay let's turn that light off there we go all right so nothing from thomas but the fact that he didn't say the word nothing makes me think he maybe just stepped afk oh well it's a live show anything can happen all right so what else is on my mental to-do list because twitchtd not much to add, I think we'll want to test perf a bit more since this might not be the only issue
There's been a bunch of stuff around SQLite that's been top of mind. And I don't think I have much else. Checking my inbox here real quick.

01:12:13Yeah. And there are a couple versions of this query. Like there's another one up here for user threads. That's what this is. But honestly, I think that's just about the only places we drop to SQL. I'm sure there's one or two more around a complicated join. Yeah, so we got one in story. Right, this one is about wanting to do an efficient update. search oh oh search i gotta bring up something off screen i it all right i got the bug tracker loading here off screen and it is off screen because there's all kinds of it turns up like api keys and cookies and stuff The generated queries in the search engine definitely had a SQL injection possibility. And I spent a minute on Sunday double checking. Nobody noticed it while it was live for whatever that was, an hour, hour and a half, but it was definitely a SQLite injection in the search engine. Because if you searched for, let's see, oh yeah, there's IPs and Cookies, hold on, can I get something I can show on stream? Yes, let's just copy this up.

01:14:35so yeah the general something about this generated syntax thomas did not work out and i'm genuinely surprised that our tests didn't catch it this was literally just a search for the word at proto with nothing else can i grab yeah so i can grab this yeah so like here's the formatted info about the query and exception which i guess i should put the All right, not query request. And you look at this and it is the most bog standard search of literally one word, not a stop word, no punctuation, no anything. It is totally possible that I missed that. I'm going to add a spec to the search engine to the outermost layer of the search engine.

01:16:14Yeah, I guess I didn't write just a smoke test. I guess this is. But if you got an exception, it didn't say zero results.

...52is it what comments actually you know that one was that one was stories were the others also stories stories no comments also through yes the same exception somebody searched for the word pipe wire in comments they got the exact same exception yes so i would rather search comments just because i know it's faster to create and then we expect the response to include and we'll say

01:17:49Create comment with the comment at proto isn't really distributed. See, this is me putting silly stuff in the text messages. This is one of those, like, I've seen this comment so many times. All right, so. Not settings. It's in requests.

01:18:28And it failed. Why did you fail? Because you certainly work in prod right now. Going to get a better format of that. Can't skim it without the new lines. And my hope here is, once I fix whatever my little bug in my test is, is, Thomas, that you can pull that into your branch, and it'll turn up whatever the SQL injection bug is. How are there zero results for these? Did I typo? Was there?

01:19:32Now, why does this work and creating one down here doesn't? Oh, this. That's right. twitchtd I'm not sure that's a sqlite injection, since you can search by column and that match looks properly formatted
I left my comment explaining. All right, so we're not going to world.

01:20:05I'm calling it SQLite injection because anytime I see a database query throw a syntax error, it feels like we're set up for an injection.

...28Okay, so that's working.

...55So I can just grab this query and run it. Get the exception. Yeah.

01:21:24So is the error here the extra single quotes? Or is the error here the extra double quotes? I don't know SQLite syntax well enough yet.

...40Nogweii it's complaining about the column it's ordering by
The error is both?

...53Nogweii hello btw
It's complaining about the column. Oh, yeah, it is. Wait, why is it order by app proto? And hey, Nagwe, hey, you know, figuring out a bug that I'm just reading right through is a great way to say hello. Order by, yeah, how is that getting in there? Let's go. Look at this. You can tell I'm just looking for the rightmost tab. That was what that muscle memory was. God damn it, get up. Come on. Oh my god. Alright. So let's search for order by. And it doesn't appear. Alright, let's look for search. Oh, I bet it's not even rendering the whole diff, and that's why I can't Search for anything. Oh, great. Great that you had an animated GIF. Now let me just scroll by every fucking thing in the world so I can actually get the text loaded. Now we'll search for search.

01:23:15Now we'll search for order.

...22yep there's our bug

...53And then it's going to be, yeah, it's repeated for comments. twitchtd yup
Thomas, did I see that you opened a new pull request? Come here. Just show me open. Yes. All right, then let's go. Or was it in its own file? No, it's in search.

01:24:47So I'm just leaving these comments so that you have, now that we've figured it out, really, Nagwe figured it out.

01:25:07All right, so there's that. And then, to write one more search because i saw a bug in prod so i expect this to be a failure here so let's go to on that search spec actually let's run all the search related specs Spec model search. Just want to see everything green and then I'm going to write the spec. Then I'm going to get mad because it's not going to fail like the bug. What did I break? I'm going to fail like what I saw in prod. Why are these?

01:26:05Where am I?

...12I'm coming off of main.

...22Build is green, right?

...34It's going to be great. This is going to be green. And then when I run them individually, they're going to fail because CI is passing right now.

...47OK, OK, what's fun about this?

...56Is it that before and after in the search feature spec? Yes, it is.

01:27:16All right. I don't want to go down that rabbit hole and I want to do. That was a spec request. Yeah.

...34All right. That's that's like having a green test suite. So what's happening here is this spec has this cludge with after and before to create a comment and trigger MariaDB updating its search index because MariaDB waits for the end of a transaction to do that, which is probably a performance thing. And RSpec really wants to manage transactions and the creation and deletion of test data And so anytime you do it manually, you end up with these kinds of really fun issues. And I say fun because they're exasperating to track down and the fix is always like add three more layers of before and after hooks or change your creation and deletion of test data. And I just don't want to go down that rabbit hole because that's a great way to lose half a day. And I don't want to do that. Just want to write a test for the bug I have now.

01:28:56I'm just going to make it worse because I'm making more data here. But I want to have different search terms. What's a good search term here to have in the title? twitchtd LLMs won
I want to have two words in the title and then reference it here. And I want to have two words in the title so I can search for one and then when I look at the page, I should have the second one on the page, right? LLMs one, not quite. twitchtd :)
I could just do foobar.

01:30:00still green after adding that test data, which is the kind of thing you have to ask when you manage it manually. No. Cannot delete record. It's exactly the thing we saw earlier.

...26Nogweii you're deleting the story first
Nogweii oh wait nvm
of dependent comments there aren't any dependent comments because we just destroyed them here no i'm not i thought about that this is the kind of fiddly cleanup bar spec generally protects you from

01:31:03Could the error message be wrong?

...18And it's actually failing because of a different associated record. Because there's like the taggings, the votes,

...49Maybe not. Oh, it just doesn't have destroy on the association. Is it destroy all? No.

01:32:28I shouldn't have any suggestions. It really should just be comments. Does the story factory create a comment? No. Up here. How many comments are there? So we only delete the one we created. Is something else creating an extra comment for extra funsies? No. The number is 1, which is the number of comments at that point. And if we do that again, we might actually get a cached value. 1 and then 0. And then we destroy it, and we can't because there are comments.

01:33:45So it can't delete it because of dependent comments, but there are no dependent comments.

...57twitchtd Could it be related to the dependent relationship rework recently?
Nogweii is the story caching the list of dependencies?
Maybe, what's the s equals story.list, s.comments.count. And then if I do it again, on developer mode, turns off some caches. What's the, yeah, that's exactly where I'm getting to nothing accesses, maybe a test access. Is it so comments, there's like a refresh or a reload in here. What happened to your ASCII references? Yeah, okay, so there's a reload. So if this, I just say foobarStory.comments.reload.

01:35:08Now we're happy. Okay. That's a fun one. And that was no fun.

...35We're going to search for foo on the stories. We're going to find bar. Now, I expect this to be green. I'm doing this because I am trying to isolate an irritating bug. And I think I may not even end up with a... I expected that to be green. I may not even end up with a test up here at this level, but maybe I will. So I searched for foo, and I didn't get var. MariaDB does index three-letter words, right? Explain it in here, right? So if I search for foo and stories, Yes, I get a ton. If I search for foe, I get the note that yes, terms must be at least three characters long. So the actual bug I saw was, I get the bug. Well, I know what it is. There was the XZ hack, right? Where they backdoored it. I tried searching for title XZ to find that story. And it ignored that. And then this, because the word was ignored, just became that. So like a search for the word title. That's why I was finding nonsense.

01:37:27So I do want a smoke test for searching for stories. So something is wrong with the test I wanted to write. I was trying to exercise this bug. So if I said XCA, I think that's a format. Title foo. Right. This works. I thought this parsing was broken, but no, it's if you are searching for something with a non-search term. Like this is wrong twice. It should still have recognized that this was a title and it should not have put the colon onto this. So this is a parser level error. All right, so let's do two specs, I guess. One to fix whatever the hell is wrong with this spec.

01:38:31Which is... I almost don't want to put time into this because I know this is going to be some fiddly test setup thing because it works in prod. And I just demonstrated it works in prod by searching for foo. But if we ever want to write search specs, we're going to run into this. And I might as well fix it now while I'm mad at it.

01:39:17twitchtd I need to step away but will read your comments afterward
yeah that's finding zero stories and it did parse the search correctly and search for foo yeah I'll leave you any more comments, but I don't think we're going to have anything else on SQLite outside of that email thread where we're sorting out the perk. Thanks, Thomas.

01:40:23twitchtd Thanks and have a good day!
okay does creating the object in the factory create its associated record that is the actual thing we get searched how am i going to see a single one or a zero in all this There's a zero. Okay, so the associated record did not get graded. So this isn't something about the test harness, this is something about the record creation. Which is one of those things, story text, that I thought got created in after save hook. After save update story text. Return unless saved change to attribute. I don't know if filling in. Oh yeah, story text is created by the cron job. Nogweii factory time?
Yeah, I don't think saved change to attribute is necessarily true. So let's in a new record. So s equals maybe. I'm thinking of like this thing is written. So story.create with the title bar and the tags tag last. And the submitter. I think that's enough to have a valid story. Oh, it needs to have a URL or a description. So let's say URL is sample.com. OK. So that did not create a story text, did it? Story underscore t. Right. It tried to do a select.

01:42:54Which is a little hard.

01:43:04Where's the.

...17Yeah, so it updates the cached columns and then the story text. So this is the cached columns. And then this is the story texts. So that was that update trying to fire. So that answers the question of... This should probably be like find or create story text. Let's go make sure that the... Background job doesn't assume it's creating story text. I think it does a find or create.

01:44:01Okay, who does it? Story text, Phil Cash does it.

...22We don't want to replace that because we use its presence to implicitly confirm that we called diffbot. And if we check any of these attributes, well, diffbot is capable of coming back and telling us no. You know, nil either because it has an exception or because it wasn't able to scrape it or anything. And so we don't track that any differently.

01:45:28I'm just going to do the easy thing here.

...46What was it called? Story text. Fill cache for bar story. And then watch. Now the tests are going to fail because we don't delete it, right? I'm just making work, digging the hole and filling it in. And then while I'm in the hole, I'm digging myself a slightly smaller secondary hole that I have to fill in. I guess that's OK.

01:46:55So let's go over to the tests on the wrong side. I almost always put tests on the right. Nogweii curious, does the test harness prevent the scrape of a page? are you making an HTTP request with that fill_cache! call?
So search parser spec and the search parser itself. Does the test harness prevent the scrape of a page? Yes, I think so. I don't think we're fired off a response. Because there isn't a diff bot API key here in prod. And so just return or here in dev, so it just returns. That could be better as an explicit test. So if there isn't one or rails.env.local, which then it means I can't really test this. And I do run stuff from the console. Nogweii cool. just wanted to make sure there wasn't a surprise external dependency. :)
So I'm going to just say test. It's a good belt and suspenders question. Thank you.

01:48:26Yeah, I do try and avoid hitting external APIs in test suites for all the usual reasons. All right, so there's the term rule, the short word rule. But what we want to say is

...53aashvikt Sorry to interrupt while you're working, but have you had a glance at #1926 (sibling comment dotted line visibility) yet? Just wondering :]
building it up, which is all down here, bugs I've seen in prod. And yeah, so here's parses a short word and a term. Oh, Ashvik, thank you for dropping in. You know, I haven't reviewed it. And when I was, I saw it in my email, but somehow I just missed it when I was reviewing stuff here. Probably because I was looking at this instead of the open ones. Thanks for dropping in. And actually, this one I haven't looked at either. And Anubis could go. All right, let's put Anubis back here. I'm done with this checklist. Done with that. Done with that. All right.

01:49:59Nogweii @aashvikt wondering if you have a screenshot of what that looks like?
Can you... add screenshots to this PR. Because yeah, I also cannot immediately picture it. aashvikt Sure! On it
And I do have the question of what this calc is doing. aashvikt Yep, centering
Oh, you're trying to center it, aren't you? Okay, so that makes sense.

01:50:37Might as well see the green build on this, but... Now what's this one? The existing test is wrong? Oh?

...54Speaking of the hassle of before and after... Did I get it wrong here? It's already been changed in the last year. Yes, this does look like I inverted it. Yeah, this is searching for changes older than a year for created ads.

01:51:34I must have written this code on stream. Speaking of being 25% less smart, to flip this condition and get it wrong in all of the specs is a pretty special bug. OK, so at least I got the title, right? But that must be where I swapped. So you created Alice three years ago. We rename Alice two years ago. There is no more Alice. Then we create really cool Alice. And we say, mm-hmm.

01:52:37So what did I have before, before I had rename Alice six months ago and expect her to be, wait, so I had create Alice two years ago, become cool Alice six months ago. No more Alice. Valid.

01:53:18This test looks correct, like the old version of it looks correct. Let's pull both of these into the to do list so I can get that off my brain.

...38And then the other one was comment sibling line. How did I get the same number? That's not right. That one is 26. This one is 25.

01:54:09So this old version said, create Alice. Rename her to Cool Alice six months ago. which should make any change invalid.

...53I'm not seeing what's wrong with this test. Is somebody else spotting it? There's a lot of updating various caches, but so we have Alice. If Alice changed her username six months ago, that should go fine. And then if we try to change again, we get this error.

01:55:43Wait, what? The sibling comments. Oh, this was, I'm already thrown on the username stuff. This was, Auschwitz. aashvikt added screenshots
If you pull in Maine, this test is not your failure. This is a thing that was broken on Maine. You don't have to pull in Maine. It'll be fine when you merge.

01:56:25aashvikt sorry, didn't catch that?
Nogweii @aashvikt merge main into your branch plz
Ashvik, the build is red here on your pull request. This is not your fault. This is not your problem. You can ignore it. This is something I broke on main. You can merge main in or you can ignore it.

...48And can you like just either in a comment here in chat or on the PR, you say it was close to invisible on some devices and browsers. And this is the first time I think I've heard that reported for quite a long time. Can you say which devices and browsers you had trouble on?

01:57:42aashvikt yep, librewolf (firefox) on 1080p 23' and 14' screens
aashvikt if that helps
So this somehow, but there's Twitch compression. So this screen you're looking at right now is, yeah, well, they are pretty darn invisible in dark mode here on mine. Nogweii @aashvikt just confirming, linux/x11?
My stream is 1080p, but then my hesitation about that is then it goes through video compression. So who knows what's going to show up on stream. They are pretty close to invisible, though.

01:58:26What was the comments subtree?

...41No, it's not. It doesn't actually have siblings.

01:59:05How am I not finding this rule?

...21aashvikt maybe go into "Style Editor" and look for 1px "dotted var(--color" ?
I just wanted to pull it up here on screen style. I didn't know that makes sense. You know, I never use this thing.

...38It's searching just file names.

02:00:03Really, you're going to go away as soon as I hit. I mean, that's certainly easier to see in dark mode. And I know it's off center because I didn't copy in the calc line.

...24I guess it is more visible without becoming so visible that it's overwhelming the comment.

...43Okay, let's paint that bike shed. I think this is good.

02:01:55aashvikt thanks for merging!
So of course, every change is going to be a little distractingly visible for a couple of days till people get used to it. So we may see an immediate complaint, but that's kind of normal. Well, thanks for fixing. So kind of popping the stack to what I was looking at. What was wrong with that spec?

02:02:28Because this one says, if Alice renamed six months ago, she's not allowed to rename. OK.

...59And you know, actually, she's created two years ago because she can't rename in the first year. Okay, so they changed this two to three because they want to have a second rename. So this is, if Alice created three years ago, changed two years ago, that's fine. no error yeah i guess i'm really just distracted by how much state we're building off and then if you alice renamed this is the original test again of alice renamed six months ago so you're not allowed to So there was nothing wrong in the original version of this. I'm just confused by the fact that they doubled up the specs.

02:04:26Nogweii I guess it was a happy path check to make sure they didn't break functionality?
Yeah.

02:06:54Beautiful.

02:07:24which is a design consideration noted on slash settings for the username changes.

02:08:07So if this was backward, why did this test not catch it? I don't feel I understand it. Alice changed six months ago. She's not allowed. But instead, instead, the test was detecting that there was a username from more than, from two years ago, because I was, because it's testing, I get it, okay.

02:10:50All right. So there's that. All right. I was working on the search parser that parses a operator and a broad word.

02:11:48So this becomes, honestly, I don't know that you can search for title colon nothing.

02:12:10Because that doesn't make sense. So I know that the search is wrong, I just don't know what it should be. So if I said short word, XZ.

...41Right so because of the short word limitation we can't get anything out of this.

...53But if I guess I could change it to that. Because the alternative is it's changes to. Well, the way it's getting parsed right now is term is title and then short word is colon XE, which is like, that's just wrong.

02:13:22The other possibility would be let that be title.

...42Isn't there something about punctuation here? Term and quoted after operators they would fail to consume. Yeah.

02:14:07Yeah, I think the problem is

...15short word isn't defined up here i don't have a good answer for this maybe i should just file an issue and punt

02:15:07yeah i'm gonna file an issue and punt because it's not a this is me trying to get better about my use of time and anybody could fix the parser

...52Just do it and grab this.

02:16:28Before I file this, do we know that this is a limitation here in SQLite 2? Because if FTS5, it doesn't also ignore two character words, this behavior is going to change and there's not a lot of point in filing an issue about it.

02:17:08They call it a short word, is that my? No, it's my term.

...21Doesn't seem to mention stop words.

...46So the reason I'm kind of digging into that is I'm going to bring office hours to an early close today. Because there's a thing going on.

02:19:23Yeah, maybe.

02:20:19A possible knock. Well, if I wanted to, just for a technical reason, I'm taking a strike.

02:22:07Nice.

...44Bye.

02:23:14Oh, you know, it says it's punctuation. Yeah, let me leave that second part off. Fix one thing at a time. We can paint that bike shed later. Eh, it's not quite bike shed painting, because it's not just a matter of taste of if the error message is misleading or confusing. It's a bigger problem, but...

...43And let me grab.

...55I don't think I need to grab it. There's plenty of examples.

02:24:33All right. Peter finds a bug and makes it someone else's problem. Theme of the stream. Alrighty. So yeah, I am going to wrap this stream up here. And that is kind of protecting my time because I know I'm going to put a bunch of time into sqlite performance and then hopefully sqlite deployment this weekend not that thomas and i have really planned that yet but we got an email thread to do that on so if there's anything else that comes up just grab me before monday's stream i didn't have one on monday just to reclaim some time and because i was doing a personal thing expect to on monday right i'm not gonna attend this weekend or anything yeah busy weekend but nothing planned on monday so yes i expect the next stream will be monday at 2 p.m chicago time yeah god i have such a to-do list all right well

02:25:55I'm going to toss that work. Oh, I didn't. Let's restore spec. And let's put that back, because I just threw away a change I actually made. Yeah, I wanted split. So this one is the keeper.

02:26:29keepcoding4everr i just watch a talk with David Heinemeier Hansson and lexz fridman i love David also buyed the rails 8 book
This one is don't hit def block and test. keepcoding4everr sup everyone
Don't hit def block and test even if an API key is configured.

...58Hey, Keepcoding. You're here at the end of the stream. So you're actually watching the last two minutes, but I didn't know there was an interview. That's kind of weird. I didn't think Friedman was a programmer. This is the thing I'm throwing away. This is the thing I'm committing.

02:27:27keepcoding4everr https://www.youtube.com/watch?v…
And JJ abandoned. graefchen dhh and lex fridman are *interesting* people limesHmm
JJ is so nice for preventing me from shooting myself in the foot, because I absolutely did with that restore command. And in Git, pulling stuff out of the ref log is such a pain. It might not even have ended in the ref log. Yeah, interesting in boring ways, unfortunately. We'll see about that gleam, huh? All right. So on that slightly spicy note, I'm going to roll out and I will be back streaming on Monday at 2 p.m. Chicago time, the usual Monday stream. Otherwise, I will see you all around the site. Feel free to email if you need anything. Take care.