maybe GitHub should port to SQLite

02:58le_romain_ethique hi
pushcx Welcome to Lobsters office hours, ask questions anytime!
Oh, that's funny. That's exactly the thing. Hold on. So I didn't look at the homepage here. I usually read slash newest, so I didn't see that post. But jump over here. I did already have my scratch notes set up, and one of them was audit all queries for full table scans because I had an idea how to do that. I wonder what idea Aaron has. pushcx https://lobste.rs/s/ko1ji1/l
so yeah where is this the big the big theme of the day is sqlite so let's grab this link which i already have on scratch well big threat about this so thomas has been working for a while like I want to say he opened his first PR or left his first comment something like nine or 10 months ago to migrate us over to SQLite. We were ready to leave MariaDB because of, honestly, the idea started because MariaDB didn't have good support for common table expressions and it gained that right around the time we got started, I want to say, but then it didn't have support for recursive common table expressions, I believe, when we opened a like, hey, maybe we want to move to Postgres issue. And we kicked that around and it kind of sat and laid fallow in the issue tracker for years. I mean, literally like two or three years. Tooxx_96 hello hope u doing great
Four? And then someone picked it up and they took a run at it, but then they kind of lost interest and dropped that. Hey, Tukes. Yeah, actually. Really am. And then, yeah, like eight, nine months ago, Thomas picked this up and started doing the really hard work of not just porting everything and getting all the tests working and rewriting the couple of queries that use raw SQL, but doing a ton of performance testing and just there's so much Tooxx_96 crazy that now it s all in sqlite 🤯🤯🤯 simplicity wins in the end
software engineering that does not show up in the diff i i don't know how to put it better than that but he he put in a lot of hard work on this and then in february we tried to deploy it and we did not succeed and the full story is in here i'm not going to recount the whole thing because this is a great write-up but we ran into serious performance issues even in a read only mode. And we backed it up. We moved back to MariaDB, because, you know, in read only mode, there wasn't any data to migrate back, which was the thing we had been afraid of. So oh, man, score one for checklists. I think it's in here. Yeah. So this one, I asked him to link to the checklist as of that day in February. And you can kind of see we worked down the list and then like, oh, then we worked down the revert checklist. And I am very glad that we wrote a whole checklist before we tried the migration. And then we wrote a whole revert checklist before we tried the migration. And then we pulled some notes in too. theGeekPirate So happy to hear about the successful switch after the previous attempt
Because I don't know if you have done a big, pushcx https://bookshop.org/p/books/th…
rod lift like this before but checklists will save you i have mentioned the checklist manifesto a number of times on this stream and this book is mostly about checklists for repeatable tasks like we're gonna put a stent in someone because we're nurses in a hospital that's one of the core examples of the book and they do you know 10 a day or whatever that is And this, well, I would have said this is a one-off that we only do once. We did this twice because the first one didn't work. But having a checklist where we very deliberately wrote out all of our steps and we thought about the order and we thought about what could go wrong. You know, there's a phrase that writing is thinking. and it was absolutely the case that writing this improved our plan and it made sure that we were not mentally burdened trying to keep track of everything while we were doing the stressful things in prod so even though that day did not result in a successful migration it was a successful experience in that theGeekPirate I was worried everyone would give up
Prod was not left in a weird state. We didn't lose any comments. Nothing broke. We minimized the downtime we had. Like, it didn't succeed, but it failed in a successful way, right? theGeekPirate When I _knew_ SQLite could do it easily =b
In the same way that if you do a scientific experiment, sometimes you learn that, I don't know, the drug treatment doesn't do anything, but you've still learned something. So... yeah and the core thing there was we have one let's just yeah so we have this view of one story with its whole comments of threads this endpoint is at least 60% of site traffic. And when I say site traffic, I am counting just traffic that makes it to Rails. If you look at stuff that gets caught by the cache, I think it's even more lopsided, but like the full page cache is effectively free. So just counting stuff that makes it to Rails, this one view of showing a single story is more than 60% of site traffic. And getting the comments out in the tree, SQL entries are long time enemies. So we have a big query down here. So let's see if there's story threads, recent threads. Yeah, we want story threads. This is the recursive comment table expression that pulls comments out in order. And this confidence order and confidence order path that we build up. I've talked about it on a previous stream. So you could jump into the archive to find that. But it's a big performance hack to work around MariaDB's limitations for data types. This is taking an array and serializing it down to a bit string. And you know, like, I'm proud of the kludge. It works. I understand bitbashing enough that it really did work on the first try. But there aren't enough bits to maintain precision. And so we've had a couple of sibling comments that get sorted a little bit differently or swap children, which is very confusing. It's rare, but it happens. And it's just a big kludge because MariaDB doesn't have an array type that we can express what we want to express. And that is that comments are sorted by scores and their replies, their children are also sorted by scores, but their children's children are sorted first by their parenting and then by their score. And that's surprisingly tricky to express in SQL, because SQL is very, very row-oriented, and we are expressing a tree, or we are expressing a path through a tree, like an in-order traversal of a tree that's really cutting against the grain of SQL. So we had a big performance hack in there to make that work well with MariaDB. And as soon as we tried to migrate, hey, guess what blew up in prod? Because the performance hack we wrote for MariaDB was not performant in SQLite. And as soon as I say it, that's small, right? It sounds very obvious that like, oh, of course, the thing you clutched for one database and tuned for one database does not work in a different database that works on very different principles and data structures. Okay. Tooxx_96 so for new devs making theirs apps after you experience you confirm that going full sqlite is more than enough right?
And then... We did more performance testing. I say we, it was mostly Thomas and a developer for new devs making their apps. I would say starting with SQLite is generally a great idea. It really does depend on what kind of scale you have at the beginning. And the biggest problem most apps and most startups have is that they don't have any users. And so they really don't have any scale problems because they don't have any users and they could store data in CSVs and be fine. I have seen real production apps that store data in CSV and they were fine. But you don't want to set yourself up for disaster. And so if you're going to grow, theGeekPirate Yeah, Tailscale was using raw .json files for a while, then went to SQLite, and grew from that after thousands of users
you need to choose something that meets not just your needs today, but your likely needs. And like, we all want to think that our app is going to get a million users and our startup is going to make a billion dollars and we're going to ride luxury yachts around itself. But like, that's probably not going to happen. Tooxx_96 does sqlite works now after all the efforts of tweaking that rails 8 did right ? especially writes ?
Yeah, I tried to like nod this direction by talking about like, I don't want to use solutions that are bigger or more complex than our likely needs. And one of the Funny things about running an all text website is. Especially given that we're not trying to scale. Lobsters is growing slower. Then hardware is getting bigger and faster. So if we grow, I don't have a good number off the top of my head, but if we grow 20% for a year. theGeekPirate NVMe are just so silly fast
But hard drives get 50% bigger per year. And again, I'm making that number up. We sort of have negative growth if you are tracking growth in dollars. I said a version of this a couple of years ago in a comment, but our hosting cost is going down over time, even though we have been steadily growing for years, just because Computers are getting cheaper. theGeekPirate Yep, I can run a stupid amount of useful services on a $5 box these days
The rest of the world's scale is getting bigger, much faster than we are growing. So yeah, SQLite really does just work. We're going to do a little bit of performance tuning here and there. It's stuff that there are like three queries that SQLite has guessed the wrong index on. Did I even see three? We'll probably find more today. And then there's my performance hack for MariaDB that SQLite didn't work like. And to give you a sense of how much traffic we're serving, let's go, we can look at product actually.

15:48Oh, yeah. This is nice. The load average and such is down. There must be a bot scraping us for it to be a bit too. So let's look at one of the latest full days. So let's grab you. That's 712. Yeah. So let's throw that log. You don't have Al? Okay, so the 13th was a Monday. So like all social sites, site traffic is highest on Mondays in our local time zone. And then it falls over the week. And then it falls kind of steeply between Friday and Saturday. That's why we did the migration on the weekend. Saturday and Sunday is something like half of Monday traffic. So I did all that and I didn't. Monday was the 13th. So let's grab that log and grep for 1.26.07.13 and then just count. So this is ignoring the full page cache, which most apps aren't going to have. Is the date formatted differently? All right, let me drop this off screen because it's going to have somebody's PII on it, like a page or a comment. timestamp is the log file is named for when it was rotated not for the week it started all right so let me clear that I should have remembered all right so this one if we want to see this Monday's traffic we look at our current week's log of course so on Monday rails served 600 000 pages And if I looked at caddy it probably served 10 or 20 times that in hits that does include things like CSS avatars, so there are many hits for one page, but it also includes just a huge number of full pages. Tooxx_96 🤯 i guess sqlite won 😅
So we're serving a pretty substantial amount of traffic. And yeah the. fixes and improvements that were made in Rails for SQLite, especially by Stephen Margime, who had a couple of nice talks about this over the last year or two. That works really well.

18:41Tooxx_96 do you use sqlite for caching and qeue with solid ?
yeah sequel like just it's the right size for our needs, even though we're serving a pretty substantial amount of traffic, we have lower rights than say a SAS APP because most people aren't doing interactive things and. genuinely I was so surprised by this

19:11So this spike here is when we did the migration. This spike here is when I did a cache cleanup. And this spike here is some bot. So if you follow up from this, here's where we did the migration. And you can see when we restart the servers, RAM usage drops. But then we entered a steady state where RAM usage is noticeably lower than it was before. And CPU usage is noticeably lower than it was before. I did not predict those. Yeah, too, because we also use SQLite for two kinds of caching and a solid job queue. We have, where are we here? Actually, yeah, well, it's a database. So we have a queue, that's solid queue, like you're thinking. Rack attack is where we have split out our rate limiting software into its own SQLite database, because it has really different access and aging characteristics than our primary cache. Where was that? I just hacked that out on Sunday, so I don't remember the commit off the top of my head. But this one was, pushcx https://github.com/lobsters/lob…
feels like it was part of those improvements. Here we go. This one was fun. It actually worked really differently than I thought. I thought I was gonna just add a second type of database, but then I also had to add this idea of sharding the cache because Rails thinks of solid cache as a single, twitchtd pushcx: hi
theGeekPirate There's no communication with a separate process so everything's a _lot_ faster, especially with gigantic queries you normally wouldn't do
theGeekPirate If you've never seen it before, worth a read https://sqlite.org/np1queryprob…
entity rather than there are multiple of them and that's a little bit weird and a little bit sus but i could do the thing i want with the sharding concept so i'm a happy camper hey thomas i have been talking about your work for like half an hour i was saying the thing you might not have heard is that i said that there is a lot of engineering lift that goes into a project like this that does not show up in a diff, and I thanked you for it. twitchtd :)
Yeah, Geek Pirate, my best guess was, you know, we're going to hit that, this N plus one. My best guess is because we make good use of indexes,

21:58The added CPU of doing the query on the web app server is lower than the cost of CPU and RAM to serialize to and from the MariaDB wire protocol. And that is my best guess for why our CPU RAM and load went down. But that's not something I can like authoritatively state because where'd that chart go?

22:34espartapalma good morning
This chart is pretty much the extent of our observability tool. Because we're not a fancy professional website where every minute of downtime costs us $10,000. So the site, actually, when we did the deploy on Saturday, we were in read-only mode for, hang on, I got to clear my throat.

23:11Pardon. So we were in read-only mode for, man, where's that coming from? Hold on.

...23Boy. So read-only mode for three or four hours. And then we did have, I don't know, it felt like about a half an hour of hard downtime. And while we were in read only mode I put a banner up on the homepage actually across every page that said hey we're in read only mode join the chat room but we're migrating here's a couple of links, you can watch their checklist if you want. And just like in February, the the chat room kind of turned into a big party everybody was having a good time everybody had really healthy expectations about. A partial outage and then a full outage of a free service and. Generally just, you know, chilled, had fun with it. People even kibitzed a bit and gave some interesting suggestions for sorting out stuff we saw. So yeah, here was the cash store. If you do use SQLite and Prod and you use Rack Attack, I would really recommend doing this. This made the box happier too, I believe. And then, oh yeah, this thing. So this is where we wire up the store. And then this is just, don't try and do that in test and dev. That cost me a minute.

24:58theGeekPirate ^_^
Oh, he just posted this. Oh, this is a response from... I thought it was funny that I saw Aaron... writing about the exact thing I was thinking about doing here on stream, and he is actually responding to us rather than it's a coincidence. How fun. Aaron, if anybody hasn't heard the story. I mean, what was this two years ago, two and a half years ago, we had an issue where we were having something that looked very much like a leak in Ruby when we had ZJIT enabled. And again, because we don't have any observability, we were really puzzled by this. And he helped me debug that problem. And we worked out that it was memory fragmentation. And we switched on gemalloc, J-E-M-A-L-L-O-C. I believe it's maintained by Facebook. And poof, that problem disappeared. So that was a neat one.

26:15To be prepared.

...29This is a very different strategy. than I had thought of.

...54twitchtd ya I thought you would have to run a separate explain statement for each sql statement, apparently not
I wonder if I could get to this statement object from active record. Yeah, Thomas, that is exactly what I was going to do. I was going to run through all of the scopes on our models. And call dot to sequel on them and then i was going to fight and lose a bunch of time to filling in parameters like you know you need a an id or a number or a story id and then once you've got that explained query plan i could just literally skim it for the word scan

27:49See the comments. All right, well, we grab this and I'm going to come back to this. My personal browser, hopefully when there's. twitchtd I believe rails instrumentation exposes the statement for sql.activerecord or whatever the name of the event was that I hooked into with slow query logs
More comments.

28:21that's that's clever i knew about this hmm all right well let's so let's grab this url and honestly i kind of want to hunt on it because it feels like if i leave it a week or so someone will write the thing aaron wants and it'll be tie it into Rails rather than me having to do it.

30:06twitchtd not sure if you saw but simonw also wrote a blog post :) https://simonwillison.net/2026/…
oh man simon always writes a blog post this guy is a prolific author so he's simon w on the site if anybody doesn't notice his recognizes username immediately he asked a couple of times how big we were ah really curious about these

...38Huh, neat. Well, let's include that. I'll just stick it in here somewhere.

...53It is a little funny to run a site that, like the famous programmers I know, are like, oh, look at what they accomplished. It feels very much like putting on a show in the barn rather than like, Oh, famous programmers. And I say it this way because I said it in my comment here, but like, one of the reasons we went to SQLite is that Thomas chose to use SQLite. And, you know, I believe Thomas's reasoning was number two. Tooxx_96 can you please share about search does it work great in sqlite
I don't mean to put words in his mouth, but there's an old political slogan that history gets made by the people who show up. Like if you just do stuff. Tukes, there's a bunch in here about it. All right, FTS. Oh no, I guess it hasn't come up. It was in that first pull request, wasn't it?

32:04Yeah. How am I supposed to know? Now I just have to scroll up and down looking for those load more buttons. Yeah. And then they don't have a consistent text, so I can't just control F. Now I got to scroll and find the other button. Oh, sometimes so exasperating. pushcx https://github.com/lobsters/lob…
All right. So here we go. Here is a comment where Thomas started talking about it, but The short version is, yes, the full text just works great. Our search engine, if anything, is a little bit faster than it was. I want to say we had more about it. Is there any issue?

33:11Tooxx_96 great thanks
And again, no, load more. See, like the PR says hidden and this one says remaining items so that you can't ever get used to finding these fucking buttons.

...29All right, now can I search for FTS? There we go. Yes. So here's Thomas's comment where... Oh, no, Rahul worked this out. I'm sorry to misattribute.

...49pushcx https://github.com/lobsters/lob…
Yeah. All right, so here's the other one for you. Tooxx_96 thnks
I forgot that Rahul did the lifting on this. This is the comment I was trying to remember that had a bunch of basically planning for it. So... twitchtd ya rahul was the one that proposed sqlite, I just ran with it :)
Yeah, the one thing we had to do was make this trigger so that our full text index would automatically get updated as comments get edited and deleted and added. And after we deployed, I did like immediately run to the search engine and I searched for a comment that had been posted since, you know, a word in a comment that had been posted since we migrated just to make sure that those indexes were updating and it's been working great. Yeah. So we talked about. I'll grab these two. Get these two links just to leave in the. Archive notes for the show. Show stream. Not putting on a shift. All right. Oh, yeah. I jumped right into SQLite, but we do want to come back to that.

35:25So, yeah, everybody's anecdotal experience, including in the chat room just as I was getting started, is that the site feels snappier. This was part of the curve. So this is part of, kind of goes with Tenderlove. But Ida Marst, who writes a lot about performance in Python, kind of gave a high level view to SQLite has this internal bytecode. It's sort of a assembly language that they run in a VM. Do they have an example of it here? Yeah. And so this, you know, you have a very simple query like delete from table one where the column two is less than 20, sure. And then you can, this is not something I really think is worth learning to read unless you are going to contribute back to SQLite. But you can just, Lance at it and immediately see the difference between this and a giant comment with or a giant query with lots of joins and predicates will be like a full screen long sometimes. And just being able to eyeball that is useful.

37:07Yeah. So this, the n plus one query, is exactly where I was going with removing confidence order path, which is something I would like to experiment with on stream here. So this, we already did this. That can be up there. Now we're down here. We can also look at the swallow query log. I'm going to change fears for two seconds and do mock turtle because it's small. And I don't want to forget, and I know like as soon as I open the slow query log there goes an hour of puzzling through slow queries and soon as I start removing confidence order path there goes another hour so. let's let's do mock turtle first.

38:00So for anybody who doesn't recognize the name, we have an irc channel or bought is called mock turtle. which is a Alice in Wonderland reference. And I don't honestly know why somebody chose that a few years ago, but we've maintained the name.

...23And one of the features I wanted for a while was for the bot to look for the word invite or invitation from someone who had joined the channel for the first time and said those words in their first 10 minutes in the channel. And Nick Bores took a run at this and it's been a little bit buggy. It's sort of working, but it's also saying the message a bunch. So let's double check this. What's changing? OK, so schedule tasks.

39:12OK, I didn't lose that. Come here. Why space? This one. Speaking of GitHub, what is this eye for? The eye doesn't do anything. If I click on it, if I hover, it doesn't give me any info. I don't know what minimize comments means. GabuTheDev maybe keybind?
Like if there's a 10 line comment block, is it going to collapse it? espartapalma it hides the comments
I don't know. I've been seeing that button for a week or two now. All right. Maybe key bind. espartapalma it's a keybind
Oh, I guess it could be a key bind. All right. GabuTheDev open the menu and press i
GabuTheDev see if it toggles it
Now nothing happens if I type by. On the other hand, I'm not sure there's a comment. Well, there's one comment, but it's a one-liner. espartapalma go to a PR with comments
Alright. GabuTheDev then idk
Oh, open the menu and press I? Nope. Hmm. hoistbypetard next up: lobsters -> forgejo? ;-) (SCNR)
I thought it was I like an info sign, you know? espartapalma it's an i?...
I guess computers mostly use the question mark as their iconography for help. hoistbypetard sorry could not resist
What is SCNR?

40:39Ah, yeah, yeah. GabuTheDev did you know you can press G+i for opening issues page and G+P for pull requests?
Well, Hoist by Petard, if you, using our new SQLite full text search, if you look for comments by me that mention GitHub, there have been a couple in the last year about my rising frustration and what I would like to see out of espartapalma you type the letter i, it will hide the comments in a PR
A replacement forge. And one of the things. Is.

41:13Now. I would like an export. And it would be really nice to have bidirectional sync. But I would really like at least an export because. Yeah, for the first time. maybe Sunday I was here on a PR page and I started writing a comment and I wrote like two paragraphs and then I clicked over to the files changed and I went okay and I clicked back and then my comment was gone and I had to type those two paragraphs over from scratch and I was super mad about that it's really been frustrating this last year and a half watching GitHub get jankier and jankier. And like, I sort of knew it was coming because you know what direction that's going, right? GabuTheDev The only new cool stuff is the repo dropdown. That's kind of it.
Like if the whole thing is falling apart and the CTO says the only things we care about are co-pilot and migrating to Azure and our response to all of this unreliability is that we're going to co-pilot and migrate to Azure even harder. GabuTheDev Think of the uptime problems earlier this year
It's not getting any better.

42:42Look, I forgot that I was hiding white space. GitHub. All right, so. Oh, I think it was responding to itself. Yeah, that could have been why it's repeating its message.

43:11Wait. Last joined.

...36GabuTheDev oh that
GabuTheDev I didn't know that was happening
yeah yeah the uptime the uptime problems earlier this year were i don't have any special info but i that's the kind of thing you would expect out of a data center migration like has been going on for a while but i i can't guess they could have been coincidental

44:06Oh yeah, no, the CTO, I left an angry comment on HN about this. Where was that?

...27pushcx https://news.ycombinator.com/it…
I am quoting someone who is a random GitHub developer, but I wanted this link. So last year in October, the CTO announced to employees, like, our big priority is migrating all this infrastructure to Azure, even though this means it'll have to delay future development. And it clearly also delayed some maintenance. And they talk about it more, that this is the result of a memo. And GabuTheDev I just wished there would be more alternatives to github. I feel like MS is kind of killing the whole git ecosystem
After Ghost TTY left, some people talked about all the outages. And I don't know, did folks see this? Like it's from that CTO. And it's like, oh, look, we're so busy. We're so busy. We have so much going on that we can't even label the y-axis on these charts. So chart go up. theGeekPirate They did say exponential growth
There's nowhere to know if this is a 1% increase or a 50% increase because there's no y-axis. We don't know where this started. So anyways, there's a book called How to Lie with Charts that's about making these things. Yeah, but people use the word exponential metaphorically. And most of the time they say exponential, they mean quadratic. So anyways, it's like, oh, we did a bunch of stuff. We did a bunch of stuff. then, how was it?

46:13We are committed to improving availability, increasing resilience, and scaling for the future of software development. How did I quote? Yeah. Executing. Yeah. We started executing on our plan to increase GitHub's capacity 10x in October 2025. No. No, they didn't. Like, that's the other link. davidofterra Your call is very important to us.
They prioritized moving to Azure in October 25. And this doesn't have a full quote of the memo, but this like 10x thing does not appear in the public article. So when he says our plan to increase GitHub's capacity, he's just saying we started executing our plan to migrate to Azure.

47:03And so their plan is to migrate to Azure harder. That's not going to fix all of this SPA jank. It's not going to fix, you know, losing my pull request review.

...22twitchtd the migration to azure will continue until morale improves
I'm sure shit isn't fixing it. I'm getting mean. Did I eat this morning? yeah migration to azure will continue until morale improves yeah that's i really have to wonder on that one

48:09Don't like this structure.

49:19GabuTheDev I'm new here. What are you working on?
pushcx https://lobste.rs/
pushcx https://lobste.rs/chat
espartapalma hello, can you read me?
hey gab with the dev the website is lobsters and specifically i'm leaving a quick full request review for the irc bot that's in our chat room so if you go to i thought it was in the footer but if you go to lobster chat you can see about the the main chat room we're old it's irc

...54Yes. Hello, Espart Palma.

50:10wait, but last joined. espartapalma I was telling about the keybind, to _hide_ the comments, you can type "i" now
Okay.

...31Okay, I don't I did see your message and I did try it, but this PR that I have up doesn't have any multiline comments. And it doesn't seem to be toggling the checkmark on that. Well, if I focus out and I type I. Yeah, well, we jumped. What is it? Oh, it's toggling the comments, not the code comments. espartapalma that's it
OK. theGeekPirate MingLee
So this thing is about the code diff, and this is about the code diff, and this is about the code diff, but this one is about the comments that are injected in the middle. And so, instead of being about the comments that are in the diff, obviously this one is about... I'm getting mean. Let's change gears.

51:37GabuTheDev Amazing UX
shit i know that's what it's doing yeah this combination of don't record a join for the bot itself and then if you don't see their timestamp print the message would be that if the bot says invite in its first 10 minutes in the channel it's going to spam the shit out of itself

52:23But then I'm kind of setting up the reverse situation where if it's seen the joined, yeah, all right, we'll leave it.

53:00Where's my IMC channel?

...09What channel was I in?

...20Just checking on something off stream.

...36Being able to toggle comments off, that is a nice feature, but that was a really confusing presentation of the UI. All right, so there was mock turtle. So Thomas, if you're still here, what's more interesting to you, trying to remove query order path or poking around in the slow query log. I'm happy to do whichever is more interesting to you. I should also skim RPRs because I think a couple are open.

54:25He updated this. And this. Maybe Thomas isn't here anymore. I know he does have a grown-up job. Let's take a look. Okay, nothing there. Let's see. Could he have force pushed? No. All right.

55:12twitchtd I think taking a quick peek at slow queries might be a good idea first to see where the issues are
Then this didn't get updated now. OK. Oh, so Zertar was in the chat room a few minutes ago. Sure. Let me just. I think this is probably going to be ready to go.

...50zurtar I believe the last change was moving the tests.
yeah and if the build is green i'm happy great yeah thanks for working through those tweaks

56:09Thank you.

...41twitchtd Do the slow query logs hide the values of the binds or do they show the values? Might want to look at the logs offscreen to prevent seeing any prod data on screen
didn't grab this into the notes oh they show the value of binds yeah that's going to be a little bit inconvenient but i can bring stuff on and honestly it's really only like

57:13You know, I try not to show user IDs on screen because I don't want to make a habit of it and start off that like really tedious leaderboard thing of like, oh, I have a lower user ID than you, therefore I'm more authoritative. That was so bad on Slashdot back in the day. I really don't want to start that. But like, if I show a user ID, fine. If I show a user ID and a vote and a story id we kind of consider votes and flags private so not from mods but you know i don't want to publish that and then the real concerning thing would be if i pulled it up and there was the contents of somebody's private message in there like that's a little gross all right so this is all of them right yeah

58:24Thanks, sir. zurtar Slightly off topic, but the comments by durierem on #2141 could be interesting to go over if there is time.
And as long as we're going to go look at the query, I'm not going to deploy this this two seconds. 2141.

...45pushcx https://github.com/lobsters/lob…
Yes. Oh, Thomas, I think I pinged you into this one, but if you didn't happen to see it, We got, man, this is the voice of experience speaking. It's someone who has compiled their own binary for SQLite to be able to enable the Unicode case folding. And... twitchtd I didn't see this
This is a little bit overkill for our needs because usernames are the thing that we care about, and those are ASCII. Oh, okay. Well, definitely scroll these. They, he, she, he. He has clearly thought a lot about what automating this looks like, and... I kind of go back and forth on whether this is useful for us. And it is a shame that it's not in the default compile for

01:00:02Let's double check. Actually, I'm running Arch here. What if Ubuntu LTS compiles in more stuff, right? Not dot version. What's the dot vars? Clearly, my SQLite is not as fresh as my memory to be.

...40pragmat compile options okay and then this one was called

01:01:10I see you. Enable, I see you. All right. So I don't even see it here as false.

...27And string ICU does not appear. So no, we didn't get lucky.

...38twitchtd what os does prod run?
Shame.

...44We're on Ubuntu LTS.

...5724.04. Wait, they do these every 18 months. We're one behind, huh?

01:02:18Every two years. So there was just one in March.

...30April. Okay. Well, I should probably do that upgrade at some point. Not gonna do that one live on stream. All right.

...59graefchen heya limesHi
Oh, good thing you asked, huh?

01:03:10Oh, hey, Gravechen. All right, so, ah, you figured out what arguments the formatter wants. I sure did not figure out what arguments the formatter wanted. Duration, SQL.

...27And that'll automatically have a timestamp on it, Thomas? Because remember, that was missing. twitchtd looks like the formatter adds the timestamp
And then the SQL has those little annotations with the controller in action, so that's adequate. zurtar Its added in the formatter above I believe
All right.

01:04:10twitchtd Zurtar wrote this, I didn't write this one so they would know :)
Oh, I'm sorry, I didn't catch it. Zertar, sorry to ignore or misattribute your work. Yeah, thank you. I apologize. Okay, so timestamp. What is severity in this context? Is that like info, warn? These are all going to be warns, right?

...42And then we're merging it with the message. Throwing it to Jason, adding in the end. OK.

01:05:11All right. So I'm going to, I'm going to hang on to this for after the swear slow query log, but that looks good to go. Thank you, Ethan. But as long as I'm looking at the slow query log, I don't want it like changing formats in this five minutes. Although Jason would be so much nicer. So one of the hassles with the slow query log is. It prints the raw SQL statement with new lines included. And so when it prints a slow query, it might say, like, select blah from blah, where blah. And that thing can go on for a while. So if I want to grep for something like the slowest queries, I end up only seeing the first line of the queries. And if I want to grep to find all instances of a query, I can't then get the times of those queries because I'm grabbing some random line from the middle of the queries. So one more reason disk logs should just all be JSON by default. All right. So I'm going to. That's weird. Opening a terminal rearranged.

01:06:47So our slow query log is 96 megs, which is not nothing. And we've turned up the timeout on it to be, I believe, 1,000 milliseconds, one second, just with the thinking that we would Get the worst of the worst and then lower the threshold for slow query logging over the next few weeks and months. So I have it up on screen and this one yeah this one, I can definitely bring over. Because it's not going to be a surprise. let's just. grab you.

01:07:41so and i know thomas instantly knows which query this is but for anybody else this is the user threads query it is basically oh this is two of them so this one is on comments create that's interesting you know people were all right so we got two queries hold on let's do one at a time This bottom one is the user threads comment or query. And it is the same structure because it's getting a tree of comments. It's the same structure as the single query view that I was talking about was the source of so much trouble before and is the thing I would like to try and clean up. So what I'm curious is if I looked at the query log for mentions of confidence order path, How many mentions are there, and it is worth noting that it appears three times in the query so i'm going to take this number and divide by three.

01:09:05So that's 11,000 where's my calculator.

...16that one one one seventy six divided by three is thirty seven hundred and twenty five instances of that in what is that we deployed Saturday and call it noon coming up on four days so we're seeing about a thousand of these a day are slow that's substantial and again that is hands down the most popular query on the entire site. So that's why it's showing up so much. And some Rails apps do not consider a 1300-millimeter second query to be a problem, but I sure do.

01:10:31so one more reason to look at that one this one we've had one person on the migration thread mentioned that it felt like comment posting was slowing was slow and i have seen like an eight second pause to post a comment so it is real interesting to me that

01:11:15is this hang on there's a weird post and if it's actually abusive i gotta pull it oh this site

01:12:27twitchtd huh I didn't expect that query to run on comments_controller#create, I can't repro locally
Why is creating a comment selecting all the parents? yeah yeah i'm puzzled by this one too so let's go let's go look at that comments create there must be a reference so there's Right, so there's a bunch of stuff that's related to the parent comment, which is to say the one you are replying to.

01:13:34widberg widberBluecat
Howdy, Whitberg. Whitberg blue cat. I don't know what that emoji means. It's cute, though. so if it's valid we touch the story the parent common association will already touch the updated at columns up the reply chain yeah and we only

01:14:14all right so so something is calling parent comments plural right what is that normal comment so this one is called parent singular can't be that many instances of it getting called in the code right aha it's on oh no that's a variable right So we pass in the parents. If they're available. No.

...59The email reply mailer.

01:15:08Does that happen inline?

...34no the notify comment job is off in the background so that's while it's using it the parents query that can't be the source of this slow query comments controller renders a bunch of stuff

01:16:12Comment parents. Could this be comments create re-rendering the edit view?

...29Render comment box. No, just directly renders. It doesn't look like it. It's not. OK, so that's not it.

...51reply yeah so this does it but that's not the action we're looking at update again it's not create The parents controller. Oh, this almost never happens.

01:17:31The comment vote hydrator is using the word parents in a different context.

...42What is parent comment IDs? Ah, there it is. This is where it's going to come from. All right, so now we know where it comes from. And this is a very plausible comp. Let's not do that on prod. Very plausible candidate for why comment posting would be a little slow. So let's look at the query plan.

01:18:41Search comments using primary key recursive steps, scan parents, which is fine. That's a small collection. Start using primary key. Whoa, scan comments. Hey, speaking of full table scans, there's a slow query. Why is it doing a full table scan? And earlier, I talked about the bytecode. There's a bigger bytecode. And I want to see this query plan. So why? Why are you doing a table scan?

01:19:27Maybe it's doing a table scan for this comments.

...35As comments recursive. Comments ID. Yeah, bloom filter on parents. so sqlite is saying that when it goes to grab those comments by id It's efficiently generating the parents list, but then it's doing a full table scan of comments and then bloom filtering against parents. And realistically, parents is an array with, I mean, I would guess the median is zero or one elements in it. In most comment threads, it's not going to get beyond like five elements. And then we do have a hard cap at, well, for new comment threads, I think it's 16 or 17. And then one place in history, we have like 30. So a bloom filter is really overkill.

01:20:58So it's not making a great choice for this join. Let's grab these for the notes.

01:21:13So that was comments. Roller create calls. What was it?

...27breaks speed limit calls parents which yeah it does actually want does it need their ids ah yeah that's right it's looking for flags

01:22:25So this one. I don't actually know how to hint the sequel like query planner not to do this in most cases. This is going to be like two, three items. It's.

...50Maybe it's because we're joining on comments to ID, which has I mean, you know, the lowest possible cardinality, because every single comment has, or is it the highest possible cardinality? So rather than it being like, we're finding 10 comments that are the only 10 rows that have this Boolean set to true, every single comment has a unique ID, and they're evenly distributed because they're just synthetic primary key IDs. All right, so what if I threw a limit in there? Would that meaningfully change it? No. What if I said limit three? Yes. See that? That flipped it. Now it's doing scan parents and searching comments using row ID So that is going to be a hell of a lot more performant. And I bet if I look at the bytecode, it's shorter. I'm not going to try and read the bytecode. So that's 60 lines instead of 58. So it's one or two longer. But honestly, it's that. Which, yeah, of course, the thing that's hurting performance here is walking all 600,000 comments. not the processing it's doing at those comments i said limit 10 just a binary search yet now we're back to full table scan seven five oh parents six scan parents seven scan comments all right so there's like a break point there if we could hint it to always do the scan parents, or we could do the N plus one, right? So,

01:25:36twitchtd can you paste the query? I wanted to try some things out
pushcx SELECT "comments"."id" FROM "comments" inner join ( with recursive parents as ( select id target_id, id, parent_comment_id from comments where id = 699950 union all select parents.target_id, c.id, c.parent_comment_id from comments c join parents on parents.parent_comment_id = c.id ) select id from parents ) as comments_recursive on comments.id = comments_recursive.id ORDER BY "comments"."id" ASC
sure so one of the things and it was in this doc that someone linked here in the chat is sqlite doesn't have a one plus n problem and Pet peeve, it is a one plus N problem, not an N plus one problem.

01:26:26Yeah. So let's say comment.last.parents. The account right so there's. Our query getting executed that one has one all right let's. How do I find i'll just find. locally, I want to come up has a couple of parents ideally more than six.

01:27:06So where's someone with a big conversation? I might have deep comments. Yeah, here we go. So this is SQ5 APM. All right.

...34If I said c.parents.count, only four? All right, let's go in the more heated conversation then. Does this one go deep? No. How about this one? Yeah, there we go. twitchtd wait why are we joining outer comments?
Rust threads always have a bunch of back and forth. JC2UZE.

01:28:14because we actually want the comments. Oh, because we're using active record and we have to. It can't know that the recursive, that the comment table expression parents is comments. And so you can't load directly from a comment table expression like parents, you have to take that list of IDs and join it against comments. And maybe I'm out of date on that one, but I banged my head on that one for a while.

01:29:29Well, let's tinker and try it. So where's my Scratch? Because now I'm doing enough things. So we've been talking about slow comments posting. This got to knows on the dot parent query can try and plus one. Alright. could try sqli and fetching could try not joining our comments maybe mutation all right now that i'm not going to lose track so here we would just give it the join And here we would say, why is it target ID?

01:30:37Why do we have to have that at all?

...49I don't think we need that. Let's try dropping that load. Grab that comment. Grab its parents. OK, yeah, we didn't need that field. And then if we just change this to select star from comments, and this is also a star from comments, And then we want to select star from parents. What's this closing? Yeah, that's what the hassle is, is we have to say, like, join with. Otherwise, I've never been able to say, like, I don't think there's a with. What is it? Is there?

01:32:25Yeah, it wants a hash. What is this method doing?

...39Yeah, that's whatever this Ruby thing is. Or whatever this ActiveSupport thing is doing. It's not a related to with on.

...57With.

01:33:05You don't see anything about common table expressions.

...14No, not round. With recursive method.

...34the google ai said it was 7-1 so maybe this was a limitation that's gone away so with recursive posts and replies if we can express this better right so comment dot with recursive how do we want to put it hosts and replies, and then you're giving two queries and they become the union. Okay. So we'll say with parents is give me that comments ID.

01:34:30So give me comment that with id equals c dot parent id and then the second query let's close that is going to be or refer to it host joins and then i have to do a string okay So let's say comment joins join parents on comments id equals parents.id. Is that it?

01:35:42I missed a parenthesis. Undefined method, parent ID. Oh, it's parent comment ID.

01:36:01No, that got me every comment.

...09Let's start from the see that comment id it's not starting from null right right with recursive parents as comment id equals that union all grab the comments join parents on comment id no join on comments dot parent comment id is parent ids there i'm not that on the other side

01:37:12Right, so this, the base case is c.id, c.parent-comment-id. Right, so the base case is start from the parent and then union it by finding me the comment where The parent's parent com ID is the comment. But then it's doing a select count star from comments. Okay, so the query was correct and either version probably will work, but then it's throwing it away and it's doing a select star. Or I have to manually join to that? That's back into the...

01:38:27Yeah, so there it's recreating Thomas, that outer join.

...41vinitkme Hello, Hello!
twitchtd if we limit the query with: .where(thread_id: comment.thread_id) it might be better since all the parents should belong to the same thread_id, i think
And I say. vinitkme How are you doing?
Sarah from.

...48Haven't it? Oh, that's almost right. Hey, there we go. So then if I said, select star from parents, it's gross that I have to give this a little bit, but then it's getting back something that is comment shaped enough. And then if I take this over to SQLite, I probably don't have a full table scan. That's the select. twitchtd ah, your way looks much better
Yeah, it only scanned parents. Hey, doing real well. Parents will all belong to the same thread ID. heypushwhyisthesitebroke have you considered rewriting in rust or zig =p
Yes, that thread ID is kind of redundant, but. Yeah. OK, so let's. Well, Thomas, you got it. Oh, hello. Hey Push, why is the site broke? I feel like you need a rename because the site is less broken this week. But yes, we have actually seriously considered it and Rust solves a bunch of problems we don't have. Like we're not running out of RAM. We're not running out of CPU. We're not doing numerical computations. We are slinging around heypushwhyisthesitebroke but don't you want three hour compile times though
vinitkme https://github.com/vinitkumar/I… I forked inconsolata and added ligatures and a narrow version. https://vinitkumar.github.io/In… Does this looks interesting to you?
strings and formatting them and sticking them in the database something like ruster zig is just like one could but we wouldn't get much out of that that's a lot of fun let me finish my thought here and i'll check out that link Because obviously, I'm running in Consolata. So we will say this. And honestly, this is going to be fast enough. I don't feel like we need that little bit of memoization in there. So let's dump that. Let's say where parents this.

01:41:38okay where was my all right so let's reload and hold on let's find that comment again undefined Did I line wrap funny? Oh, because I just copied that straight instead of. So this should be just parent comments ID because we're running in the context of the instance. Yeah.

01:42:27I thought I was typoing things. Look at that. heypushwhyisthesitebroke can you do an sql query to see which post has the most tags on lobste.rs?
Oh, look at that. 0.8 milliseconds. What was it before?

...45twitchtd you switched to local db I think
heypushwhyisthesitebroke sure
heypushwhyisthesitebroke un moment
0.7 on the other hand this machine is not under load 0.5 huh yeah you want to write that query yeah this is local And if you only care about the number, you can just join against taggings or have a subquery rather than actually write it.

01:43:36Yeah, you know what? Whatever this says, it's not going to... Do a full table scan on comments. There is no way that can be slower.

...55Yeah, all right. These timings are for the explain, not for actually running the query.

01:44:06So if I ran the query here,

...16You know, it may actually be doing something clever because it's a count. Okay. Yeah. That's still only scan parents. And then if I go back up to six, where was it? Let's say eight.

...40Although actually we know there's nines. So let's say just like 10. Yeah, look at that. The real time, instead of being like a millisecond or four milliseconds, was 124 milliseconds. Yeah, I'm sorry. Instead of being half a millisecond. Sorry, hard to read all these little digits. Yeah, so that's a substantial improvement. All right. espartapalma amazing
So we're going to commit that. Because that counts as a fix.

01:45:26Let's remove that big comment. Thomas, I'm really glad you asked about that outer joint. I think we might actually do a with recursive somewhere in the code base. It just fell out of my head.

...49Are there. Or, you know, the fact that it. Oh, no, this is the first one. All right.

01:46:30twitchtd ya the with_recursive is a nice find, it seems to work a lot better with active record
heypushwhyisthesitebroke gosh every time i try to remember SQL after not writing it for a while i feel like a toddler
I didn't want that indent. That indent is not correct. All right, let's... I can't put the... Can I do that standard? All right, that's acceptable. As long as it still works. I always get nervous about splitting things like that. All right, it works. Great.

01:47:04If I said c.parents.select ID, no, it grabbed star comma ID. Yeah, ActiveRecord is tricky this way because it's building up state. It's not like chaining immutable objects. All right. You know honestly that might be a better fix for. The commentary query and the user threads query so we'll take a look at those two.

01:48:29I want to write the number.

01:49:14twitchtd SQL is worth learning, it's one of the few things that has stayed consistent in tech for me.
heypushwhyisthesitebroke i mean i have learned sql many times, is the problem
See the exact text here. Start from parents.

01:50:22twitchtd I think writing raw sql helped me the most in memorizing it
heypushwhyisthesitebroke there is something about it that my brain just hates remembering--it is worse than apl in this regard for me
Yep.

...52Thank you.

01:52:32What was that number? 2,400 milliseconds. To what? 0.8?

01:53:02Then, where was that.

...11way down at the bottom right. yeah.

...35So I want to. Where's my. So comes posting. This is fine. Reply. And then I want to check for more versus queries. And then we have. So a lot of.

01:54:06All right, so we're going to come back to you pretty quick here. Just as soon as I'm juggling three things, I'm going to start dropping things, right?

...23Where did I just create this query? I should have been upon main. All right. Let's rebase over to main GitHub.

...41Let's bump main. And I'll have to merge something. So we will rebase onto main again. And I have the tree I expected. Yes. And now we will bump main and push that up. and we will deploy, and then we will get distracted by Dependabot.

01:55:25All right, so don't need this up. Come here. So we got Dependabot in here, which... I don't even know what this gem is. Oh, we don't use this, but I will bump it anyways. All right.

01:56:05Slow deployment. These have slipped from being like 26 seconds to being like 41 seconds.

...23All right. 62 seconds. What are we doing here?

...43all right so off screen i did that comment reply and then let's do depend about real quick what did it say is somebody's got a somebody we don't use but might load and socket driver looks like it's vulnerable to some kind of bad data, like if you tell it the header is super long or if you throw a zip-on at it.

01:57:38heypushwhyisthesitebroke did something in rails or the default ruby interpreter get way slower lately? i was writing a rails app for the first time in forever the other day and it seemed way slower than it used to be
So what I'm doing there is heypushwhyisthesitebroke to deploy
explaining why i did it but i'm also saying like i don't think that matters did something in rails or the default ruby interpreter get way slowly i have no idea i have kind of expected and experienced the opposite lately people seem to have put a ton of work into making Ruby faster. All right, so let's go grab my Scratch and say, where's Finite's links?

01:58:39Oh, fine. twitchtd did you use rust or some other low level lang previously? could be highway blindness if you're just jumping back into ruby after one of those langs
Oh, yeah, I have a temp there. Consolata with sharper operators and a narrower voice. Sharper. I'm not sure what sharper means. Locatures that stay close to code.

01:59:16Highway blindness. I could see that. Narrow face pins the width access to width equals 85, keeps the weight variable. Hmm.

...40heypushwhyisthesitebroke hahaha no; rust projects take like ten years to deploy; ruby is still much faster at "actually getting code running"
Oh, you've added an italic.

...59Alright.

02:00:05Well, Vinit, that looks like a neat project. Good deal. I always love the look of ligatures, but then I don't actually want to use them because I have to do that mental transformation.

...28But that's fun to look at. I hope you had fun making it. All right. heypushwhyisthesitebroke could run that project cold like ten times before cargo & rustc would finish a hello world
so let's see if there are more recursive things queries just these okay so they're all in this file all right so let's

02:01:09heypushwhyisthesitebroke on the other hand i unfortunately have been redpilled on ocaml recently
So we find the thread IDs. All right, I'm going to close this scratch file because we have reviewed that. So we have. OK, so. Yeah, comment user threads and. Comment recent threads and story threads. So recent threads really should be named user threads because I keep doing this. vinitkme Need to go for dinner. Have a fun stream, will join back after dinner if it's still on.
Comment story threads. That is not the threads for recent. It is threads for a user. And this is for an individual story. Cool. I'm going to go for about another hour. So see you if we see you.

02:02:08all right so this i want to do story threads first because it doesn't have this added complexity of thread ids and we may be able to just drop that entirely so but we have a similar grab a bunch of ids thing happening with story ids okay So let's find a story that has a bunch of... Reload a SQL story. heypushwhyisthesitebroke i see your fancy italic font but have you considered making the non-italic text comic mono https://dtinth.github.io/comic-…
Count. All right, so we have 39 comments. And if I said comment.storythreads, have I considered making it Comic Mono? No, I haven't. But that is a very fun font. I've seen that one. Story threads. I just want to see the queries happen here. It didn't actually.

02:03:38Because I didn't use the result. Okay. Can I just touch it? No. Can I 2A it and return nil? Yes. Okay. So putting it all together.

02:04:12So that's.

...21So it's grabbing the story, it's grabbing these ideas. I'm going to just. Check this query, but. I'm expecting this to be instant.

...45heypushwhyisthesitebroke lobsters should have an easter egg where if it's april 1 *{font-family: "comic sans ms"}
Yeah, search using covering index. That's as fast as you're going to get. And if I run it, especially because there are no rows, it runs in, you know, rounds to zero. Why is that? Is April 1st BSD day? We do actually have a Comic Sans Easter egg in the code base.

02:05:21There you go. All the open BSD hats are in Comic Sans. It's a running joke in the open BSD community. It's an old joke in the BSD community. heypushwhyisthesitebroke now i need to get an openbsd commit bit
So now we'll grab this.

...49heypushwhyisthesitebroke wait, i don't have a lobsters account, nvm, i am still bsd-free
just get Claude to help you knock out a couple of PRs, right? You'll get one in no time. heypushwhyisthesitebroke ew
I actually have no idea what their stance is regarding LLMs.

02:06:07heypushwhyisthesitebroke i know netbsd is anti and i think freebsd doesn't care
All right, so... Oh, yeah, let's grab this one over to the notes.

...42heypushwhyisthesitebroke oh! wait, i remember now
heypushwhyisthesitebroke openbsd (at least like two years ago) was like "that's obviously stolen code"
So we have coroutine confidence. We set up. We do the recursive step. We search comments using the index. This bloom filter might be a little expensive. But that's really not bad. Although it did take, what is that? Let's round that to nine milliseconds. And again, this is our most expensive query, and I'm running on a... I mean, my desktop is significantly larger than prod, so these numbers are going to be only higher on prod.

02:07:31All right, so here, what if I... Let's look at the other query off to the side. let it go recursive so here we will say comment with recursive we won't say parents no let's just call it confidence just like it is now heypushwhyisthesitebroke does lobsters run on jruby or truffleruby or something?
And then we'll grab everything, which I may not retype that. All right, so there's our structure. And the core of it is, nope, we're on whatever we're calling MRI. See Ruby? I forget what it's called this decade.

02:08:40So we're going to say comment where story ID is in story IDs. So that handles merging. And that's our base case. Yeah. heypushwhyisthesitebroke is yjit enabled by default now on cruby? or wait, i think they threw out yjit for a new rust jit
Then really, you just want to do this. princessxen good morning push
join confidence on comments id right we're just we're walking down the tree no widget so widget is enabled by default i want to say that since ruby4 oh hey see and then zjit is still a work in progress Last I checked, RubyConf is happening, I think, literally right now. So maybe there will be a talk about that happening right now, but I couldn't tell you. So we'll say this. So what I really have part-time expressing is the order. Can I just say dot order?

02:10:16heypushwhyisthesitebroke tfw rubyconf is happening at the red rock casino in vegas
No, I need a JSON array type here, right? So SQLite has an array type, and that's what I'm going to replace with. heypushwhyisthesitebroke that is somehow the least-ruby most-rails place i could imagine rubyconf happening
I don't remember if that's just called their JSON array or if there was a secondary non-JSON array. Yeah, a lot of people were scratching their heads at that choice. Well, you know, maybe they'll have another special guest. that special guest can be another Rails guy again.

02:11:19So what I want is

...37Let's tinker. Let me just do that. Should this be a string literal? Switching between single and double is going to get me. OK. If I said JSON B.

02:12:27heypushwhyisthesitebroke i don't think any community has lost as much aura as rails has in (at least recent) computing history
It turns constructed JSON array. So there are more. So rather than calling JSONB and parsing it, I would rather just construct it. So we'll say JSONB array. Yeah.

02:13:07then we want to give it the list that's going to be like the comment id say one two three four and a confidence of 0.8 what don't we like i have to pass individual arguments not looking like an array okay I want an array so I want to say I want an array of arrays jsonb array of jsonb array yes okay I want to select star and

02:14:07jsonb array of jsonb array of id and conf no confidence and then id so the reason we need Actually, we don't have to nest that. Yeah, confidence and then the ID. We just need the confidence, but then we have to use ID as a tie breaker. So there is low cardinality in the confidence field because lots of comments have like two upvotes and no flags. And so they all have the exact same confidence value. And we have to separate by ID to make sure that we're not accidentally swapping siblings around. Yeah. And then this one. It's going to pick up a select star. This has confidence. Well. So confidence order. And then. This is bad, say. Parents. On.

02:16:06we're joining in the wrong direction yeah comment id is in p.id so we will select star comma and then we want

...38This.

...45But we do want to prepend. Great insert. Is there an append? For example, when pending.

02:17:12Jason insert okay. Yes, we will say, then we will take the. parent.com fit. confidence order.

...33Now we'll come json insert.

...42onto that, this value, and then the location is this magic string. So dollar, sine, square, hash, square.

02:18:03Okay, and then order by,

...12Does confidence run low to high? High to low. No, it's the story values.

...45princessxen i spaced out, what is going on in this stream?
oh hey we are doing some follow-up performance work from the big sqlite migration and one of the things that we've found is that we can improve the way we wrote our recursive queries to avoid an extra layer of joins and we got there because comment posting was slow but then we figured out oh we have a general technique here so let's use it some more So I'm rewriting this big SQL query to get rid of this join, which hopefully will speed this up. It's certainly less custom SQL, which is nice. And it doesn't have the big bit bashing that we used to have.

02:19:43So we want to actually... I think that... maximum value of confidence is. Because we want to sort these ascending. So just

02:20:27just to see how the query looks and then we can order by confidence path ascending that looks pretty reasonable let's see what query gets generated an exception oh it's not called order by it's just called order all right that's a more interesting failure Whereby confidence path ascending. Because if we do that, so let's say we have a JSONB array.

02:21:14Sort function, I kind of want multiple rows to make sure that SQLite does what I'm assuming about sorting arrays.

...28twitchtd just have a question about that select('*') usage, if you select('*') and then activerecord selects some columns, like id, wouldn't the resulting query *, id have a double column select error when activerecord tries to map it to the model, or does it handle duplicate columns in the result ok
It doesn't like as. OK. Do you want it just this way?

...45I think it handles duplicate columns in the result, but we can double check. SQLite error.

...59near confidence order. I want to back up one because I thought that as was required. And then we will just go look at that in the SQLite. twitchtd ok, i need to step away from the computer for a little bit but just wanted to throw hat out there
So it's not just the ads, huh? What if I said select ASDF as foo? Yeah. Select ASDF, select ID from comments limit one, right? Select ID as foo limit one, okay? select id food yeah both of those work what is it mad about i'm not reading it right with your cursor parent as select this as conference plan let's grab just this

02:23:27So you like that just fine.

...36chamlis_ think you're missing the closing paren on the json_insert?
You don't like this second one then.

...45I put the has confidence order inside the parentheses instead of outside. All right. So this.

...57Where's my ass? Get the ass back. This parentheses wants to be back here.

02:24:13No such table parents. I called it parent singular.

...25No such table. Confidence. Or do you mention that still? Ah. Join parent.

...44Ambiguous column name confidence.

...59This one has to be comments.

02:25:06Ambiguous name ID. Sure. No such column P ID. Parent.

...31selects for the left and right of union all do not have the same number of result columns oh that's interesting they sure look like they do limit one and this is going to fail because i don't have a parent relation select star comma json insert as confidence order parent as

02:26:47Make it a line break in there as confidence. Yeah.

02:27:15heypushwhyisthesitebroke has any joker made a coffescript for sql yet
Yeah, I think a few people have made those.

...22dlamz prql looks super nice
princessxen i hate it
If you squint a little, that's kind of GraphQL, right? I didn't get anything. PRQL. heypushwhyisthesitebroke significant whitespace
I don't think I've seen that one.

...48heypushwhyisthesitebroke is what i meant
Oh, well, at least they put things in a more sensible order.

02:28:00So.

...12That should have found rows. And because it didn't, we're not getting the headers to know its cardinality. But I don't see how that cardinality would not have matched up. Because JSON insert returns a single value, right? JSON insert. Grab this, comma, comma, and then the new value. Except I don't want to nest that array. I just want to have those values. Maybe that's it. Nope.

02:29:30selects to the left and right of union all do not have the same number of result columns except that they do all right so if i say select json b insert And then into a JSON. Let me be array with the elements, I don't know, 0.9 and one with the string here, and then I want these elements at the end, and we'll say four and five. chamlis_ do you need "SELECT comments.*" to avoid the columns from parent?
JSON insert needs an odd number of arguments.

02:30:35Oh, channel list, that's a great idea. That's probably it.

...51Yes, all right, no such order confidence path. All right, we will order by confidence order sending. Jason insert needs an odd number of arguments cool so we're back to this was it I can't append multiple I want to pen multiple. insert multiple elements to the end of an array. Can I say four comma five.

02:31:36always take an odd number first argument is always the json subsequent arguments are current pairs the first is a pair and the second is okay so then if i specified that five also goes to the end there we go okay then We'll grab that and put it between. We will not grab that. We will retype that between.

02:32:16Malformed JSON. How is it malformed? You made it. Did I mix JSON and JSONB? Yes.

...32Malformed Jason. I mean, I know he's funny looking, but you don't have to call Jason malformed. Hmm.

...59zurtar 🥁
yeah i'm here all week you can't get rid of me why is it read line no you don't want to just let me grab the right thing

02:33:37heypushwhyisthesitebroke does lobsters is hosted on openbsd?
heypushwhyisthesitebroke or did it finally succumb to the evil penguin
heypushwhyisthesitebroke tragic
it's like all right all right at least i can get back by word no lobsters is hosted on linux it's a ubuntu lts okay so we didn't get an error but we didn't get anything else i don't I don't remember if Lobster's was originally hosted on BSD even. Because I want to say back in 2012, there were some hassles with running Rails apps on BSD. Maybe I'm misremembering. It's been a hot minute.

02:34:29heypushwhyisthesitebroke mfw
So we're not getting an error here. We sure are here.

...42Outformed JSON. What don't you like about this?

02:35:05But you like it just here, right? How weird is that, that I pasted the query over to the database. twitchtd it says malformed there as well
And SQLite's like, yeah, no, I'm fine with that. Could it be something about the value? No, we're starting from the exact same story ID. Does it? Oh, it says it here. I just thought I was getting no results. Seeing it below the stat made me think it actually ran. All right.

...54What if I use JSON? Now form JSON.

02:36:12So if I run this part of it, let's just grab that. Yeah, these are exactly what I wanted.

...39So it's something that it doesn't like about the insert.

...56so let's say let's get rid of this get rid of that replace this with its value json array and we don't need any of that stuff so pull that back from comments plural that's a weird line break probably caused by the doing the whole database I should have said like limit two but all right well it's malformed huh it'll it's happily to let me append multiple items

02:38:24chamlis_ is is the order by?
chamlis_ is it*
Oh that's possible. Let's try that.

...41No, it doesn't care. Yeah, seems fine with that.

02:39:09Is there a limit to its JSON arrays? Two, three, four. It's weird to think that we could hit that. So our deepest comment is like 30 levels deep. And so that would be... 60 numbers, because each comment is two sets, right? So 10, 20, 30, 40, 50, 60, and one to grow on. Yeah, you're fine with that.

02:40:00What if I commented out the order there? No.

...19So really the only difference left is reusing this from the parent across the union. versus having a literal value in there. So if I replaced this with JSON array one, two.

...46Okay, there we go. We've isolated the error. It's something about passing the parent row.

02:41:06All right, so let's grab this new one.

...18This terminal just keeps fighting me.

...37Then it won't let me join up the lines.

02:42:30Yeah.

02:43:09So confidence is 0.9. Actually, I think confidence is in the range 0 to 1, right? Let's select min.

...35So we can, instead of this 999 stuff, we can say one minus, one minus.

...54And then conference order pass should be the last thing. So next line. Yeah, that looks, reasonable no it doesn't hold on let's just select confidence order got rid of the space

02:44:41Let's drop out of JSONB.

...56It's not the right query. All right, so yeah. let's grab that query again, ink it, bring it over here, and then only grab confidence order, right?

02:45:56Do you want me to say JSON be parent dot confidence order instead of just giving it the value? And I'll form JSON.

02:46:39chamlis_ is there still a confidence_order on the table that it's using instead?
So it feels like there's something about the way we're seeing this encoding now.

...50Form JSON. Is there still a confidence forum? I thought it was just called confidence on the table. There is a confidence order on the table, wait. Oh, I've been serializing it. Oh, that's the error. I forgot we had a serialized. All right, so as pat, as pat, order by pat.

02:47:36Hey, look, we didn't get an exception. You got it, Chambliss. That's another one.

02:48:05That's a nice short query plan. Look at that. so if i grab this come around yeah instead of this whole thing with the bloom filter we've totally cut it off now we're going to do a golden master test

...39Story threads to. Takes a story. And it has all this in it. And then. The original.

02:49:03Can get uncommented. And this will go away from here. right all right so if i said give me the story threads count i get 39 i say give me the story threads two story threads underscore two names it with an underscore

...42is a reload there right comment story comment.story threads two did i not save oh i guess i've been programming for a minute because i am missing some obvious difference between these two undefined method story threads two for class comment self 134 okay so I'm getting different results

02:50:3412. 127.870. 127.870. Hmm. How am I getting more comments? And parent. twitchtd and parent_comment_id is null
Ah ha ha. Yeah, you got it.

02:51:07Thanks, Thomas.

...18Now I'm getting 39. There we go. You got it in one. All right. Now am I getting the same comments in the same order? And I'm deliberately choosing map instead of plug, so I don't fight that select.

...40Can we do that on one line? Not what's there. Nope. Bad dot join with a space.

...58Actually, you know what? No, multiple lines is fine.

02:52:07twitchtd sort first
yeah let's look use puts to get rid of the punctuation and we'll grab all of this and i will grab a i just vim diff no it's not let's just do the easy thing i know there are nicer ways to do this but yeah greater than story threads now let's do the same for story threads too oh story threads too

02:53:10No, we got them in a different order. Okay. That's that's a problem.

...29The beauty. 641 and 575. twitchtd oh there's already an order by, nvm
So I bet if I went back here, the top comment has a single. Yeah, top comment has a single reply. So top comment has a single reply. So we got and then 575 is the next parent. So we got all of the parents.

...58Yeah, so the order chamlis_ I'm not sure if sorting by json(b) is defined in sqlite, you might be sorting by the internal blob representation
is not what I was hoping for.

02:54:17That's totally possible. Yeah. How do we check that? Well, one way to check it.

...45I'm going to do, well, let's start with JSONV, right? These are probably going to be in white order.

02:55:11Yeah, it's just, Printing the representation.

...32chamlis_ and if they represent arrays by their length first that would explain getting the top-level comments first?
But we didn't get, so let's order by J. I was kind of wondering if this 10 was going to jump up to the top.

...54It's represent raised by their length first. That would explain getting concerts, right?

02:56:20Let's let's grab the query and take a look.

...32So we want to run that. Why didn't I see the query happen?

...53I saw the first two. Where's my third query? Oh, because I threw away the 2a. Otherwise, the query is just hanging out. Yeah. Let's take this.

02:57:20So let's then select ID and path.

...32heypushwhyisthesitebroke i have said this before but you have asmr voice (i was zoning out and doomscrolling and something you said "...and take a look") reminded me of like, bob ross, or one of those 70s-era educational hosts
try selecting id and parent comment d and path yeah so we definitely got all the roots first i have asmr voice all right you know i am a big fan of bob ross and mr rogers and the other fairly chronostory i also get bob ross vibes here lol
chamlis_ the joy of programming
chill gentle masculine folks so is that thanks but you know i don't really have happy little accidents i just have like frustrating little accidents heypushwhyisthesitebroke ah, but you deal with them so calmly
shouldn't these arrays be getting longer right like so i look at this story and this gets nested and nested and nested but this doesn't all right so we've got

02:59:10chamlis_ you hardcoded json_array(1,2) instead of the parent path
I hard coded. Oh, shoot. I did when we were figuring out the other thing. So what was you supposed to be? You were supposed to be. At.

...27And. Now we're going on, was it parents? The path. Cool, now we're back into getting exceptions. Near as. Oh, I didn't edit that correctly. All right, there we go. You know, that's silly enough that that could have been our bug for why they're out of order. let's go back here instead of say id comma parent comment id comma these values look better so now we're having long arrays okay so we do still have all of the thread routes first and then we have 580 right so here's that first reply. That should be up here.

03:00:46So I think channels that you're right it's sorting the shorter items first. 575. 16, 575, 1, 2, 3. And if I look at the thread, the second gets 1. No, it's doing it. Yeah, it's doing all the levels in order. So yeah, this problem, of course, is a recursive problem.

03:02:13Minify version of that JSON string. Turns the binary JSON.

...56Now we're getting children before parents. So there's 580, and then there's its reply. So if we do a naive text search, so Chambliss, I believe you're right, that it's sorting by the binary representation of JSONB. So I changed it here to use JSON instead of JSONB, and now we are getting a string sort. And I bet if I look up an ASCII table, comma is before square bracket. You know, it's funny to have that in my history. So square bracket is in small font. Comma is 44. chamlis_ making the lookup table an image is evil
yes square bracket is down here okay so now we understand we are very clearly sorting by the string or the binary representation not by the values do they have a sort function they do not have the word sort they have to pull it in to have any in order.

03:04:54So I could reintroduce the terrible string hack from MariaDB, but I was hoping to avoid it. I really did not expect that SQLite was gonna sort these arrays by their string representation rather than their values. Hmm.

03:05:33twitchtd what was the problem with jsonb? it's sorting by array length first?
No, it's not sorting by array length first. It's sorting by the... I mean, yes, it is sorting by array length first, but that's intendental to the fact that it is sorting by the internal binary representation of them. So, you know, we saw that way up here somewhere.

03:06:04I don't remember what that query was that got it to print it raw instead of, yeah, here we go. Or no, those were confidence order values. Those were what we were storing in our database.

...20We could go back to that, right?

...42Some of SQLite's limitations are pretty funny.

...51I mean, every database lets the seam show here and there. This one surprised me.

03:07:31I'm not sure what to do with this. chamlis_ `ORDER BY path->'$[0]', path->'$[1]',...`
This feels like it may be a dead end. Unless we wrote a custom sorting function. Order by... Ooh.

...59Yes. that would be you know the other thing is just a silly thing is can i tell it to so i shouldn't be able to do this but because sqlite is sort of low level and lets the seams show What if I said, what if I got rid of the, no, if I sort those, all the children are gonna fall at the bottom. chamlis_ or you could do the final sort in ruby?
I was starting to think, what if I sorted inside the recursive, inside the definition of parent? And what I was expecting was that that order would be preserved. Even though, you know, the SQL spec doesn't guarantee it on common table expressions. But no, I'm going to get all of the parents. And then I'm going to get all of the children. Or I could do the final sort in Ruby. Really been trying to avoid that. That's where we started, and... the sort was increasingly expensive. That's what pushed me into all of this. And then because it's multi-level, it ended up being a recursive sort.

03:10:00chamlis_ there was that depth-first style thing we came up with a few months ago that made use of how sqlite processes recursive ctes
gtfrvz https://sqlite.org/lang_with.ht… ?
it would work wouldn't it your idea so you said order by path dollar zero oh chamois i don't remember the specific your thing you're thinking of but if you can remember what it was that sounds interesting and Come here, just let me, oh my God, let me finish the query.

...46That URL looks very promising. I'm going to click on that in a second. I just wanted to see this because I think it'll work.

03:11:02chamlis_ this one I believe https://push.cx/stream/2026-05-…
twitchtd gtfrvz oh that's a very interesting link
yeah oh this isn't the one that i changed to say just give me the id all right comment id path that's the wrong first chamlis_ yeah this documentation came up on that past stream I think
number all right let's try this link with the promising anchor order by clause on the recursive select can be used to control whether the tree is depth first or breadth first illustrate we'll use a variation on the org table yeah that's basically what we're doing name and boss references org sure that's what we're doing with id and parent comment id Yeah, this is ringing a bell.

03:12:15Values. If we change the order by addDesk, it'll cause lower levels in the organization. larger level values to be processed first resulting in a depth first search what is this values

03:13:00It's basically our two element array.

...15No, they're just hard coding Alice and zero.

...47princessxen oh my god i just realized how go time format strings work
So in this stream, is there any chance we wrote this exact query?

...56Yeah, confidence order is JSONB. One plus N to fit to the tree in order. Oh look, chamless is SQL pastebin. What are the odds that this is chamless writing the exact query we want? Order by confidence order. princessxen the reference date is january 2nd 2006 at 3:04:05 PM
Order by depth descending confidence order. princessxen 1 2 3 4 5 6
chamlis_ I think you corrected confidence_order to confidence, yeah
Why confidence order and not just confidence?

03:14:36Wait, Z, what is that time format string? princessxen Go's time formatting is fucked lol
What is the reference date?

...52Ah, go. Could be. Select.

03:15:08princessxen you do it in terms of format strings like 20060102T150405Z
Order by depth descending, confidence descending. That looks pretty good, actually. So instead of all this,

...35Grab comment star.

...43Join order by.

...52Where was it putting the order inside?

03:16:01Inside. So no order on the outside. And then inside here, we were saying order by confidence descending. Order by depth descending and confidence order descending. order by clause should come after the final union all okay, so I can't give an order to the first one let's just see. Such calm path right.

03:17:10chamlis_ yeah the order comes at the end and is conceptually how sqlite picks the next item from the queue to expand on
There are way too many rows. All right. That should have been 39 rows. chamlis_ I think you need a `parent_comment_id IS NULL` in the initial select
Select. Start from parents. From parent. We're back to 134. We lost the null. Yep.

...43It's not another where because SQL is so irregular. And OK. Now ID, parent, comment, ID, confidence, parent.

03:18:09That's the wrong order. It was supposed to be 570 first. And then the second thing should have had a reply.

...38What am I... Am I looking at the exact? Oh, look. So Thomas, there's the figuring out how to get rid of the outer join. We already figured that out. What's the date on this? A year ago. Oh boy. espartapalma 2 months ago
And parent comment ID is null. Join comment tree. Did I write two months ago? Oh, god, it is. Man, I can't read. Ah, we are at three hours and 20 minutes. No wonder I'm making silly mistakes. So that one said depth and then confidence. What did I say? I said depth and then confidence. What if I just run this query?

03:19:48And swapped in.

...55Got line wrapping again.

03:20:12And this one also just does the merge story IDs in one step. So if I did this, and then I swapped in, what's our stories ID? 127.870, because I want to see familiar comment IDs.

...49Okay, I got comments. Now just show me ID parent comment ID confidence.

03:21:04That looks correct. Yeah, 580. It's child 575. It's child 44. Yeah. And we're back up. All right.

...19So what was different? Aside from the story part, what about depth descending, confidence descending? Isn't that what I typed?

...48So let's Come here. I want to see the difference. This is another one that mentions JSONB. Wow, that's the order by path version. I must have lost some history lines here. I bet when I cancel, it doesn't save history. That's obnoxious.

03:22:36Huh? And parent is null. Join parent. On comments, join comment tree. On comment tree ID is the parent.

...56Okay, these just swap sides. They're fine. Order by depth ascending confidence desk. twitchtd top has merged_story_id
It's the same thing.

03:23:15yeah all right well let's grab this version i wrote previously come on and we are going to just put this in here and comment out number two And then we don't have to do that.

03:24:01What is my exception?

...16Don't like that I have a parentheses around that select.

...27But I don't have one, unless MariaDB is doing it. Let's see it here.

03:25:06I'm seriously just mad about that parent. Why is Active Record generating it then? It is, okay. How do I get Active Record to not generate that?

...46And why is it doing it on this one when it didn't do it on my previous?

03:26:02chamlis_ is it trying to group the order by in with that query too tightly
It's trying to group the order by in with that query too tightly. I mean, this is where the order has to be.

...50Because I didn't have a select on this.

03:27:06No, that still generated one.

...25There is something really subtle and irritating happening.

...37Is it because this one has an outer pad? Or outer order? Order by ID. I just want to see the query get generated. No.

...54Honestly, this feels like an active record bug, but there's some difference between the two that I'm not seeing.

03:28:19twitchtd could it be https://github.com/rails/rails/… ?
I mean, as silly as this is, is it swapping the from and the select? Could it be?

...36Let's take a look at this.

...44Building union or union all involving a limiter order by, ahaha, Errol generates invalid SQL as described in 401 .

03:29:01missing parents around the first query SQL syntax care. Same error occurs with order.

...15chamlis_ good find
Using URL directly.

...25Yeah, this is prior to recursive. We couldn't have written the with recursive call, because this is when with recursive was added. Turns out this syntax is not supported. Wrap in parens the presence of limit orders or offset in the select statement.

...56I mean, that's still invalid. Select one night.

03:30:14Not in the right one.

...25There's my error.

...33All right, OK. That's the difference. If I take the order off, it's not going to put parentheses around it, right?

...54And then if I actually call it, we don't get an exception.

03:31:05princessxen incredible
OK, so this is an active record bug.

...20Well, let's grab this ID.

...34I don't find a lot of bugs in Active Record.

03:32:10twitchtd https://github.com/rails/rails/… looks like the order by causes the parens
heypushwhyisthesitebroke damn i went and had a full demented conversation where i tried to convince a roommate not to violate can-spam for like ten or twenty minutes and we're still on the sql query?
loud and single like three correct only references if the node select statement has non-empty in which case that's well no that's all right so like this is fine it's not invalid without that we're we're down the rabbit hole We found a, I guess, a bug in ActiveRecord or Errol specifically. Still invalid, but it would have been invalid as well without parentheses. Limit clause should come after. Yeah, but that's only on the left side. I'm doing it on the right side. That's totally different.

03:33:05Fixes the union select parentheses. Is that the one I was just on? Yes.

...19princessxen oh holy shit: MS Comic Chat is now open source
And nobody has reported this bug.

...33princessxen https://opensource.microsoft.co…
How do we get that to work? Because that really implies, I'd have to go back through the video, but this sure looks like it's generating this query. But it can't be because of this bug.

03:34:16twitchtd ya not sure how it worked with that .order
Oh, GitHub.

...23I clicked on the diff here. princessxen yeah you have to press control-l and enter to load it
chamlis_ the engines cannot take it captain
Man, I thought we had some performance issues.

...46I'm teasing, you know, this is a huge repo. This is a huge site. Maybe GitHub should port to SQLite and they'd be faster and save CPU. It's going to be very funny when this finishes and then it just like 500s.

03:35:10twitchtd I've noticed that bug in github, i had to reload
heypushwhyisthesitebroke do you listen to the sinatra cover of that song or the original?
twitchtd to get github to work
Or just doesn't do anything. if i reload there we go man that was grouping parentheses ah damn i was kind of hoping we could reach inside and yank those parentheses out

03:36:06This work. chamlis_ I'm not sure this worked last time either
There's no return.

...39Yeah, I guess it didn't. No such table comment tree. That is not the exception I expected.

03:37:40Oh look, the exact same exception.

03:38:18let's write a worse version i can't even

...59I want to build up a tree and memory and then flatten it to a list. I just want these damn comments in order. I was thinking of writing the N plus one version, right? But then I have to build a data structure. I want to build a data structure. And I have to walk the data structure to get a list to pass to the template. So I have to build a second data structure. And you know, the fact that that works means it's faster than story threads too.

03:39:48All right, I think I'm done for the day. This is kind of a dispiriting note to end on, but I'm tired and I'm making silly mistakes. I'm not going to see anything nicer. If anybody wants to report this bug derails, put this in the scratch here.

03:40:41heypushwhyisthesitebroke go have hot cocoa! enjoy your day!
Not Coco. It is like 90 here in Chicago. Not doing Coco. heypushwhyisthesitebroke even better
heypushwhyisthesitebroke cold cocoa
I want to see the exact query for certain word story threads to go.

03:41:15heypushwhyisthesitebroke yoohoo
Cold cocoa. How did I? No? All right, let's try again.

...38There we go.

03:42:10heypushwhyisthesitebroke in my mind yoohoo is a midwestern staple
Oh boy.

...18I don't drink a lot of pop, couldn't tell you.

...27Soda? I didn't even say soda or pop. Well, often enough to have a preference. Alright.

...41heypushwhyisthesitebroke yoohoo is chocolate
So if we'd done user threads, we would have run into the same bug.

...58twitchtd can you gist that bad with_recursive?
heypushwhyisthesitebroke https://en.wikipedia.org/wiki/Y…
twitchtd ya
You want this section?

03:43:11Once I look.

...20Here we go. chamlis_ ooh fancy
pushcx https://gist.github.com/pushcx/…
So the whole thing or is it the selection? twitchtd ty
Yeah, it's the selection. Cool. Nothing wrong with putting that. Yeah. Got to be a Tim Pope original, right?

...44Oh no, mad end.

...52heypushwhyisthesitebroke > As of 2019, the drink is primarily made from water, high-fructose corn syrup and whey. The drink comes in glass/plastic bottles and in drink boxes.
heypushwhyisthesitebroke nothing more midwestern
All the struggling I've done to grab things with and without line wraps. I figured I'd save some hassle. chamlis_ thanks for the stream!
Sounds like a real charmer. All right, well. I'm gonna roll out. The likely next stream will be, yeah, probably next Thursday is more likely. zurtar Was fun, thanks for the stream.
Tuesday's a little unlikely. And there will be a break in, what is the date? Maybe in 10 or 12 days, I'm gonna take a break for two, two and a half weeks for some travel. So I will sort out my calendar and, heypushwhyisthesitebroke adios! enjoy chicago heat
update Twitch and whatnot, and tweet about it. Yeah. This feels very close. heypushwhyisthesitebroke maybe go to one of its concrete beaches
I'm glad we got the one big slow performance thing, but I would really have liked to get those other two queries today.

03:45:02All right. Ah, yes, the concrete. Yeah. Honestly, I'd rather go swim. There are plenty of beaches. All right. Well, I'm going to roll out. Take care, folks. I will see you around the faster sequel later site. heypushwhyisthesitebroke o7
Yeah, take care. zurtar Have a good one
Thanks for tuning in.