That guy could've been chopping trees for the last five years
Streamed
styled_user_link caching performance fix.
Still no perf monitoring.
SQLite stat tables.
Verse web framework](https://github.com/GrayHatter/verse).
Zig language zen.
Middleware.
Blurring lines in web frameworks.
Playing with time.
Diplomacy, the adjudicator test cases, TLA+.
Rails with_recursive.
F# type classes discussion.
scratch
topics
no PR movement
perf monitoring https://github.com/lobsters/lobsters/issues/2016
/users username helper - caching perf fix
sqlite perf again
blurring the line between web framework and stagic site generator
ssg: blog post -> template -> html file
framework: db -> blog post -> template -> html file
lobsters cache hassle: db -> current time -> template -> html file
story_threads
current, mariadb: confidence_order_path
sqlite 1: confidence_path as jsonb
sqlite 2: 1 + n to fetch tree in order
sqlite 3: chamlis dfs
chamlis's sql pastebin:
inner join (
with recursive confidence as (
select
c.id
from comments c
join stories on stories.id = c.story_id
where (stories.id = #{story.id} or stories.merged_story_id = #{story.id}) and parent_comment_id is null
order by confidence_order
union all
select
c.id
from comments c
join confidence on c.parent_comment_id = confidence.id
order by depth desc, confidence_order
)
select * from confidence
) confidence
on comments.id = confidence.id
WITH RECURSIVE "comment_tree" AS ( SELECT "comments".* FROM "comments" INNER JOIN "stories" ON "stories"."id" = "comments"."story_id" WHERE ((stories.id = 43385 or stories.merged_story
_id = 43385) and parent_comment_id is null) UNION ALL SELECT "comments".* FROM "comments" JOIN comment_tree ON comment_tree.id = comments.parent_comment_id ORDER BY depth desc, confidence desc
) SELECT * FROM "comment_tree";
return Comment.with_recursive(comment_tree: [
Comment.joins(:story).where("(stories.id = #{story.id} or stories.merged_story_id = #{story.id}) and parent_comment_id is null"),
Comment.joins("JOIN comment_tree ON comment_tree.id = comments.parent_comment_id").order("depth desc", "confidence desc")
]).from("comment_tree").select("*")
WITH RECURSIVE "comment_tree" AS ( SELECT "comments".* FROM "comments" INNER JOIN "stories" ON "stories"."id" = "comments"."story_id" WHERE ((stories.id = 43385 or stories.merged_story
_id = 43385) and parent_comment_id is null) UNION ALL SELECT "comments".* FROM "comments" JOIN comment_tree ON comment_tree.id = comments.parent_comment_id ORDER BY depth desc, confidence desc
) SELECT * FROM comment_tree;
sqlite> explain WITH RECURSIVE "comment_tree" AS ( SELECT "comments".* FROM "comments" INNER JOIN "stories" ON "stories"."id" = "comments"."story_id" WHERE ((stories.id = 43385 or stories.merged_story_id = 43385) and parent_comment_id is null) UNION ALL SELECT "comments".* FROM "comments" JOIN comment_tree ON comment_tree.id = comments.parent_comment_id ORDER BY depth desc, confidence desc) SELECT * FROM comment_tree;
addr opcode p1 p2 p3 p4 p5 comment
---- ------------- ---- ---- ---- ------------- -- -------------
0 Init 0 144 0 0 Start at 144
1 InitCoroutine 1 116 2 0 comment_tree
2 OpenPseudo 1 2 23 0 23 columns in r[2]
3 OpenEphemeral 5 4 0 k(3,-B,-B,) 0 nColumn=4; Queue table
4 OpenRead 2 15 0 23 0 root=15 iDb=0; comments
5 OpenRead 6 76 0 k(2,,) 2 root=76 iDb=0; comments_parent_comment_id_fk
6 OpenRead 3 36 0 16 0 root=36 iDb=0; stories
7 Null 0 3 0 0 r[3]=NULL
8 Affinity 3 1 0 C 0 affinity(r[3])
9 SeekGE 6 48 3 1 0 key=r[3]
10 IdxGT 6 48 3 1 0 key=r[3]
11 DeferredSeek 6 0 2 0 Move 2 to 6.rowid if needed
12 Column 2 4 4 0 r[4]= cursor 2 column 4
13 SeekRowid 3 47 4 0 intkey=r[4]
14 Rowid 3 5 0 0 r[5]=stories.rowid
15 Eq 6 18 5 68 if r[5]==r[6] goto 18
16 Column 3 15 5 0 r[5]= cursor 3 column 15
17 Ne 6 47 5 BINARY-8 84 if r[5]!=r[6] goto 47
18 IdxRowid 6 7 0 0 r[7]=rowid; comments.rowid
19 Column 2 1 8 0 r[8]= cursor 2 column 1
20 Column 2 2 9 0 r[9]= cursor 2 column 2
21 Column 2 3 10 0 r[10]=comments.short_id
22 Column 2 4 11 0 r[11]= cursor 2 column 4
23 Column 2 5 12 0 r[12]= cursor 2 column 5
24 Column 2 6 13 0 r[13]= cursor 2 column 6
25 Column 6 0 14 0 r[14]= cursor 6 column 0
26 Column 2 8 15 0 r[15]= cursor 2 column 8
27 Column 2 9 16 0 r[16]= cursor 2 column 9
28 Column 2 10 17 1 0 r[17]=comments.score
29 Column 2 11 18 0 0 r[18]=comments.flags
30 Column 2 12 19 0 0 r[19]=comments.confidence
31 Column 2 13 20 0 r[20]= cursor 2 column 13
32 Column 2 14 21 0 0 r[21]=comments.is_deleted
33 Column 2 15 22 0 0 r[22]=comments.is_moderated
34 Column 2 16 23 0 0 r[23]=comments.is_from_email
35 Column 2 17 24 0 r[24]= cursor 2 column 17
36 Column 2 18 25 0 0 r[25]=comments.depth
37 Column 2 19 26 0 0 r[26]=comments.reply_count
38 Column 2 20 27 0 r[27]= cursor 2 column 20
39 Column 2 21 28 0 r[28]= cursor 2 column 21
40 Column 2 22 29 0 r[29]= cursor 2 column 22
41 MakeRecord 7 23 33 0 r[33]=mkrec(r[7..29])
42 SCopy 25 30 0 0 r[30]=r[25]
43 SCopy 19 31 0 0 r[31]=r[19]
44 Sequence 5 32 0 0 r[32]=cursor[5].ctr++
45 MakeRecord 30 4 5 0 r[5]=mkrec(r[30..33])
46 IdxInsert 5 5 30 4 0 key=r[5]
47 Next 6 10 0 0
48 Rewind 5 115 0 0
49 NullRow 1 0 0 0
50 Column 5 3 2 0 r[2]= cursor 5 column 3
51 Delete 5 0 0 0
52 Column 1 0 34 0 r[34]=id
53 Column 1 1 35 0 r[35]=created_at
54 Column 1 2 36 0 r[36]=updated_at
55 Column 1 3 37 0 r[37]=short_id
56 Column 1 4 38 0 r[38]=story_id
57 Column 1 5 39 0 r[39]=confidence_order
58 Column 1 6 40 0 r[40]=user_id
59 Column 1 7 41 0 r[41]=parent_comment_id
60 Column 1 8 42 0 r[42]=thread_id
61 Column 1 9 43 0 r[43]=comment
62 Column 1 10 44 0 r[44]=score
63 Column 1 11 45 0 r[45]=flags
64 Column 1 12 46 0 r[46]=confidence
65 Column 1 13 47 0 r[47]=markeddown_comment
66 Column 1 14 48 0 r[48]=is_deleted
67 Column 1 15 49 0 r[49]=is_moderated
68 Column 1 16 50 0 r[50]=is_from_email
69 Column 1 17 51 0 r[51]=hat_id
70 Column 1 18 52 0 r[52]=depth
71 Column 1 19 53 0 r[53]=reply_count
72 Column 1 20 54 0 r[54]=last_reply_at
73 Column 1 21 55 0 r[55]=last_edited_at
74 Column 1 22 56 0 r[56]=token
75 Yield 1 0 0 0
76 OpenRead 4 15 0 23 0 root=15 iDb=0; comments
77 OpenRead 7 76 0 k(2,,) 2 root=76 iDb=0; comments_parent_comment_id_fk
78 Column 1 0 57 0 r[57]= cursor 1 column 0
79 IsNull 57 114 0 0 if r[57]==NULL goto 114
80 Affinity 57 1 0 C 0 affinity(r[57])
81 SeekGE 7 114 57 1 0 key=r[57]
82 IdxGT 7 114 57 1 0 key=r[57]
83 DeferredSeek 7 0 4 0 Move 4 to 7.rowid if needed
84 IdxRowid 7 7 0 0 r[7]=rowid; comments.rowid
85 Column 4 1 8 0 r[8]= cursor 4 column 1
86 Column 4 2 9 0 r[9]= cursor 4 column 2
87 Column 4 3 10 0 r[10]=comments.short_id
88 Column 4 4 11 0 r[11]= cursor 4 column 4
89 Column 4 5 12 0 r[12]= cursor 4 column 5
90 Column 4 6 13 0 r[13]= cursor 4 column 6
91 Column 7 0 14 0 r[14]= cursor 7 column 0
92 Column 4 8 15 0 r[15]= cursor 4 column 8
93 Column 4 9 16 0 r[16]= cursor 4 column 9
94 Column 4 10 17 1 0 r[17]=comments.score
95 Column 4 11 18 0 0 r[18]=comments.flags
96 Column 4 12 19 0 0 r[19]=comments.confidence
97 Column 4 13 20 0 r[20]= cursor 4 column 13
98 Column 4 14 21 0 0 r[21]=comments.is_deleted
99 Column 4 15 22 0 0 r[22]=comments.is_moderated
100 Column 4 16 23 0 0 r[23]=comments.is_from_email
101 Column 4 17 24 0 r[24]= cursor 4 column 17
102 Column 4 18 25 0 0 r[25]=comments.depth
103 Column 4 19 26 0 0 r[26]=comments.reply_count
104 Column 4 20 27 0 r[27]= cursor 4 column 20
105 Column 4 21 28 0 r[28]= cursor 4 column 21
106 Column 4 22 29 0 r[29]= cursor 4 column 22
107 MakeRecord 7 23 33 0 r[33]=mkrec(r[7..29])
108 SCopy 25 30 0 0 r[30]=r[25]
109 SCopy 19 31 0 0 r[31]=r[19]
110 Sequence 5 32 0 0 r[32]=cursor[5].ctr++
111 MakeRecord 30 4 5 0 r[5]=mkrec(r[30..33])
112 IdxInsert 5 5 30 4 0 key=r[5]
113 Next 7 82 1 0
114 Goto 0 48 0 0
115 EndCoroutine 1 0 0 0 end comment_tree
116 InitCoroutine 1 0 2 0
117 Yield 1 143 0 0 next row of comment_tree
118 Copy 34 58 0 2 r[58]=r[34]
119 Copy 35 59 0 2 r[59]=r[35]
120 Copy 36 60 0 2 r[60]=r[36]
121 Copy 37 61 0 2 r[61]=r[37]
122 Copy 38 62 0 2 r[62]=r[38]
123 Copy 39 63 0 2 r[63]=r[39]
124 Copy 40 64 0 2 r[64]=r[40]
125 Copy 41 65 0 2 r[65]=r[41]
126 Copy 42 66 0 2 r[66]=r[42]
127 Copy 43 67 0 2 r[67]=r[43]
128 Copy 44 68 0 2 r[68]=r[44]
129 Copy 45 69 0 2 r[69]=r[45]
130 Copy 46 70 0 2 r[70]=r[46]
131 Copy 47 71 0 2 r[71]=r[47]
132 Copy 48 72 0 2 r[72]=r[48]
133 Copy 49 73 0 2 r[73]=r[49]
134 Copy 50 74 0 2 r[74]=r[50]
135 Copy 51 75 0 2 r[75]=r[51]
136 Copy 52 76 0 2 r[76]=r[52]
137 Copy 53 77 0 2 r[77]=r[53]
138 Copy 54 78 0 2 r[78]=r[54]
139 Copy 55 79 0 2 r[79]=r[55]
140 Copy 56 80 0 2 r[80]=r[56]
141 ResultRow 58 23 0 0 output=r[58..80]
142 Goto 0 117 0 0
143 Halt 0 0 0 0
144 Transaction 0 0 652 0 1 usesStmtJournal=0
145 Integer 43385 6 0 0 r[6]=43385
146 Goto 0 1 0 0
sqlite> explain query plan WITH RECURSIVE "comment_tree" AS ( SELECT "comments".* FROM "comments" INNER JOIN "stories" ON "stories"."id" = "comments"."story_id" WHERE ((stories.id = 43385 or stories.merged_story_id = 43385) and parent_comment_id is null) UNION ALL SELECT "comments".* FROM "comments" JOIN comment_tree ON comment_tree.id = comments.parent_comment_id ORDER BY depth desc, confidence desc) SELECT * FROM comment_tree;
QUERY PLAN
|--CO-ROUTINE comment_tree
| |--SETUP
| | |--SEARCH comments USING INDEX comments_parent_comment_id_fk (parent_comment_id=?)
| | `--SEARCH stories USING INTEGER PRIMARY KEY (rowid=?)
| `--RECURSIVE STEP
| |--SCAN comment_tree
| `--SEARCH comments USING INDEX comments_parent_comment_id_fk (parent_comment_id=?)
`--SCAN comment_tree
title
let's remember to ask the tests how many things I broke
post-stream
steal all grayhatter's good ideas in https://github.com/GrayHatter/verse
fix awesomewm tab bar order on streaming layout; streamable always first
Transcripts are generated with whisperx, so they mistranscribe basically every username and technical term. They're OK but not great, advice appreciated.
Recording
02:38Oof. Boy, my voice is rough. One sec.
...48Guess I haven't been talking this morning.
Let's see how that holds out.
Alrighty.
So howdy.
This is the Lobster's Office Hour stream.
This is Lobster's.
I'm Peter.
I take care of it.
And then let's see man I am rusty.
pushcx This is Lobsters office hours, ask questions anytime!
So let's see, we want to say in the chat that this is upstairs office hours questions anytime and when folks are not asking questions.
I work on maintenance of the site.
So let's take a run through the holes, which I think are quiet, and the issues, which I think only have one that I haven't looked at.
Let's open.
Yeah.
Working on this performance might be a really good thing to do.
And issues just didn't load.
OK. OK, GitHub.
Are we having another outage?
All right, worked on the reload.
I wonder what happened there.
04:11Why do I think there was a newer issue? Oh, yeah, I fixed that one. OK. Cool. So there was Thomas's looking for performance.
...29Speaking of performance, I could fix that.
...43glamourousbbq code
Sorry, my windows are out of order.
...59yeah good morning glamorous so let's see topics you know pr one issue the thing that's always been weak on the site is that we have no real ongoing performance monitoring there's like when people complain in irc
That's our performance monitoring.
Not exactly a robust professional solution.
All right, so there's this.
twitchtd o/
And then the other thing I was going to look at was... Hey, Thomas.
05:59that's that's plenty to keep me busy for a couple hours, even if nobody asked questions.
06:14was 27 yeah.
...40yeah you know i was just looking at this no
07:18Was it analyze? There is a SQLite query. There's analyze. That gets the internal stats that you're thinking of, Thomas. SQLite stat one. That's what I was thinking of.
08:21Yeah, that's really just getting integer sizing. So this really is specifically for the query planner. And I don't know for sure that you could restore it to a new database.
...50twitchtd you can insert into sqlite_stat
twitchtd 1
So even if we exported this data and dropped it into a database that you created fake data with... Yeah.
Yeah, I see it.
09:15I guess, as long as you didn't run an analyze in that it would keep the data and then the query planner would use the production data. That might get you what you're looking for.
...42twitchtd but then you would have to analyze the query plan which isn't that easy
twitchtd but might be a good option to try
I'm trying to think because it's not really a performance test because the data isn't really there.
Yeah.
Yeah, the analyze of a core plan is beats nothing.
10:30with so what i'm trying to think is like is there a way to take and recreate prod shaped data but that feels like such a rabbit hole of well
Tooxx_96 hello how are u doing
straightforward thing to do would be like loop over the existing tables and replace all the usernames and the comment texts with faker calls right takes a while to run but you only have to run it once and that's fine but then howdy toques you end up in perverse situations where like
Your comment text is a little bit bigger, so it spills onto another database page.
Does that change the perf that you care about?
I guess probably not, but that feels like a really brittle solution that's going to end up with.
So it's going to end up with like accidentally leaking data as soon as you add another column to a table.
Yeah, I don't have a great answer here.
It feels like somebody must have come up with a general purpose answer and I just don't know it.
So let me tag this as a feature request.
13:30twitchtd I was initially thinking a rails active_record hook that would get the query plan for every query ran during a test and analyze every query but I gave up on that, I might give it another go though
le_romain_ethique hi
How do you remain, remain ethic?
Is that like ethical lettuce?
I was initially thinking of Rails Active Record hook that would get the query plan and analyze every query.
That seems pretty good.
14:00le_romain_ethique it's like ethical roman
Tooxx_96 beside you and chaelscode is there others rails streamers you recommend? that i maybe missed
yeah i don't have a really strong idea on that one hmm well i've read some roman history and i suppose being an ethical roman perceived from a modern standard is a
Pretty tough, so maybe you are a unicorn.
Between me and Chails Code, I actually don't watch any programming streams, so I don't have any other to recommend.
Twitch has the, there's a Ruby on Rails tag.
I have no idea how active it is.
I don't really watch streamers.
I barely remember to run a raid at the end of my stream.
15:08This one is bugging me every day. As long as we're talking perf, I want to tidy this one up and then maybe come back to the SQLite performance problems we had or Thomas's PR there.
...47Yeah, so there's this style user link. Is this the one that care? Yeah, it's the you invited. That looks at, yeah, see this is.
16:07So it's a helper, so it's allowed to just go look at the user global variable.
...25le_romain_ethique did not see ruby code for 25 years
How many places do we use styled user link?
Didn't see Ruby code for 25 years.
Funny, that's almost how long I've been writing Ruby code, or I have almost been writing Ruby code that long.
Oh, I don't know if the mic picked that up, but here comes Big Sur.
Big Sur, you wanna hop up on your filing cabinet and make this a cat stream?
Everybody loves a cat stream.
We'll see.
Maybe he will grace us with his fuzzy presence.
Good boy.
17:18So many places we use this. So do I want another helper helper that wraps this one. I don't think I want to touch all these call sites.
...52I hate passing Boolean options to functions. Okay, so.
18:12So I think I wanna just move this to
...26Yeah, all right, hang on.
...35There's what, 30 call sites? 33? How many of those are tests? Five of them?
...58le_romain_ethique lobsters is my rss of choice (I'm xcombelle there)
Oh, hey, XCOM Bell.
Nice to recognize you.
I always wondered about that username.
Tooxx_96 are you on linux ? if yes which distro plz
Are you a fan of the XCOM games or is it something else?
le_romain_ethique My name is Xavier Combelle
What was it?
19:26Ah, you were Xavier Combell.
...35pushcx https://archive.org/details/xco…
Yeah, so I always read it as XCOM Bell, like you are a young woman who likes to play XCOM.
Terror from the Deep, that was the really good one.
They polished the UI a little bit.
Great game.
20:14You know, it's funny, if we didn't have global variables in Rails, I feel like I would have caught this bug as I had to touch every call site.
21:04So here's another style thing, as long as I'm distracting myself. My sleep got disrupted, so that's why I'm chewing on very obvious code stuff, is I really prefer keyword arguments. We use them in one or two places in the code base.
...30Yeah. We have... two functions that use keyword args. That would be such a big refactoring for almost nothing.
22:09grayhatter_ why is this page cached logged in?
grayhatter_ (define slow)
grayhatter_ then I don't understand the issue title
it's not the well maybe it is so this user or this styled user link that depended on the global variable forced putting the user into the
grayhatter_ blew up?
cache key so view user three yeah so here we started caching it for the viewing user and that just blew up performance so that's what i'm ripping out
23:10grayhatter_ as it, everything is worse?
loading that tree once.
Yeah, because loading that tree once takes a couple of seconds now.
24:10Take it.
...17Attempted to force.
grayhatter_ cache the page as logged out, and then load the (you invited) tag client side?
Being explicit all the time.
Given that we just are chasing a bug caused by the global variable.
Tooxx_96 as a beginner learning ruby and rails do you recommend i stick to the default authentication in rails 8 or go with devise ( i see it s used a lot )
I've tried to avoid.
Making the you invited
Well, I've tried to avoid making anything, especially JavaScript base.
...48Nowadays, I would say use the built-in stuff instead of devise. Unless you have a specific need for like devises awath functionality.
25:07The stuff that Rails has with the like has secure password and the other stuff is probably good enough. It has that core functionality.
...31grayhatter_ separate the data from the state? pre-gen the userlist, and then only crawl the invited tree?
Tooxx_96 ok thnks
Pre-gen the user list and then only crawl the invited tree.
...50Yeah, gray header, there's kind of a...
The way caching works, we can't pre-fill the cache, which is the thing I would really like to do.
Tooxx_96 are u on rails 8 ?
I mean, I would also really like to move the cache over to its own SQLite database rather than having it be files on disk.
grayhatter_ I'm thinking outside the caching layer, inside the logic layer
But after that, I would like to be able to pre-fill the cache.
grayhatter_ I'm doing something similar for my commit heat graph on srctree
pushcx https://github.com/lobsters/lob…
grayhatter_ the heatmaps are cached per repo/per commit
yeah too so we're open source and you can pull up our gem file so the gem file lock has all the specific versions but outside the caching layer inside the logic layer
27:19grayhatter_ so when a repo updates, it doesn't recrawl all the other repos
grayhatter_ it's cached is the logic itself
Oh sure I follow, because the the heat graph is the combination of all of the heat graphs on all of the users repose effectively.
...40yeah you know years ago.
grayhatter_ it's looking for every commit with my email across all 40 repos https://srctree.gr.ht/user
Not Maria db What was the name of that.
28:13grayhatter_ there's also the zig repo (hidden logged out) which takes multiple seconds to crawl
Why is there a story about a thing that hasn't happened?
Oh, two months ago.
...29Noria, that's what it was.
...52Is it this one? No, this is an interview.
29:07pushcx https://www.usenix.org/system/f…
Ah, yeah.
This is the one I was thinking of.
So we gave some usage stats to this database team because they wanted to optimize MySQL for...
having internal caches for subcomponents of queries that were not explicitly managed views.
Because the go-to performance fix in especially dynamic code is always memoization.
And once you see that, it's like, oh, so we do
We're memorizing at the page level.
We're memorizing at the query level.
We're memorizing on individual fields.
We do it so many times.
And it is not quite architecture astronauting, but it's not terribly far away to say that we're doing the same thing in our templates as we're doing in our chunks of our database queries.
Oh, the other big one is calculated columns.
30:43I'm going to just call that function user invited by.
31:21In which case I should call that U.
...44Maybe swap these, trying to think what reads a little better.
...55It's funny how non-OO ERV views are by default.
33:11graefchen Heya limesHi
Yeah, and since ERB functions are just...
So I've been thinking a lot about writing my own framework lately.
being, I don't know, the last six, eight months.
grayhatter_ in ruby, or are you considering other languages?
And the way views have been coupled to views, the way ERB templates are coupled to controllers via instance variables makes it so easy to shoot ourselves in the foot with this sort of thing where
34:08I was considering writing my own language, but then I found Gleam.
...21And Gleam is like 80 plus percent the language I designed.
...37So let's make a... Maybe I don't want these at all here.
grayhatter_ I wanted to encourage you to use verse, but instead I'll just suggest you might wanna steal a few ideas from it
Yeah, you know, especially because they set up these time bombs.
Let's just only put them on places where people are talking.
So like direct messages, comments, and I guess profiles.
And then like, that's it.
That'll be done.
What is verse?
I don't think I've heard of it.
grayhatter_ it's the web frawework I wrote
Oh, it must be your thing, huh?
500 commits, you've been at this for a minute.
35:43grayhatter_ I'm kinda proud of it
Well, I definitely prefer this kind of builder approach to templating because it rules out just giant classes of bugs that we constantly run into.
36:04grayhatter_ it's typed at comptime
Oh no, this is interesting.
Oh wait, so this is,
message features feature how does this for features.
grayhatter_ it's so nice to edit an html template, and then it fails to compile, because the call site is missing an arg
Oh, I see what you're doing.
That is not what I was assuming just from skimming the code okay.
...36grayhatter_ it's caught so many bugs
What is zigzag.
...42grayhatter_ very end of the user lang docs
I'm guessing you're not referring to the hair salon in the North Hales of Hollywood.
...53grayhatter_ https://ziglang.org/documentati…
The very end of the user line docs, okay.
37:03grayhatter_ exactly
Ah, so it's like the zen of Python.
Communicate intent precisely, edge cases matter.
A reading code of a writing code.
That's a smart one.
Only one obvious way.
Runtime crashes are better than bugs.
Compilers are better than runtime crashes.
Incremental improvements.
Avoid local maximum.
Reduce the amount one must remember.
Sure.
Resource deallocation must succeed.
grayhatter_ that's why it's a rule
I can't think of a lot of ways that you can fail on resource D allocation, but I've rarely written code and memory ran managed languages have just very little experience on that side.
Sure.
38:11So where.
grayhatter_ I'm sure there's something out there that panics when deallocating something with an unexpected value
So when you say don't have hidden control flow.
i'm guessing you're thinking about this kind of communicate intent precisely.
...41grayhatter_ there's no middleware
grayhatter_ there's no way to trick verse, into routing something in a weird non-standard way
yeah before i ask a question about the middleware i'm trying to ask about what part of this are you referring to because i want to make sure i understand
39:18Maybe favor reading code over writing code.
grayhatter_ there's also no "logic" in the html templates
All middleware patterns break this rule.
...36Builder has full control over the request frame and the call to the endpoint.
If you need to inspect or alter the response before it's returned and understand the downsides, you could implement your own middleware
grayhatter_ templates don't have an if statement, that's logic, and belongs in the code, not the template
Hmm.
40:05I follow that makes sense.
One of the things i'm chewing on a lot is is middleware because the straightforward thing to do it, you know the rails thing to do is pass a request through layers of middleware or an array of middleware and then they mutate the request or they mutate the response on the way back out.
grayhatter_ yeah, verse will never support that
And gleam, all variables are immutable by default.
And there's a problem in HTTP forwarding where, I mean, we see it even in our code base here where,
grayhatter_ (for some definition of never)
engine X are now caddy forwards requests to rails and it rather than wrap up the request into an envelope.
let's see it must be an application right.
41:22Where are we.
...59grayhatter_ another example of hidden control flow that I can't stand, is doing things like adding, checking, or injecting CSRF via middleware
let's say there's a four.
42:07No.
...18I'm not remembering the exact line of code and maybe I got rid of it, but we used to have something
that checked the remote IP and then it has to look for an X forwarded for header.
grayhatter_ the "correct" solution to that is define a form generator that will follow it's own rules, not shoehorn those rules into the request state/routing
And mutating the request on the way in is super convenient if you want all of those mutations.
grayhatter_ lol
but they muddle whether you are looking at user data because users can write their own HTTP headers and do, especially the malicious ones, right?
I mean, I see the probes in the prod logs every minute.
43:17And so working by mutating the request on the way in is like,
grayhatter_ yeah, exactly that!
the happy path it's very convenient but it opens you up to lots of security vulnerabilities and we even see it down at the layer of our reverse proxy caddy talking to rails where rails has to look back at the request sometimes and be like well do we think this header came from caddy or do we think this header came from the user and then rails and caddy have to be kept in sync about
which headers caddy is going to provide like x forwarded four and especially it's tricky with the headers that get repeated or appended to like i want to say x forwarded 4 has a like a list of ips because a request could have been forwarded multiple times
And then you have to trust that the user didn't modify any of those in transit, which of course you cannot trust.
44:27grayhatter_ I've had to "fix" something in that class of problem so many, many times... I'm so over invasive middleware things... it makes it impossible to maintain
And mingling your control plane and your user data just leads to endless confusion.
Yeah.
...44grayhatter_ all the docs are the README, and the examples
yeah so does verse have docs somewhere that i could skim it has examples request user data hey what were we just talking about okay well i will have to check this out thank you
grayhatter_ oh, I should update that, I added a slightly better API for use input validation
grayhatter_ :) cheers
and i will borrow all of your good ideas because that's how we get anything right we'll go steve jobs styles and we'll steal all the good ideas thanks for sharing your project i appreciate it it's
I'm in one of those phases where a lot of stuff is settling out and I'm figuring out what my priorities are going to be.
That's going to answer whether I decide to take on writing a web framework.
46:03grayhatter_ it's a lot of work
So this one is.
Oh, yeah.
47:05grayhatter_ I actually forked this out of srctree because I wanted it for something else, and realized it was good enough to stand on it's own
Well, you know, that's where Rails came from.
That's where Django came from.
...21graefchen I am currently playing with the idea of writing a static site generator for a git repo, and that already is *some* work. I assume a full web framework will be much much much more work. limesO
I'd have to come up with a clever name.
I definitely don't have a good name for a web framework.
So stories list detail.
...40grayhatter_ graefchen you should do it
Yeah, great Chen and a big chunk of what I'm thinking about is blurring the line between a static generator and a web framework because the sense of please steal my good ideas.
48:11If you think about like in an SSG, you have like, just to write this in an FP style, you have like a blog post and you have a template and then your resulting is like an HTML file, right?
grayhatter_ the static site generator I mean, a full framework is super deep... and not a good project if you don't wanna dogfood it
grayhatter_ verse would be so much worse, and unusably buggy if srctree didn't live ontop of it
And a framework, like if, so if we made like, I run my blog with Hugo versus Rails, like the only thing added is, well, the blog post lives in a database, but this is still a fixed input.
Right, so I've talked about how a pain that we have is we have to constantly refresh queues like the homepage and the individual story views because they have things like four hours ago on them.
They have these relative timestamps, which means that the signature for our page is
DB plus current time plus template gets you HTML and you know, we cache that so it's a file.
So if a
50:01graefchen I am currently way to much into trying to port the ed editor from the bsd repo I found from C into Odin. It is fun. But also hard and rewarding. And I still need to create a presentation about Dolni Vestonice for my university. limesLurk
Codebase forced you to think about this function signature of producing a page, and especially if it had a notion of controlling side effects and writing pure functions, then your individual pages
would have type signatures that told you whether they could be cached and for how long.
And if you don't include something like current time or the side effect of a database, you have a static site generator.
51:35graefchen @grayhatter_ oh yeah. the static site generator I will 100% do. and for fun most of it in posix shell, because that sounds like fun. limesD
wants to grab that.
...49When you see similar stories, do you care if you invited the person you shouldn't be able to see because new users can't really resubmit.
grayhatter_ have you heard of bash-stack?
yeah let's keep that one simple oh speaking of writing signatures where's my
52:38graefchen I spot μF limesPoggers
pushcx https://github.com/fsharp/fslan…
I have never heard of bash stack.
Oh, you must be asking Graf Chen.
And what is micro F?
...52graefchen microformats
Speaking of writing type signatures, this was just a great, great comment on why F sharp does not include type classes.
I liked this a bunch.
There's a bunch of like,
graefchen @grayhatter_ not before you mentioned it.
very theoretical and very practical implementation details put together in a very thoughtful way.
I got to it from that Scatter Thoughts blog.
I think that's Matt Kalad is the author.
Oh yeah, and we do have some micro-formatting stuff in here.
We try.
I don't know how much effect it actually has, but.
53:57Definitely here.
54:05graefchen maybe I have seen it sometime, but I cant 100% remember it. limesLurk
And then definitely not in the user tree.
Definitely not in the user list, which this can also lose.
What is that for?
...44grayhatter_ perhaps back before it got a lot more depth, I remember badcop putting a lot of work into it for a few weeks, and then all of a sudden it was actually a usable stack
Oh, it's the filtering.
That's fine.
It sounded like a user.
It's a lot more depth.
All right.
So single detail, yes.
55:10Individual comment. And this one, I mean, that's just going to be a heinous inline partial, but I can do it myself.
...37Search.
...43graefchen It also uses on of the languagess I havent worked much with, but that i rally like: awk. limesPoggers
koffiedrinker just dropping in to say: thx for lobsters, i visit it every day for finding new interesting things to read :)
Why does search print?
...53You're inlining list detail or single detail, aren't you? Yeah. All right, so we got to prompt that to run. I can't just half-ass it. Hat requests. That's a pain in the ass too, because I actually wanted to have caching on search results. Yeah, that's awkward.
56:52I have not been thinking about these user arguments. I'm going to have to run through the diff again.
57:58graefchen same, and what I also very much like about lobster is that you dont need an account to see most of the things on it. not like other websites limesYay
oh thanks coffee drinker appreciate you dropping in and the kind words so let's add for you invited and make sure those are correct and there's one that's not
58:31grayhatter_ @graefchen yeah, it's weird how it's a nice surprise when a site treats users with respect
On mail message user, there's my original.
This one definitely wants to be story user.
...47And hey, one of the other fun features of Lobsters is you don't need an account to set up filters if you really don't want to see Perl.
59:06graefchen respect? whats that? /s limesGiggle
Okay, so that'll be the merge story user, yeah.
...23You know, I'm a little iffy on respect, and part of that is because I read about video games online, and there is a weird thing where
...39gamers write that they're not being respected by the developers which is as far as I can tell just an all-purpose complaint that the developers did something I don't like and it must be some kind of like honor culture thing that I'm too American to get where they phrase it as like, ah, he has disrespected my honor. I have lost face by this interaction. I really don't get when folks talk about respect in video gaming.
01:00:40graefchen Many gamers honestly can fuck themself. Parts of it is really some of he worst stuff on the Internet. limesHeck
grayhatter_ lol, I'm sure some of them are trolling... but a lot of the AAA companies treat users like cattle, so I suspect it's a lot of semantic widening from people complaining about them
But you know brain assault on that because, like not even writing my arguments to my functions correctly, even though I just wrote the functions five minutes ago so.
...57yeah I don't know I feel like they're getting treated like customers and. Maybe you shouldn't have a lot of your personal identity bound up in whether you're a good customer to a company.
01:01:21I think that's all of them.
twitchtd companies only want to see what's inside your wallet
Let's ask the tests how many things I broke.
Every once in a while, I remember to do that.
...47grayhatter_ loot boxes? microtransations? idle games? always-online? root-kit based anti-cheat? banning whole countries without warning?
All right, so there's going to be something in, like, list detail or single detail or comment that's broken to be replicated across, you know, half the test suite.
Missing keyword U.
How did I get that one wrong?
Have I been just pasting the wrong thing over and over?
I called it inviter.
01:02:20yeah so anyways in case you need another reason to like not take my opinions too seriously look at that blog i just wrote
01:03:02You know, idle games, I actually have kind of a soft spot for idle games.
I think it's an interesting niche that plays with time in fun ways.
I have no interest in the idle games that have microtransactions to speed up the idle.
But I have played a couple of them.
What was that one that knocked off RuneScape?
grayhatter_ yeah, I enjoy real time games that have a chill vibe
pushcx https://store.steampowered.com/…
grayhatter_ but I wouldn't call them idle games, because everyone just assumes idle means microtransaction hell... and they're right
on steam now oh what was that melphor that was the one yeah i played this for a long while years before it came to steam
graefchen I am playing much Fallout 4 again in the last time. limesSit
it's it's very clearly you know the game is an electron amp because it's browser based and you have a little rpg character and i didn't know it was runescape based at first until i've been playing for a couple of weeks and then i was like wow how have you guys not gotten sued for copyright infringement you are knocking off all of the core everything but i know you can't copyright
grayhatter_ this sounds like fun
gameplay just individual terms or like art assets and they they sure look like they redrew all their own art assets but yeah i played this game with a co-worker for god i don't know must have been close to two years and it was just kind of fun where every day or two in the work slack we would chat about oh yeah i got my foresting skill up to 99 and all that kind of stuff
This is a very chill game.
I think...
Okay, so they charge a couple of bucks for it.
Oh, it has a DLC?
That's a little funny.
grayhatter_ I wouldn't call this an idle game, because it's not exploitive... but it meets *most* of the requirements
I want to say it was free to play, and then it had an optional pay tier that got you more inventory slots.
It was like paying to win a little bit.
graefchen Alos the publisher is Jagex Ltf, the creator of Runescape. limesSit
But I was not so hardcore about it.
I don't really care about winning, and it was just an excuse to have a conversation with my coworker.
So, you know.
grayhatter_ I don't mind paying down the grind if it's not a tactical advantage in PvP
The publisher is Jagex.
Oh, that's new.
I swear it was a totally solo guy when I saw it.
Because when I saw it, it was under fairly heavy development.
When did I see it?
It must have been...
It must have been 2019.
graefchen Pre 2021 limesSit
And then also 2018 and 2020.
grayhatter_ click on the publisher
Or, you know, somewhere in there.
Yeah.
01:06:20grayhatter_ see what other games they have
Yeah, no, I understand Jagex makes it, or made RuneScape, but this guy, Malks, is just a guy?
grayhatter_ ah... I was gonna say, I bet that's how they got around it
I guess he came to some legal arrangement with RuneScape.
Because back when I played it, I'm pretty sure it didn't say that it was inspired by RuneScape.
Anyways, cute game.
...51grayhatter_ I mean, sounds like a good deal for both of them
grayhatter_ I hope
was kind of funny to like just have a game that i played 30 seconds a day for two years i i think that's an interesting way of playing with time and very closely related is incremental games and the first time i saw one of those i was like oh so it's it's the progression of a system but none of the gameplay and there's stuff like that
Progress Quest was maybe the first one I saw.
I don't know.
Maybe it doesn't qualify as a game.
I don't want to go down the rabbit hole of figuring out what is and isn't a game because, boy, am I not an expert on there.
And actual experts have failed on that one.
Because I read books on game design and, like, everybody wants to waste a page or two on, like, this is a game and this is a puzzle and this is not a game.
And that's, boy, am I not interested in those.
I skipped those pages.
01:08:05grayhatter_ there's a diplomatic tactical game like that where it happens in real time, and it's very much a social deduction influence game... but also minor turn based warfare
So anyways, Grey Hatter, you said you wouldn't call Melvor an idol.
Is that solely because it's not exploitative or...
were you using the term idle to mean something else?
Like what are the key features of the genre to you then?
grayhatter_ yes, because I don't want to confuse people and have them think it's like a mobile game that tries to get you to buy "tokens" to play a bit faster
And I'm super curious about a diplomatic tactical game that happens in real time because I have like three pages of notes and a half finished prototype for a game like that that would fit that description.
I don't know about the social deduction influence game, although that was a little bit of a part.
Ah, yeah, yeah.
graefchen I am still madly in love with the game "Return of the Obra Dinn", such a good game for Lucas Pope limesLurk
I wonder if my Melvor account is still around.
That guy could have been chopping trees for the last five years, you know?
01:09:21grayhatter_ you can't win without alliances, and the alliances aren't enforced by game rules
Oh yeah, one irkham thing about idle games is they have really strict, like, you have to log in every 12 hours or your guy stops doing things.
That's super annoying.
Oh yeah, Return of the Obra Dinn.
What a gorgeous game.
grayhatter_ so A and B could agree to attack player C, but A could actually attack B instead
We've had that long post on how its graphics work once or twice.
Or maybe there's two posts, one by the author and one by somebody recreating it.
...53grayhatter_ maybe?
Oh, Grey Hatter, are you just talking about the actual game Diplomacy?
01:10:16grayhatter_ I'm looking for the video I saw it in
pushcx https://en.wikipedia.org/wiki/D…
There's a lot of, wow, Wizards publishes this now?
All right, good get for them.
But this is just a classic game, originally created in 59.
Yeah, this is actually the box set that I have over on my shelf.
And this is one of the many online versions of Diplomacy
grayhatter_ I think maybe I am
But yeah, a lot of people play this game at a pace of one turn per day or one turn per couple of days, one turn per week.
Yeah.
Yeah, this is a...
There are a couple of things in the rules that are a little bit fiddly and ambiguous, but there's actually a great... What is it?
There's a great effort to canonicalize the rules.
Who is that?
Let me see, rule variations.
Is this the site I'm thinking of?
01:11:31graefchen Lucas Pope also has many cool videos on his youtube channel, like this 2:30 h massive video: https://youtu.be/90vqCKEEj3s limesLurk
This might be the site.
So...
Because diplomacy has simultaneous turns and you have to resolve conflicting actions, there are a handful of ambiguities in every version of every rulebook that has shifted in the various printings of the board games.
It's a shame Chamlus is not on the call because this is the kind of thing that would nerd snipe her and she would find this.
But ages ago, I saw somebody write up a canonical list of rule interpretations and rules because there are very clear differences between some of the versions.
01:12:53It's funny how this is alternating between the board game and actual diplomacy.
01:13:04I include armies, does it? Diplomacy rules. Yeah. Yeah, so this is just a copy of the rulebook. Someone someone has done a wonderful bit of domain modeling where they for all the versions of the rules that have been released over the years, figured out all of the ambiguities in those rules and said, Like, okay, this is a self-consistent way to play. This is not. And it was just such a nice exercise in game analysis and domain modeling.
...58People have been playing diplomacy for a while, but there's...
pushcx http://www.dipwiki.com/index.ph…
a bunch of sites that are still just http oh this is by the author wow i gotta read that where's my personal browser read that later throw that link in there anyway easily distracted because i'm tired
01:14:44It's funny, I wasn't sure whether I wanted to do keyword arguments, but they sure made for an instantly readable error message.
...59If I had done positional arguments, well, I couldn't have had the bug. But it might have leaned towards failing silently and just passing a nil.
01:15:23Okay, so this is just a test and that's correct. I shoved that out to its own test, our own function. So let's update that.
01:16:16Actually, I don't think we need most of these up here anymore. So we need the viewing user and the unrelated user. That'll do it. Yeah.
...59chamlis_ https://www.petermc.net/diploma… ?
chamlis_ had the stream on in the background, but you have my nerd-sniping weak spots down pat
that get rid of that having to close a global variable aha chanlis yes the diplomacy adjudicator test cases thank you for letting me nerd snipe you this is the exact link i was thinking of yes i had forgotten that it was produced to
provide a set of comprehensive test cases and like help settle arguments over various programmed implementations of diplomacy.
And it is kind of great that this guy has been working on it for 25 years now.
I want to say I first saw this somewhere in the early 2000s when I learned diplomacy.
that's wonderful that it's still going added unwanted support oh no wow yeah this oh man this has gotten long long but so it has a copy of the rules and then like
there are various algorithms for how you resolve the simultaneous turns and like whether you loop over the players in order or whether you loop over the countries and then loop over the players changes the way you're likely to resolve some of the ambiguities and it's very i would love to see what somebody could do with this and an smt solver
01:18:57Anyways, thank you for letting me nerd snipe you. That's so delightful. This is such a great article.
01:19:15Oh, and he has an active list of upcoming diplomacy tournaments that is very clearly still live.
Huh.
Oh, there we go.
I can play.
The Weasel Moot?
What a great name.
The Windy City Weasels?
chamlis_ yeah this feels like the sort of thing you could write a TLA+ spec for
How did I not know that we had a Chicago Diplomacy tournament that's been running for 20 years?
Wow.
Okay.
Well, if I ever want to make some friends and enemies...
01:20:01My board gaming group used to refer to that. And yeah, TLA spec. Maybe speaking of nerd sniping, maybe you can nerd snipe Hillel with it. Every time I see something related to like, can you model this? I think of nerd sniping him. Equal.
...25So this one just becomes empty string because you didn't do it.
...33For new invited user. Yeah, OK, so. You invited. And you is going to be the viewing user. And users can be the new invited user. So that will equal. Just. This part.
01:21:06Let's get the palette syntax. It's not nested. That's not a list anymore. All right, there we go. Rubikop took care of it. That's a reasonable place to line break it. I could line break it there. Yeah, that's fine. And then for a new unrelated user, we will say. OK, and then up here we want to add back.
01:22:08We're testing something else here. Hold on. All right.
...42Do happy path and then sad path. And then this, some version of this wants to be up here. So this should be a link to, and then it should say it's the new user.
01:23:01Now we need a new unrelated user up there. That ought to do it. I mean, I say that ought to do it, but given that I'm missing obvious stuff, probably we're going to have a silly test failure and have to sort it out. Yeah. Okay, so nil instead of empty string. I expected an actual logic failure.
...42I'm trying to think. 2 for true, and 47. What did I rewrite?
...58Oh, on Earth? That's a weird edit.
01:24:08Yeah, so it just doesn't show it all there.
...17let's try and have the same style of spacing at least okay and one more time let me run the whole suite because this thing is just buckshot over the view templates
...46And let's double check the user controller has a caches page. Yeah, caches page, true, if cache page, OK. How was that caching? Because this constant should say, if no user was logged in.
01:25:17It's now cached per user.
...26Oh, that card can just go away, I bet. Yeah.
...43I got it while I was editing, OK.
01:26:00Is there an extra link there on the user tree?
...13No, we must have cleaned that out in the bug fix then. OK.
01:27:10guess we'll see that run just in case so one of the reasons i wanted to take this one on was the user page is pretty popular more than i had anticipated and It's become very slowly it's become such an expensive page just because there's 200,000 users in the database ish. That I don't like seeing spike load on prod.
01:28:08And the bug just went purple and closed. Good. All right, so there's that deploy running. So if folks have any questions about the site, now is a great time to ask. Otherwise, I can... Oh, man. How likely am I to make meaningful progress on SQLite performance in my personally degraded state.
...48twitchtd for the perf test I think I'm making some progress with the sqlite query plan analysis during test runs. Though one thing that sucks is that the output query plans are not guaranteed to be stable
Well, you know, I wanted to do experiments, so probably I'll be OK.
01:29:07twitchtd between diff sqlite versions
All right.
Playing with my phone.
...25Oh, that's an interesting hassle.
I mean, you sort of wouldn't want the output query plans to be stable because ideally they'll be getting better over time, right?
twitchtd I mean the format
but I can see how that's a hassle.
Oh, the format.
Oh, that's just inconvenient.
01:30:16Huh. So I reloaded the pull request page, and this guy jumped back up. Did this pull request get worked on?
...40Doesn't look like it.
01:31:13So Thomas, I was thinking about experimenting with replacing the user threads function that has the complicated recursives query with the naive one plus N version. because SQLite doesn't care. I don't remember. I didn't leave myself enough notes. I see you merged in main.
01:32:09Is that where you were focusing your performance attention?
...26GitHub, give me the text.
...50twitchtd ya, I wanted to get a test for this reproducing the problem before I do an actual fix for it
twitchtd that's why I opened that "how do I automate perf tests" issue :)
graefchen i recently watched a video about sqlite and one thing that D. Richard HIpp said got a chuckle out of me limesGiggle
Hmm, I mean it does make a lot of sense to Actually have a baseline As opposed to just making things up Is D. Richard Hipp one of the core authors?
01:33:20Yes. Oh, there's an untold history. Well, I guess they can't call it that anymore.
01:35:06graefchen It was about how SQLite is Open Souce but not Open Contribution and he said, in a riff to "Free as in beer, Free as in speech and free as in puppy" limesGiggle And when talking about code contibutions it kind of fits. limesGiggle
I have definitely heard, free as in puppy, about open source software.
And one of the other criticisms from years ago was Linux is free if your time is worthless.
twitchtd haha so true
Haven't heard that one for a minute, I guess because the desktops have an extra decade of polish and, oh man, it's installing a distro easier than it used to be in the 90s and 2000s.
...50grayhatter_ linux is free if being disrespected bothers you
So.
Hmm.
01:36:37grayhatter_ linux is free if you want to install apps that aren't on **the** app store
grayhatter_ linux is free if you want to decide when to update your software
I had two of these for a reason.
Why did I have two of these?
It was so that I didn't accidentally wipe the database, right?
01:37:12grayhatter_ just because the time spent on linux is making it better, doesn't negate the fact that you have to spend just as much time fighting every other OS, only to make it slightly less shitty
yeah i i have a windows install on my desktop for video games but what it's slowly turned into is like once a quarter i reboot and i install a bunch of windows updates and then there's weird ass garbage and
grayhatter_ lol
grayhatter_ it's the point of the thing
grayhatter_ it's the disrespect, not the upload bandwidth
have to ignore that or try and turn that off or look up like no really how do you uninstall onedrive because it wants to grab all of my files not that i have any files it's just obnoxious and there's the possibility that like if i have to run some windows only device driver update thing i don't know maybe i would actually copy a file to windows and then it would be in onedrive two years from now without me knowing about it which is just ooky
01:38:22twitchtd btw steam on linux is very impressive nowadays
Oh yeah, I mean, where was it?
I gotta find this off stream here.
There was a incredible story.
grayhatter_ twitchtd yeah, it's rare to find a game that doesn't work
pushcx https://www.windowscentral.com/…
Yeah, here is the link.
Let me grab this link.
...48This was just a week or two ago, I wanna say.
But they were talking to Microsoft employees about improvements to Windows 11.
Microsoft views SteamOS as the benchmark and is working to optimize the platform so that SteamOS and Windows gaming performance are comparable.
Within the next year or two, Windows will be able to truly compete head to head with SteamOS in gaming performance.
Sometime in the next year or two, Windows hopes to run Windows games on Windows as fast as Windows games run on Linux.
grayhatter_ that's exactly what I would expect
Which is just an incredible state of affairs.
01:39:48Yeah, so I've got the one friend who's really into PC gaming. And he's the one who constantly is hassling me to make sure that my Windows is up to date so that we play some game and now i i sent him that one and i was just like well when are you going to get higher fps buddy when are you going to get with current performance maybe 2028 with the year of the windows desktop
01:40:59chamlis_ now it's up to WSL to outperform linux for native apps
You know there's actually a. simpler way to do this, what if I just threw out.
The confidence order path string nonsense.
yeah well so there's.
01:41:50it's up to windows subsystem linux to perform linux for native apps oh that's a very funny state of affairs
01:42:37grayhatter_ I just read something about that being true
grayhatter_ windows is using parts of WSL to make it run faster
Yeah, it's just going to be JSON, right?
These answers are 15 years old.
Let's go to the SQLite docs.
01:43:00There's that.
I don't remember if I would have to do it or if it has another array type because
theyagich the solution is to make a new system that will be slower so that they can both benchmark how slow windows and linux can get
twitchtd winux next?
No, I'm thinking of Postgres has a native array type and JSON.
Yeah.
...22Huh.
...29I feel like this list hasn't been touched for a couple of years. There must be a SQLite book released in the last decade Right.
01:44:11grayhatter_ @theyagich isn't that called electron?
Yeah.
...46twitchtd I got to head out, will you be on next week tuesday or thursday again? I'll try to make some progress on writing a quick test
Thomas, I will try to be on next week, Tuesday and Thursday, but boy, have I missed the last three Tuesdays, I want to say.
There's been some nonsense.
01:45:14twitchtd ok, see ya
twitchtd thanks for the stream
Take care.
Yeah, likewise.
Thanks for dropping in.
...28So if we want to grab.
...54So what we want to do is make an array. So we want to say JSONB array with just that one value.
01:46:44Let's reference an element. Well, we want to be on the end of the element. Is there an append? No.
01:47:07does not end with the right element then an error is raised instead of overwriting the arrays enlarged as with replace okay so we want to take
...44And then the new value is going to be. No, it takes the position and then the new value, right?
01:48:04What is this? $1 in brackets.
...14Let's go look at JSONB replace.
...32So I can pass that. And the new value will be this one's . Actually, it's just confidence, not confidence order.
01:49:17It's well contained. Man, that is so much simpler. So then that becomes order by confidence path.
...43Let's see if I made a valid edit.
...51Did not. What are we mad about?
01:50:14There's an extra comma.
All right, that's the first.
veqqio Hi friends
No such function JSONB array insert.
JSONB array insert.
Did I typo this?
And hey, Vic, do I have a SQLite that is compiled without JSONB?
Let's try regular JSON.
But I didn't get an error on the first one.
Yeah.
graefchen hi veqqio limesHi
You're mad about regular JSON array insert, even though JSON array insert.
What am I missing?
There is something obvious and bad happening.
01:51:46Who would mean that, Google?
...53That's just a hallucination, isn't it? Unless they, yeah, that one's a hallucination.
01:52:11Can only be used to fetch data on a virtual table. That's conspicuous. Three years ago, so pretty recent.
...32chamlis_ I think they were added very recently (3.53, 2026-04-09)
Can't use it as a field in your query.
They are table-valued functions.
...43Wait, really? They were added last week? I swear I saw this a while ago. How are you not finding it in the docs?
01:53:13Nothing about Jason in this.
...21chamlis_ https://sqlite.org/releaselog/3…
Old version.
2008.
...39They are indeed. Very new functions. so okay let's find out if rails is using a version of sqlite that is installed in the gem or if it's depending on system sqlite
01:54:24Okay, so now I have that. Let's bounce the web server. So I'm really hoping that it's going to use the version of SQLite that comes from the bundle rather than I have to maintain something. I've got a different error.
...56No such function JSON array. That looks like a typo.
01:55:07Okay. So that loaded in a reasonable amount of time. But we didn't really see this one locally.
...21But that does say I created a valid edit.
...34what has a heck of a lot of comments my go-to oh search engine's broken on this branch oh no i searched comments not stories So our trend is broken on the branch. All right.
01:56:26Here we go. Oh, this only had 96 comments. I don't think that's our largest thread. Let's go ahead and close that out.
...46Well, that did feel like it loaded pretty zippily. We have a reasonable number of comments.
...59So if I change both to JSONB. Feels just as zippy.
01:57:12Where are we spending our time?
...26So this, boy, that sure didn't feel like it took a third of a second, but I guess it must have. All right, so if that took 257, set over in the terminal as well.
01:58:01See the render, I don't see It lists the performance of that individual query.
...16Yeah, the number 257, which is the number of milliseconds that took, does not appear there. Dang it. Just wanted to reload a couple of times without having to click in. And scroll.
...42253. All right, so let's throw caution to the wind and not sample. I just want to see if there's instantly a big difference here between JSON array and JSON B array. I kind of lean towards no, because the difference between those functions is about serializing. And we're using that value in memory and then throwing it away. like it's not even getting selected back yeah 268 we're in the same neighborhood 256 yeah so these two are equivalent that's fine
02:00:14Oh, actually, we can confidence order entirely.
...29Yeah. You don't have to deal with that ridiculous thing. So we can execute an update with the new confidence.
02:01:07So where is the query I just touched? OK, so this one, I wasn't even thinking about it, but I replaced confidence order with just confidence. Good. Let's do the same thing here. So this one becomes a this.
...31And then this one, yeah, just becomes this. You know, if nothing else, I like how much clearer this expresses what's actually happening here. Then this becomes confidence path. And is there any confidence order left? And drop that validation. I mean, I don't want to. I don't want to write a migration to drop this. So keep writing that. Just have a placeholder value.
02:02:25That goes away. I'm mostly deleting it so that things crash if they're still calling it. There's the old migration.
...43bit packing stuff that'll fail spec that can go away doesn't sort comments under the wrong parents when they haven't been voted on the problem there was they had the same parent
02:03:19Think about this a second.
...51This is... I'm trying to remember why I'm materializing this at all, because with recursive...
02:04:31Oh, we have to materialize it because this gathers everything, but then the sort is down on that order on that last line of code there. OK.
...46But these are sorting by the confidence path. Which does not include. The ID.
02:05:31That JSON array insert. Can I give it multiple values to append? Or do I have to call it twice?
02:06:38What a weird error. It needs an odd number of arguments. Not that it needs three arguments, that it needs an odd number. So like five is okay.
02:07:01Add JSON path. Okay, you're doing something clever with function overloading. It's a very C-style interface sometimes.
02:08:25chamlis_ is it ...,'$[#]','new','$[#]','newer'...
So if it just takes anywhere, it's got to be CID.
...56needs an odd number of arguments. It doesn't say if it's the inner or the outer.
02:09:15So I was thinking of, we've got to resolve the ambiguity of what if two parent comments have the same confidence? That's already a bug we've had. And the way we handled it was the confidence order column had two bytes of the confidence and then one byte of the ID, which reduced the collisions. Let's try that. Oh, that's real clever. I see what you're doing there. So let's say a and then.
02:10:10Oh ho, that is the exact thing we wanted. Thank you, Chambliss. Wait, did I not get that from the docs?
...26The path and value arguments can be repeated multiple times. Okay, so that's why it said an odd number of arguments. Okay. All right. Let's make these, let's grab, can I just say JSON array, and if I pass one and B, yeah. All right, so let's make these consistent. So JSON array and comment ID, each one will be effectively a pair. And then this will be confidence path.
02:11:17Yeah, and that's always confidence before the ID.
...32OK. Wow, these are definitely sorted properly now in that it's a tree instead of flat. have said that out loud but it caught my eye that it was basically flat instead of a proper tree so hold on cache world sequel and i do expect this to be in the same neighborhood of you know 250 milliseconds yeah all right so we're still at 255. which is not acceptable to take 250 milliseconds to get 100 comments. That's kind of huge.
02:12:46That's a little painful. I would really like to be able to have a multi-line edit. Do we have full read line? No. No, it's not kicking into multi-line editing.
02:13:10And it's not printing timing info.
...41Okay, so 262 milliseconds. We're in the same neighborhood there, all right.
...58Yeah, 257, all right. So same neighborhood, still bad.
02:14:17chamlis_ I've just seen that sqlite lets you be explicit about the order of the recursive cte, so I think it should be possible to do the whole thing without having a confidence_path?
And since I saw the same thing for appending one value as two, it's tempting to say,
02:15:00chamlis_ if you order the recursive select by tree depth and then confidence, I think?
To show the sorting down into here.
If you order the recursive select by tree depth and then confidence.
What we do need to grab the parent ID because multiple parents can have the same confidence.
...25I was just thinking about.
...33order by right and then we don't actually have to include the id as a tiebreaker we don't have to include the confidence but then this sort is going to do nonsense Yeah.
02:16:23This isn't the naive. This is a bookmark, JJ desk.
...42These are the same thing. Why did I think they were different?
02:17:40not rm it's r for remove no it's r for rename it's a d for delete delete okay oh and i didn't make the edit yeah i didn't do that same thing here let's do that again just so they're Both updated, even if I'm really only playing with the one. I don't like having half an edit.
02:18:20All right.
...31So if we have a new on that.
02:19:35doesn't story threads need to be a relation I bet it does let's find out yeah because we called for presentation on it so I can't really do So what I wanted to do was, like, comments is empty array, and comments is comment.find, story ID, and what is this? Story.comment.where.
02:20:37But they're not going to be in order. I'm trying to get totally imperative.
...55I don't want to keep mutating this array and inserting elements into it.
02:21:27Yeah.
02:22:22I just said return order by descending.
...46Is that missing? I was just skimming these comments, but are they in reverse order? Yeah, so there's three. Yeah, that should have been 7, 18, 42. Yeah, it should have been the other direction. That's okay.
02:24:02So I was tempted to do the sloppy thing and just pass the list of parents down to the view and let the view trigger all the queries to say, give me all your children sorted.
...24But we have all this stuff related to
...33The performance hack to get the comments back as a flat array and then rebuild. The nesting and I'm going to have to rip all that out.
02:25:06This goes away.
...32chamlis_ I haven't tested it but I was thinking about this sort of thing, it's essentially a DFS with ties broken at each level by confidence_order https://pastebin.com/7Mdg6uaD
DFS with ties broken by confidence ordered.
That sounds correct.
And it's probably a lot better that you wrote it in an actual editor instead of the Twitch text box.
Interjoin with recursive confidence as just CID.
02:26:16That doesn't look like it's going to try.
...25See parent ID.
...33chamlis_ sqlite makes guarantees about how it does recursive ctes, which I think makes this work https://sqlite.org/lang_with.ht…
You know, I wonder if rather than order by depth,
...43Well, actually, order by depth can't be doing anything because all comments with the same parent comment ID should have the same depth.
With clause.
Where did my Firefox go?
chamlis_ sqlite processes one row at a time from a queue, and if there's an ORDER BY it uses that to pick the next row
Did my Firefox crash?
Yes, that was obnoxious.
All right.
Let's bring this back.
With clause goes on the end.
All right.
02:27:34chamlis_ scroll down a little bit from that heading
Can be used to write a query that walks a tree or graph.
...57When the queue is not empty extract a single row from the queue to the single just froze the only table in their cursor and run the recursive select.
02:28:37If the orderby is present, it determines the order in which rows are extracted from the queue. If there is no orderby clause, then the order in which rows are extracted is undefined. OK, and then a note. Extract the single row from the queue. Insert that into the recursive. The single row is the only row in the recursive and rerun the recursive select. So you're trying to use the order by to always have the value of
02:29:27I think that has to have the parent comment ID in it then.
chamlis_ further down that page they explicitly mention using the ORDER BY for BFS and DFS
Oh, even better.
02:30:01ChaelCodes Hello! 👋
Okay, hierarchy.
Hey, look at hierarchy.
...14Cords of family tree. Well, that's very familiar.
...27Unidirected graph.
I mean, we are a,
degenerate unidirected graph.
Three, four.
Oh, hey, Chael.
Who was it?
I bet I can't see because I got dropped out of the chat room for a second.
Yeah, I can't scroll up far enough.
chamlis_ heya!
So someone in the chat an hour or two ago was asking for
rails streamers on twitch because they knew you and i and i don't watch enough to know any other rails streamers so if you are that person who asked for more rails streamers maybe chael will know some but you should put your hand up because i don't remember your name after a couple hours
02:31:33ChaelCodes So lonely. It's just us and milesfoob
I'm glancing at the list of users in chat and none of the names are popping out to me as the person who asked so I guess it will just remain a mystery, maybe they'll drop by your stream and ask you directly.
Is it just us and miles food oh.
...58davidofterra I'm here.
Well, you know, in the start of the stream, I was noodling about what I would like to see in a web framework.
And we talked about what... Oh, God, I'm so bad at names at some point here.
ChaelCodes purpleelf does Rust now and monospacementor only talks about devops
At what Gray Hatter did in his first framework.
And...
I talked a little about wanting to blur the lines between a static site generator and a dynamic web framework.
Hmm.
So depending on how some things land in the next.
How long do I have to wait to hear back?
They said like a week.
So realistically, it's going to be like three.
So maybe by the end of the month, I'll know if I'm starting my own web framework.
02:33:19Here's a query.
...26Same as order under Alice level plus one. Resulting in a breadfruit surge.
...44Outlandish recursive query examples. What a wonderful subheading title.
02:34:08veqqio This is a good page, I'll get a lot of millege from it. I've been building up a declarative language lately
So who actually responds to Cindy?
Or reports to Cindy?
Just Fred and Gail, so we definitely don't want that bread first.
...37veqqio outlandish query examples
Check you, which is a good page for, let's grab this text before I forget about it.
Ah, I haven't scrolled down that far yet.
02:35:12Order by depth. Cause the lower levels in the organization.
...38Hm. Yeah, you know, Chamos, I think you're right there.
...54But then how do we, hold on, let's make a commit and play with your query. JJ makes this so easy. Say it's on this with the description in the SQLite for experiment chamless DFS. Great. So we grab this query.
02:36:38And then jump over here. So this becomes this.
...54chamlis_ I presume inner join preserves the order?
So now that we're not materializing a comment order path or a confidence order path or a confidence path,
I have to delete this order clause.
One way to find out, right?
So let me delete that.
And.
02:37:41let's reload that all right we have a typo is there a trailing comma order by clause should come after union all not before well that's interesting that it won't give us the well we don't want to use confidence order in any case we wanted to say confidence desk and this wants to be confidence desk Why can't we sort the parents.
02:38:33I didn't really think about it, but it's example with the hierarchy sort of has. Alice as a single head rather than multiple heads?
...57chamlis_ I guess it uses the same one for both, in which case you should be able to just remove that line
Oh, you think so?
chamlis_ all of the initial select results just go in the same queue
I was thinking change it to a blank base case.
The initial select may be a compound, but it may not include an order limit or offset.
02:39:16So it runs the initial select and adds the result to the queue.
...38Yeah, but if you can't control the order of that initial select, our parents are going to be in the wrong order.
Our root level comments.
If a union operator, does it say anything about reusing the order?
Yeah, if an order is present, it determines the order
chamlis_ I don't think so, they all get chucked in the queue, and then that final ORDER BY tells it which one to pop first
in which rows are extracted oh okay oh no I see how it works because if it order by is how they're extracted rather than how they're added these are equivalent yeah okay okay I see how this works this feels plausible
First order by term does not match any column in the result set.
02:40:38Is depth implicit? It's been so long since I've thought about it. Schema.
02:41:03AnakimLuke greetings smixelLobster
no depth is a column on comments i was thinking that we materialized that the first there's another order here right order by depth actually shouldn't it be depth ascending because i think they were counting down and said well let's
Oh hey Anna Kim, been a minute.
Does not match any column in the results.
I bet we have to select depth.
So let's let's just add that here.
...55ChaelCodes Hi @anakimluke !
OK.
So that changed the error in a way that sounds like that was the issue.
AnakimLuke @ChaelCodes smixelLove smixelBusiness
Okay, so now it wants us to select the confidence that makes sense if it's shoving these rows into its queue.
chamlis_ oh confidence is also the table name, that might be tripping it up
it's not maintaining a reference to the original row, and so it wants us to include the things that it's going to sort by.
02:42:26That really is a very literal description of how that works. Yeah, that seems fine. Okay, so now... Yeah. These look good. These look like they're sorted correctly. Let's check against the prod thread real quick. Where is the...
02:43:13You guys can't see this, but I have some off-screen UI that is just running when I'm streaming, and it's driving me a little bit nuts. All right, so let's take this and now change this back to broad.
...40Can I just make you a new? Yeah.
...53And I am immediately running into the bug I just wrote down. All right, so these.
02:44:06This did not sort correctly.
Owl has a bunch of replies that are not present.
Yeah, this got shoved down.
chamlis_ I don't think the join preserves order
Okay.
...42Yeah, it's gotta be that the join does not preserve order. So can we get rid of the join?
...53Because the only reason I did this inner join is so that I can slap the for presentation on.
02:45:23chamlis_ I guess if you select * from comments that can be the final result set? I'm not sure how that works on the ActiveRecord side
All right, so I guess if you select star from comments, that can be the final result set.
02:46:03I was thinking of just making them a scope. If this thing becomes a scope, can I turn this into just an order clause?
...37Not really, right?
02:47:34Yeah, the hassle with doing the select star instead of just doing the join here is... Right, well, because I mean, it is. I could delete these two lines, delete these. The problem is the Rails Query Builder
02:48:10will no longer know that that query is pulling from comments to be able to apply the for presentation scope to have all of these in.
...31We don't care, right? Wait, this is SQLite. What do we do in fixing one plus N errors in SQLite? For presentation only exists for one plus on issues, so if we threw that away. Right hang on let's put this back.
02:49:02AnakimLuke what are you working on now?
No, it's the inner join that's losing sweet if we said this.
This Anakim performance.
There is an open pull request to migrate us to SQLite, which we attempted to deploy a previous version of and were unable to.
So if we said, the variable name is wrong, right?
But then stories controller.
Let's just throw away for presentation.
Because the whole point of for presentation is to avoid one plus ends and Maria DB.
If we just let all of those happen, SQLite shouldn't care.
And instead of being one query, there will be
AnakimLuke what does "after the great Chicago fire of 1871" mean? 😅
i don't know hundreds many hundreds but sqlite is supposed to make short work of those right so this is sort of a hybrid between chamlus's original idea and the other idea of go full sqlite and one plus n everything oh so there was a funny coincidence
The pull request 1871 was our first attempt to migrate to SQLite.
And coincidentally, the Great Chicago Fire was in 1871.
And when we tried to deploy pull request 1871, it did indeed burn down the site.
AnakimLuke 😅
The added performance load made the site just struggle.
There was a comment thread on it too, about two months ago.
So here, where'd that go?
pushcx https://lobste.rs/s/oz7ebk/lobs…
Stories, SQLite, tag, meta.
Can I just drag that over?
Bang, there you go.
So that has some more info from the day we were trying to deploy.
02:51:34right so let's see what happens on localhost here probably a sql error all right near confidence syntax error right because it probably shoved this directly into so select with recursive this needs to be before the select does active record have a nice way to do that
02:52:09What was that they auto-completed with a person?
...38Yeah, Rails has a with recursive function.
...47Come here.
02:53:13And then reference with another select. OK. So that's funny. It chooses posts and replies. So if I said, format with recursive confidence tree. Just call it tree. And then it wants an array. of two queries. Okay.
...56Well, you know, that would allow us to put back the presentation just to get a like for like. So we'll say, thanks Vupo Cup. So then this first one is Actually, all of those fields are going to be there. So we want to say comment where and then this.
02:54:34And then comment where.
...50How does it refer back? Requires a join? Join to the name of the thing on.
02:55:04That is actually our exact use case. OK, so join or joins? Joins, plural. Always gets me.
...26Let's call it a comment tree. I don't want to fight any identifiers. Join comment tree. Hold on.
...41Comment tree ID equals... Oh. On parent comment ID equals confidence ID? Oh, right, because the name was called confidence. Yep, yep. I'm confusing myself with that table name. Never mind. Getting to the end of the stream and getting more confused on top of being half asleep equals.
02:56:09Parent comment ID. And then this one needs the order by. I will say. just do it as a string depth desk confidence desk all right i'm not ready to delete all that so we'll just say return no such column select comments
...51why are you selecting from stories oh because it was written as a join so we want to say where the story id is that yeah so this does have to join to stories
02:57:25Joins, joins. Can't join comment association named stories. It's story singular. No such table. Select comments dot star from comments. Join comment tree on. Show me the whole query you generated.
02:58:00It didn't generate the with recursive up front.
...28With recursive.
chamlis_ I think with_recursive takes one keyword argument? maybe that's ruby sugar I don't know though
I passed two arguments and it wanted a key pointing to an array.
Types would fix this.
Oh, you got there too, Chambliss.
All right, so syntax error.
I did pull in a couple of strings, so I must have an extra parent.
02:59:02yeah stories id or merge story id and parent comment is null all right what are we mad at and actually does the terminal have the nicer output
...37no that's very confused union all with recursive comment tree as select comments and join stories 1, 2. 1, 2, 1. Union all.
03:00:19Extra parentheses there in the sorting. But not visible here.
...39Give me this. We'll jump over to SQLite.
03:01:11doesn't like a parentheses around the union all second query.
...28So if I got rid of that parentheses. And this one. Well, that's taking a little longer than 220 milliseconds. Or 250. And Rails is generating invalid syntax, which is... I feel like that's a feather in your cap there, Chamlus, that we found a Rails feature that generates invalid SQL. Like, that's pretty uncommon.
03:02:28chamlis_ well that's something lol
It's not just.
...38Read line had percent.
...46You know what? No, it's this. This blob too big thing is. It's not joining to the with recursive.
03:03:07And then I'm going to have sorting as a problem right here, right? Let's not redo that. Let's say select comments tree dot star. comments tree.
...32Did I call it comment underscore tree? Yeah, okay.
...40How are you saying that the with recursive doesn't exist?
DMC need a from right?
Because I see it as with recursive comment tree, open, close, select comment tree dot star.
03:04:03veqqio Woo, I implemented the mandelbrot in my query language :D
Yes, that's what I'm thinking from commentary.
Okay.
...22Was it time on? Timing on? Timing equals on? I don't remember the dang
...35So.
03:05:18graefchen @veqqio limesHmm How?!
Look at that, Chamlus.
That takes 176 milliseconds instead of 250.
I'm going to trust that it's giving them back in the right order because I don't want to try and read them out of this tree.
but that doesn't get us to Rails because Rails was generating extra parentheses and then was putting a select comments star at the end or select star from comments.
So we're still two steps away from
03:06:15what we want. Can I say, select comment tree dot star.
...35And then if I reload off screen, I'll get a syntax error. Yeah, but what query did you generate? You did select comment tree dot star from comments. Can I change the from? OK. Can I just say from comment tree?
03:07:13select comments dot star from comment okay okay okay so now it's doing the right select
chamlis_ I can't think of a way around the join ordering issue apart from doing two queries from rails. It looks like the only time the CTE order is guaranteed is if you just do a straight SELECT from it
Why does it stick parentheses around the second part of the union all if SQLite doesn't like them?
03:08:00See if this is something that's reported.
...37Yeah, not immediately turning it up.
...55That seemed close. Oh, but it's Kotlin ORM. Yeah, that's not us.
03:09:23So it's with arcs, with value, with this recursive.
...49Arrow. with recursive with statements okay Makes sense that this would be an issue in Errol.
03:10:07Which... Where the heck is the Errol source?
What?
Chambliss, I got around it.
Like...
Right here.
This should be correct.
We just need to figure out why...
chamlis_ ahh, nice!
rails arrow is throwing parentheses around this second part because the generated query is not valid sqlite syntax so like if we take exactly what it generated on that last run
...59You must've been busy Googling stuff. Take that. We can, there's a space in an identifier. 43, 85. No, 43, 385.
03:11:3743,385, and then that next one is also broken. Did I miss a space?
03:12:08Ah, yeah, I accidentally pasted a little extra. With recursive. Well, let's grab this thing again.
...30Let's grab it with vim so we don't have to fight terminal wrapping. There we go. Why does that cut off?
03:13:23There we go. Just got to fight the clipboard. Okay, recreated the error. And if this goes away, it looks like it's correct. So it really is just something about that. extra parentheses huh i'm gonna have to punt on this one because it's been more than three hours i've been streaming And this feels like a Rails bug.
03:14:33Like I'm not using with recursive wrong. And I see that it generated it with an extra pair of parentheses. So this has got to be somewhere down in Arrow.
...53That's frustrating. And it's, what is that?
03:15:12So it's about 45% faster.
...30AnakimLuke are you writing the sql by hand because of the migration to sqlite or are you doing an optimization activerecord doesn't let you?
Hmm, 35, I don't know.
...37Doing not an optimization, a structure. So there's a complicated join and this could get turned into arrow with an or, that's fine. And this, we wanna pass that exact order to get the results back in the correct tree order, but Rails is generating invalid SQLite SQL, which is surprising.
03:16:27All right, so we've checked out
AnakimLuke so it *was* a compiler problem, for once?
two of the three options for rewriting that function.
And I didn't say it, but I'm really hopeful that doing the one plus n would give better results because
250 milliseconds to pull 100 comments is pretty bad.
And 180 milliseconds to pull 100 comments is less bad.
But that's still pretty bad.
Like 18 milliseconds per comment.
If that's our approximate ratio, and it's almost certainly not linear.
03:17:46chamlis_ what are the indices on the comments table?
All right, here you go, Chamos.
chamlis_ thanks
We have, yes, we are sorting by, we're not, we don't have an index on depth.
Let's try it, right?
I'm just messing around.
Create index depth IDX on comments depth.
and then let's just kind of up enter what was it timing equals on no timer on dot timer on and it's it's the same number it's the same like 170 milliseconds all right explain oh my gosh
The explain does not fit on the screen.
That's a little trouble.
03:19:09chamlis_ what about ('depth', 'confidence') as an index?
Oh, that's smart.
Let's try it.
...24What's the drop index? I just want to get rid of this other one and get us back to a known good state or a known familiar state.
...53Drop index, schema name, and then, all right. It's either just depth index, okay. Let's go back up to the create index.
03:20:25It looks like the same neighborhood of in the 170 millisecond range.
chamlis_ ahh, sad
seeing one or two below so we're picking up like 10 milliseconds maybe that's better yeah like every time you add a index to a sql query you always hope like it's going to jump from 180 milliseconds to 1.8 right i was also hoping for the one or two orders of magnitude on that one
I would have accepted one, I think.
Then I would be like, great, done.
Let's figure out the parentheses thing and move on.
Comment tree.
Open ephemeral, seek.
And then it's grabbing all of the columns.
chamlis_ what does "EXPLAIN QUERY PLAN" say? that might be clearer
I barely know how to read these.
Explain query plan.
03:21:40Oh, that is a lot more readable.
...55I don't see the merged story ID in here. First, it finds the story in cursive using, well, it's got to be.
03:22:22I stopped that sentence because I don't know what it's got to be. Let's grab that because I don't need timing.
...56chamlis_ maybe ('parent_comment_id', 'confidence') as an index?
as long as i'm winding down the stream let's throw this in too maybe somebody knows how to make sense of it because it's sure not me i'm going to put them in the order we had them on stream all right
03:23:23Well, thanks for hanging out, folks.
I'm going to wind down the stream here.
AnakimLuke o7o7
chamlis_ thanks for the stream!
And if anybody wants to figure out that apparent Rails bug or report the apparent Rails bug, please at me on GitHub if you report it.
Well, Chambliss, thanks for all the help.
veqqio @graefchen ctrl-f mandelbrot here in about 10 mins: https://codeberg.org/veqq/vutil…
I'm glad I was able to repeatedly nerdsnipe you into figuring out something very clever.
This is so much closer to what I want to say.
Ah, in fact, that's fun to end on a link.
Oh, oh, you know, we were talking about raid.
ChaelCodes Thanks for the stream!
Should I say, let's raid somebody?
veqqio raaaaaaaaid
Can I say raid browser?
veqqio shaaaaaaaaaadow
And then I can't see chat anymore.
graefchen @veqqio thanks limesOk
If I search for Rails, I don't know what I'm finding.
Filter by tag would be.
03:24:42graefchen Thanks for the stream!
No one for Rails.
Ruby on Rails.
No one for Ruby on Rails.
How about just Ruby?
...57No tag for Ruby? There it is. There is someone playing Pokemon Ruby. That is not exactly the same thing. I don't think I'm going to raid anybody. Beck, you should start a stream about what you're making. All right, folks. Well, I, like always, I plan on Tuesday. Maybe I will even see you on Tuesday afternoon. Some stuff that's out of my control keeps happening. We'll see how that goes. but I hope to see you all on Tuesday afternoon, 2 p.m. Chicago time. Until then, take care.