Pour boiling tea on yourself

01:45This is, where are we here? There we are. This is Lobster's office hours. This is the site Lobster's. I'm Peter, I run the site and work on the code base. It's funny, I just mentally autocompleted issues. We'll see how we're doing today. I'm a little bit underslept. Because I was just having fun and stayed up late hacking on something last night. And I can tell because I'm making more typos this morning. So let's throw the message in the chat.

02:34pushcx This is Lobsters Office Hours, feel free to ask a question about the site anytime.
Any time. That's interesting. I don't think I saw this one.

...49I don't think I know too much about Z-Jit. Oh, neat. You know, being Gen X and then seeing what was called Gen Y and Gen Z, now I see Y-Jit and Z-Jit. I wonder if we're going to wrap around to Millennial Jit, Alpha Jit.

03:23I understand about half of this. I haven't dug into Ruby internals much. And obviously, because Lobsters is a Ruby on Rails site, I care a bunch about it. But I will have to come back and read that more and do a lot of Googling. Neat. So anyways, folks can feel free to ask questions about the site any time. What's the policy? Why is the code like this? What is this feature? Why did you make that moderation decision? What's the community norm? What should the community norm be? We can do pretty much whatever you'd like. And when folks aren't asking questions, I work on the code base. Sometimes that means very exciting things like, coding a spreadsheet of thousands of stories which I think I've done two or three times on stream now for a total of like four or five streams because both of those were long and yeah that's that's the exciting part I really liked this SMS post. We were just talking about this in the chat room this morning. pushcx https://lobste.rs/chat
If folks haven't seen that, that's worth throwing a link out. I'll be cross promotional. Lobsters does have a chat room. It's on IRC. For better and worse. It's a little more informal and loose about dlamz I'm only here for vegan metaphors
topicality because it's ephemeral as an irc chat maybe a little too ephemeral but yeah don't need to go down that rabbit hole of what's wrong with irc unless folks want so yeah otherwise i work on the the site and i got a fun notification from twitch this morning well i noticed it this morning apparently it came in yesterday that where's this oh that's been done for ages Which, it's weird that you can see all... This one I've actually... Oh no, Affiliate is the one I finished. Anyways. I think they... This is a little bit conspiratorial, and I am making this up based on a mild hunch. But I do wonder if You reach the little affiliate thing and then you don't sign up to put ads on your stream if they don't cross promote that stream as much because it doesn't make advertising money. That makes sense. I'm not really in this for the advertising money. Oh, the fun thing I was in here for was said i have streamed for 250 hours total and this includes a little bit of streaming i did i don't know five years ago when i was locked inside the house like everybody else i think i streamed a little bit of lobsters coding maybe something else but anyways fun little milestone for the office hours streams these don't really motivate me because I'm not trying to grow a giant channel but it's neat so oh I should turn on the office hours footer there's the footer on this site that the office hours are live and I always forget to turn it on because it's a manual process because It's hard to script optional things in OBS. And anyways, DLimbs, I did see what you said. Thank you for dropping back in. Nice to see a familiar face, as always. Yeah, there's lots of silly metaphors. Oh, I should get my, speaking of silly things that come out of my mouth, I should get my notes set up.

08:04So if you're new to the stream, I have this scratch file where I talk through, I keep links for topics. It's handy for the stream archive pages, which I title based on silly things that come out of my mouth, or at least things that are silly out of context. And I know I want to talk about type ID in Lua. And then, oh, there was a, a couple of merged prs there's an open one about about slash about and then yeah so that's kind of what i have queued up for this stream but we can talk about whatever you want on lobsters office hours so there is i'm gonna jump into the closed stuff. Cause I merged a couple of quick PRS from Chandler's yesterday. That was nice. She continues to be a regular contributor to the site, which I really appreciate. So there is a let's semantic tags. So this one came out of her PR for. Why can I never clipboard? There's something about my streaming setup that breaks all of my muscle memory for clipboards, and I always struggle and paste the wrong thing. I don't know why it only happens on stream. So anyways, this was in her big grid PR that we mostly rolled back on the last office hours, and then she's pulling the features out. This one is making our markup a little more semantic which is always a nice treat the i wish i had a magic wand because this is the how long ago that's on top of com well like all of these 16 hours 28 hours I mean, the front page is going to be pretty repetitive, but let's look at, you know, sometimes they do shift into three days, nine months, four years, whatever. But since that's just text and we don't serve JavaScript to visitors, that means we can't really cache the page for longer than the shortest increment of the youngest comment on the page, right? So if the youngest comment on the page was posted five minutes ago, we can only cache for one minute because the next line of text will be six minutes. If the youngest comment is a month old, we can cache that page for a month. This is a lot of complexity. And what basically I end up doing is only caching pages for two minutes as a flat two minutes. And it's fine because full page caching that gets served by Nginx and doesn't touch Ruby is wildly cheap. It's why I haven't been tempted to... Not tempted. It's why I haven't been driven to install something like Anubis or some other... scraping proxy because this stuff just never hits the actual expensive server. Nginx can deal with a million static requests. So

12:27I dream of replacing and it really is silly but it's this one specific html tag if I could replace this one html tag with. Something cashable. We could cash so many pages for visitors indefinitely we could also cash stuff better for. logged in users, but we don't serve JavaScript on the site to visitors. And the only way I can see to implement this or to cache this tag would be to serve, you know, this date time and have JavaScript do the how long ago label. which is that code right here? No, it's often extras or something. So I don't know. Absent a magic wand, that's how caching works. We're probably good for another, I don't know, two orders of magnitude before this has to break. So that was Shamless's nice little PR. And then Another one was when I was streaming maybe a week or two ago, I opened this bug because I noticed that a private range that's used by Tailscale is not in our bad nets. So if you create, I don't know, naughty.mydomain.org to point at 127.0.0.1, We have a... In Sponge, we have this bad nets thing to filter it out so that we do not follow naughty DNS that says, hey, why don't you go enumerate your local network and serve stuff to me from your local network? And not seeing those ips in there was very odd and channels did the very unexciting unglamorous work of finding the current rfc because I linked the old one and. figuring out what we were missing and adding it, so this is the kind of very unexciting maintenance work that. keeps the site working.

15:20I really appreciate that kind of thing. I don't mind doing a lot of those chores, but the help is very much appreciated. So I don't know, Chamlus, you're a stream regular, although not always on Thursdays, I assume because you have a job. Maybe you'll see this on the archive, but thank you. I appreciate it. So then, the next thing was I didn't actually look at the state of that PR. I cleared the notification because I peaked, but I didn't thoroughly read it. So this one, Gray Hatter did not give a lot of context here. So they're a new user on the site. And in the chat room that I liked earlier, this last night, we were talking about LLM stuff because there was a giant threat on via coding. And raised the question of this is just a change to about right i can approve yeah jank jank jank thanks to github so he asked hey what do we think about submitting llm basically slop from llms in comments and the answer has been i've been deleting that for a while including stories that are obviously llm slop and that's been pretty unobjectionable but it's not actually documented anywhere which is one of those things that only new users notice and so i said hey why not pr that and so he did they did i don't actually know greyhatter So what have we added? This is actually a little stronger than I would say, but it's a good first draft.

17:51So one of the things about... rules is that they have to be enforceable and when comments are very clearly and stories are very clearly llm slop that part is easy but if someone says hey claude give me five bullet points about I don't know bit shifting in Lua just because that's the thing I was up late coding and then they take those five bullet points and expand them into a comment. realistically we're not going to recognize that.

18:53So as written.

19:00This rule is kind of this would be unworkable where. it's kind of like I am sure someone who knows more political science or legal theory could explain this, but if you make a law that's kind of a joke. In that the law could never be detected. violations could never be detected or rules enforced, you weaken the actual legitimacy of the law. So, you know, for an offline idea, if we said it's illegal to cross your fingers with your hand in your pocket, how would you ever know that? Are you going to like x-ray people randomly on the street, like total recall style? You just kind of make a joke out of the law. And if you try and enforce that grayhatter_ I considered that and didn't include it
law because somehow you actually know due to some weird fluky circumstance it looks like selective enforcement so god that is probably somewhere there is someone with you know a political science degree who is screaming about how badly i have naively half-assed an important concept but it's something i think about when Writing loss. Oh, hey, Gray Hatter. Nice to see you again. Yeah, it's. So I'm kind of thinking out loud as I'm writing. As I'm going to write a response comment to respond to your PR here, but. I can't in good conscious merge something that. I can't enforce. because it just kind of makes a joke out of it. Like, I am not actually a lawmaker. Oh, thank heavens. But, you know, if somebody uses Grammarly, especially if they're an ESL speaker, and they want to make sure that they are submitting stuff with reasonably good English, I'm Not super bothered by that and I can't detect it so like there's problems in both theory and practice of that. grayhatter_ my argument was if you didn't understand the nuance between using AI to edit your document, and using AI to generate ideas, you shouldn't submit it
And I guess the theory part there i'm saying theory, but I mean my personal opinion, where.

21:55grayhatter_ if you can tell that editing is different from generating, you'd see the loop hole
yeah so your argument was if you didn't understand the nuance between using ai to edit your document and using ai to generate ideas you shouldn't submit it i think that's a pretty good way to put it but it's not what you've written here because you've written generate any part and exclusively your own time and attention and then grayhatter_ ahh, exclusively is the problem then
If you can tell editing is different from generating you'd see the loophole I don't see the loophole in what you've written here, though. Alright, so let me write the contextualization part of this yeah exclusively is the problem well it's. grayhatter_ i meant that as dont take soweonmeone elses work
yeah and exclusively really gets at that issue of are we writing a guideline, we can actually enforce.

23:03Where's the 178? I want to say that only had 100 when I looked last night. I got to look back in that. Make sure it's not getting too heated. grayhatter_ well, you're deleting them now without a rule so I figured the enforcement was implied and didn't want to State it specifically to retain any flexibility
And yeah, plagiarism is an issue too, which I think it's only come up once in the time that I've been an admin where someone has... Like they publish blog posts where... bits of it were very clearly lifted from something that was pre-existing. I'm actually surprised that's only come up once on the site, but it's hard to spot plagiarism.

24:08grayhatter_ this was a guidlieline not a rule... I HATE CoC that dont have specific tests enumerated
The Twitch manager is so flaky. Okay, that's better. A guideline, the specific tests enumerated. When you say specific tests, are you talking about my concern about detection and enforceability? And yeah, the... it's less about retaining flexibility as a deliberate strategy and more. This is a very new thing. Honestly, it's what a year and a half old that chat GPT could start outputting something that looked pretty plausibly human. And

25:03So I just didn't think to write it down. I guess it also helps to sort of wing these things for a little while and get a feel from actual examples before trying to make a policy, you know, build up some practice and experience and some evidence for people to have these kinds of meta discussions about. before trying to make a policy. Because if you had asked me in, you know, the month before ChatGPT came out, what should our policy be? It would be like, well, these things are toys. Obviously, it doesn't really matter if we have to write a guideline. It should be, yeah, do what you want. And then we would have had a policy that didn't really make sense a month later. As it is, I hope we can take what we've learned in the last year and a half, two, and formalize that with our greater experience rather than YOLO in a rule.

26:31Actually, I didn't explain shit. I was kind of brief because I was distracted coding.

27:04grayhatter_ the contrevoversy over CoCs is because they can be weaponized... but that doesn't apply to reasonable laws, because they include specific tests that the finder of fact must find to be true... that prevents abuse
Yet I don't think I think that's a whole rabbit hole. I'd rather not get into, but I, I will suffice to say, I think I disagree with you about codes of conduct there and what the controversy is. All right. So the most important thing is

28:16grayhatter_ IMO it's often better in smaller communities for the BDFL to have lewaeway to enforce guidelines how they see fit, eschewing specific rules
GitHub. All right. At least I can trust that GitHub is going to save that comment.

...32And then what's the...

...48grayhatter_ I made the determination that applies well here because of themprev previous line, expect to be frustrated
Oh my God, Twitch, please just... Twitch is doing a really annoying presentation in the manager and I can't actually fix it. God damn it, Twitch. So in the Twitch chat, every once in a while, it puts a giant box around commenters and it's like, this is a returning chatter. And it's great, but for no particular reason, it's doing it about every single one of your comment, which makes them hard to read because they're all spaced out. I should grab the extra any part of any comments.

30:17What?

...50grayhatter_ illI'll give it some more thought and include some more specifics
Sure there's.

31:32Nations are...

32:10This is not a word that I just use AI by clicking on this right so like I actually know that firefox is doing levinstein distance for. This. But it is not far fetched to think that this like right click find me a proper word. is going to become llm powered, especially as the models get smaller and run on device. grayhatter_ it's an easy fix
And that ubiquity, which seems really, really plausible to me, is... Like...

33:25Make separate.

34:19grayhatter_ i'll add a part about AI is prodihibited from geneterarating ideas, but not produhibited from correcting ideas
Right, it's that. It's funny that I'm actually like contemplating. Did I spell Levenstein distance correctly? No. Levin. Wait, how did I get it wrong? Levin. I missed the L. Just a dumb typo. Like I said, missing sleep. grayhatter_ please forgive the splledelling ... twitch chat is fighting my phone super hard, and refesies to let me conrrrect it... ivI've given up fighting it
So there's going to be a lot more typos and such now.

...52Thank you.

35:39grayhatter_ It's really impressive how badly twitch has ruined text entry
grayhatter_ ive never used anything this bad before
oh yeah there's i'm sorry that's frustrating for you i appreciate you tuning in so we can talk through this stuff live but yeah i don't know if you really want to see frustrating chat experience try slack i was just posting on blue sky yesterday about a ridiculous slack bug i've been dealing with the last couple of months where Every Wednesday at 10 a.m., it texts me to go check out the new discussions that are happening in the exact same... Every week, it's the same channel. And it's on a Slack instance that hasn't seen any activity. Zero people talking for like four years, let alone in that actual channel. It's kind of hilarious how Slack keeps picking up these very frustrating bugs in what should be their core experience. grayhatter_ lmao, I love it
like man the fact that they can't consistently get markdown links working is the one that drives me up the wall like come on the one thing i do on an internet chat app is link to things you can't get markdown right and that one's a regression they used to get it right and then i don't know a year or two ago broke it and i reported it a couple of times and they were like yeah no why would you want to use markdown links guys Because it's the internet.

37:19That is like... I guess that's a personal gripe I have, and I am nerd sniping myself. Sorry to jump from one topic to another. Anyways, so...

...59We're going to.

38:25Yes.

...39you put this in the you put this in the

39:10Swear in the guidelines section, right?

...23Yeah.

...40Grey Hatter, if you're still here, there's another language nit. Actually, if you could sound off if you're still here before I waste my breath. grayhatter_ lurking
I can leave it as a comment. It's nitpicky. grayhatter_ still here
grayhatter_ you cant offend me, don't worry
I would rather be more encouraging in public and I say that it's just yeah the and I know this is a first draft thing your your first draft has some redundancies like you say comments or replies you don't need to say or replies and it also includes It gets into being a little threatening, I think, when it reiterates what the punishment is of your commentability deleted. If you keep up at it, you will get banned. That is all of the site guidelines, so we don't need to say it again. And then you kind of say the word human a lot. These are all very normal first draft things. My first drafts look exactly the same. That is the point of first drafts. If anybody hasn't read it, and I love recommending books on here, Lamont. pushcx https://bookshop.org/p/books/bi…
There is an excellent book on writing by Anne Lamont called Bird by Bird that is about the process of how to write. She calls them shitty first drafts. It is very easy to get writer's block and get stuck on how do I say things? And she, I believe she's a fiction writer, but this is true of all writing, that an excellent strategy, especially if you are stuck and you don't know where to start, is just to write a bad first draft. It's okay that it's very bad. There's somebody else's quote. grayhatter_ h
What is it? Good writing is editing? Editing is good writing? No, it's not that one. This is some guy on the Nazi site? grayhatter_ holy shit twitch you good?
I don't know where it is.

42:16There's a lot of them. Weird. Writers write about writing. grayhatter_ i can't type anymore
I swear there's one that's something like... writing is editing or good writing is editing. And it's very short and direct. There is no such thing as good writing, only good rewriting. That's pretty good. That's very much in the neighborhood of what I'm thinking. So anyways, very normal first draft stuff. Thank you for opening it. And I should just say this here. grayhatter_ yeah. this is why you never trust the dyslexic to write any copy lol
Edit, I never think of everything on the first comment. Very meta.

43:02Dyslexic. I don't think... I didn't even look for dyslexia kind of typos, but...

...43grayhatter_ tte same word appearing a lot is a sign
Let's say 2015. Man, the year is just flying by.

44:13Same word appearing a lot as a sign. I didn't know that. Well, in any case, I would not have.

...24I don't think you need to be hard on yourself like this is. Normal first draft writing and you took a shot at a. Writing policy for a community you're fairly new to, so that kind of thing is especially hard. So I appreciate you taking a shot at this.

...49grayhatter_ can you paste that link in IRC too? I'll lose it on my phone but my bouncer will catch it
And I also hope you don't beat up your writing about being dyslexic. This stuff is fine. You're not doing bad. If you have anything else, can I paste? Which link? The bookshop? pushcx https://github.com/lobsters/lob…
The comment? grayhatter_ yeah the book shop
can just grab the comment link and then you got all of them. That's probably faster than you saying which one you want. Oh, bird by bird. pushcx https://bookshop.org/p/books/bi…
I closed that. I meant to, where'd I go? Sorry about that.

45:39There's some of my Lua actually. usually clear the clipboard or streams good thing that wasn't anything personal the other thing about this book about where do you even start is where this title comes from she recounts a story from when she was a kid trying to figure out where even to start with a big report on I think it's like the birds that live in her neighborhood or something like that. And her dad suggests, and just go bird by bird, just start the simplest way. A lot of the times when I struggle to write comments, the thing that occurs to me is whenever you're unsure, Also, whenever I'm unsure, let's be specific here. This is my advice to myself. Whenever I'm unsure or I'm tempted to be really clever about organizing something, I just stop and go, yeah, but if you tell it chronologically, it'll make more sense. Humans are very good at reading stories. so to make this specific there was that big thread on uk osa and i ended up writing oh man the links to specific comments finally work again i'm so happy to have that working i've had three people i want to say mention how glad they are that that's working again so pushcx https://lobste.rs/c/xevn8a
This one, the OSA post, I wrote this big kind of post-mortem finalized ending. I'll throw this in the chat in case anybody actually wants to read this post, because it's been, what, a month? So maybe somebody doesn't know it. A lot of this is... How do I sum up a giant complex topic? And in this comment and in this meta post and then in the prior meta threads, one thing you can see me doing is figuring out how the hell to organize these. And a lot of them has been, a lot of the organization I landed on is let's just start chronologically and tell the story. Because if you tell a story in order, it can make sense where being clever about, let me start through this facet and then expand. I am sure, well, I mean, I read a lot, so very good writers do that all the time where they're like, let me take this specific example, this one niche thing, this one fascinating anecdote, and I'll start media and res, right? In res media. God, I don't even remember the term for the thing I can't write. And Then they blow it out into a neat story. And I can't do that for shit. I just have to go brick by brick, bird by bird, and build it up in order.

49:18All right. So, anyways. Fun little digressions there. That's office hours. We can talk about anything we want here. Gray header, unless you had anything else. I'm going to wander on. The fact that you haven't said too much the last couple of minutes means, well, number one, you're probably at a job because you write like an American. And then number two, we can just kind of keep going. So let's see. We got the, this one marked as a draft. Jamless, if you're here, I'm looking at your refreshed PR about CSS Grid. Return to Castle CSS Grid. Which, of course, is a reference to an old game because I'm old. Let's see. I don't see that it's marked as a draft. And she usually marks them as drafts. Yeah. so let's let's take a look at the actual pr so on the last stream we backed out a bunch of the css grid and closed i don't know 10 bugs and now we're gonna slowly sneak back into this stuff and so one of her prs was already doing that that was the one about the semantic time so this okay labeled grid is the bit about user profiles

51:03And so, oh no, it's not user profiles. It's the form grids. Okay. Okay, so this is, hold on. Git, can I get you to shush about whitespace? I wish Git would remember this. pushcx https://github.com/Wilfred/diff…
let alone, oh man, if you don't use DiffTastic, I have been loving this tool. I'll throw it in chat here. And I wish, I'm aware it's computationally expensive, which is why GitHub doesn't do it, but it's a structural diffing tool where you'll see it in my Git output if I ever write code on this stream, but it versus and then it shows you a diff based on what semantically is changing not just a naive what's the text changing and it is super nice for being able to highlight hey it's not that this line changed it's that this one term in a data structure changed or it's not about this indent changing which is the very simplest version of it it's also about How cute. Ugh. Yeah. Like, are you making a structural change to how something works? So I really, really have been enjoying this tool. I've used it for, man, at least a year. I'll throw this in the notes here, but...

53:14I'm starting to. So where is it?

...30I thought.

...37They. It's not actually linked here.

...51I thought it was on their blog. This is interesting. Hold on. Let me grab it off my personal. It is on the change log. So I guess that's more articley things.

54:15pushcx https://github.blog/changelog/2…
This is May 8. Yeah, so this very vague changelog post.

...34We're rolling out updates to rate limits for requests made without authentication. What this means for you, you may experience the new rate limits. Like what actually happened is they lowered the rate limits. They are not updates. dlamz is that recent? I got 429 errors from my package manager the other night for the first time forever
You can actually be clear about what you changed, but this is very, I mean, this is like bad corporate writing and it's deliberately bad because they don't want to make headlines for this, especially because Microsoft runs an LLM scraper. And there is, Yep, DLAMs, this is what you hit. So you see the date, 5-8. They wrote this a week ago. This is a pain in the ass because I have multiple browser profiles for testing stuff. You know, like I have this one for streaming. That's a separate one. I have my personal. dlamz ah yeah. a lot of arch packages pull patch files off github :-/
And then I open private windows when I need to clear cookies. And I just started immediately running into this because... know i mean you can see me click around stuff like here i might click around into a couple of files and if you open like three files you hit this oh arch pulls patch files off github oh man i didn't even think of that i've seen that in packages god i'm gonna hit that and get mad yeah so that's what this pain in the ass is and they wrote this in the most corporate pr style possible and I say this because I am familiar with corporate PR writing for personal reasons and this was written to be as anodyne and unquotable as possible and that is why it is so non-specific about hey there are a lot of you know like a human writing this for other humans in a conversational tone would be would start here, and it would say, like everybody else, we have seen a lot of shitty, badly behaved scrapers, and we're starting to have issues meeting, you know, and it's making the site unreliable for actual well-behaved users and visitors. So we have to lower our rate limits for unauthenticated users. If you authenticate, you can have higher rate limits, as always, but we've got to get really strict because of how many scrapers there are. And then they would explain, so that's, you know, close quote, the imagined conversational version of this post. And then they would actually explain. Number one. You know if they really wanted to be interesting, they would say, well, here's an idea of the traffic increase and. For a traffic increase to bother one of Microsoft's flagship products, it's gotta be just absolutely stunning scale like. I don't know, at least a 10 X increase in requests and realistically more like a hundred X plus. And then the second part of it is scraping activity. There is a whole thing people don't appreciate that has changed the scraping ecosystem, and it's just the last couple of years. So I learned about this, I don't know, maybe two, three months ago, and I was like, oh, that makes my logs make so much sense. So I want to say it must have started five or ten years ago, and I only just learned about it, right? Because it's going to take me five years to learn about this kind of stuff because I don't keep my... i'm not a real ops person and i'm not well informed about ops kinds of issues but there is a major issue with scrapers where one of the ways that apps support themselves mobile apps is they have ads right everybody knows about this the other thing that they do is they basically are secret proxies for scrapers so if you want to scrape any particular website and you start hitting them from one ip especially on like digital ocean or hetzner they're probably going to notice you and rate limit you and maybe block you entirely what would be best is if you could blend in with real traffic And that looks like coming from a residential IP that is doing other normal browsing. And the best way to this is you get a contract you the scraper get a contract with an ad network or a tooling that people install in mobile ads and you pay them to proxy your traffic through apps that are open so yeah that's just saying this like i'm on the radio let's just actually diagram it how scraping is concealed in now and probably for the last few years so this is an interesting thing that microsoft didn't want to be interesting so they didn't write but scraper alice pays app maker bob to proxy through peril phone owners install of the app right To blend in with real traffic. App maker Bob to proxy turn.

01:01:23I don't have to draw the actual ASCII art diagram. This is actually pretty clear. to hit to scrape sites that would rate limit or ban obvious scrapers so i don't know maybe maybe everybody knows this and i'm very late to the party i have never seen this written up I have seen this alluded to on like Blue Sky and then I kind of put it together and I was like, oh, that explains so many weird ass logs I've seen. Where...

01:02:18I see something that is very obviously a bot because it has a weird shitty user agent that's out of date and it is scraping a page like, you know, 400 pages back into browsing slash newest. And then I look at the IP and it has a bunch of normal traffic, even from a logged in user. And what's happening is they have some app that is dlamz that blog post is labeled "improvement" heh
on their phone that is quietly turning their phone into a proxy service for secretive scrapers so anyways microsoft did not want to explain that because it doesn't want to be interesting and quoted in the face of shitty dealing with shitty llm scrapers in part because you know Microsoft runs an LLM scraper, I am sure. And there seems to be well-behaved because I don't even know it. Yeah. Oh, yeah. This whole thing is just like, is this a tag? Yeah. So it's tagged as ecosystem and accessibility. What do these two things have to do with each other? What do these things have to do with rate limiting bots? Nothing. Jesus. So the... The bullshit-o-meter of this PR post is pretty high. And I'm kind of... I'm beating it up in a way I don't usually criticize public writing because...

01:04:08I guess, because there's an opportunity for Microsoft to have actually improved the public conversation about this of. what's actually happening. What how are sites choosing to respond why. And instead they went we just don't want to get bogged in with that public story, especially if we're going to get criticized because we run our own scraper, we have to write something because. Otherwise, people are going to see the 529s and make up their own conspiracy theory. Let's write the dumbest, least quotable version of it. And this is why I don't work in corporate PR is, I mean, I run a site that is about learning things and improving communication. And this is writing to promote a product, to minimize the fallout for the product. And I would rather be forthcoming and straightforward. And I don't I spend all my fucking time explaining what I'm doing and why and we just spent 40 minutes ish reviewing that PR that is probably going to end up adding two or three sentences to our page. I feel like I spent a lot of my time re explaining things and finding better words for it and trying to share knowledge, and so, when I read something like this, that is. it's asked covering it's here because. They want to avoid reputational harm, and it is intentionally uninformative just it drives me up the wall in the same way that i'm driven up the wall that. Slack can't manage its basic chat features because it wants to be an everything else app. It's, you know, supposedly you are writing to communicate things, and this is deliberately uncommunicative.

01:06:21You know what it is? The mental image for this, as long as you're talking about books I like,

...34pushcx https://bookshop.org/p/books/th…
There is this famous book on interface design and user experience design called The Design of Everyday Things. And this image sticks in my head. There is an artist and I don't remember the artist's name who made this teapot. This is not like a render. This is someone actually made this along with a bunch of other impossibly bad objects as a set of art pieces and I love it. So this is a teapot where if you hold it by the handle, you can only pour boiling tea on yourself. That GitHub blog post is the writing equivalent of this teapot. Someone has written something to communicate as little as possible, as obtusely as possible. they got paid for it they probably were like ah i did my job i was responsible i don't know there is the chance the actual person who wrote that blog post or the office person who cranked down that rate limiting is watching person is that what you want to be doing with your life is not communicating things

01:08:07Man, that's a long fucking digression from whitespace in diffs and semantic diffs. I guess that's the other way you can tell that I didn't sleep super well is I have a harder time staying focused. Ramble a bit more. Anyways, it's office hours. We can talk about anything. Sometimes you can't even stop me.

...35So aside from... That 429 thing, the rate limiting is now so strict that if I view three or four source files in an incognito window, I get rate limited. And it's funny, the post says the API, but now that they've got a whole React front end, everything hits the API. So again, there's one more deliberately misleading thing in there. pushcx https://github.com/git-bug/git-…
Between that and the endlessly broken UI, I've run into stuff like Gitbug, which it's a little funny, their homepage is Git on GitHub. but it seems to be the most mature track your bugs directly in your Git repo kind of tool. I started keeping an eye on this because one neat thing it can do is, where is it? Ah, third-party bridges easily synchronize issues with platforms like GitHub. So there is the potential there of, well, hey, what if, We took our issues, sucked them off of GitHub, because it's a ton of data and discussion. pushcx https://tangled.sh/
And then, ooh, something that hasn't come up on stream. Hey, Anubis. Speak of the topic, there is this, both of these things, you know, are pre 1.0.

01:10:31But they are, hey, what if we took the app protocol, this is what powers Blue Sky, and we used it to distribute the various low-level objects of a Git store. And then if we had all the features we care about, because if you look at what I actually do on the GitHub lobsters repo, Lots of issues, a little bit of pull requests. We don't use discussions, but I can't turn off this view. I've unchecked the box, but GitHub so wants to promote this thing that I still have to see this. There is the wiki. We have, I think, one page. Yes, one page. This could just be a page in the repo, a file in the repo. And then... Pull requests is the big thing. Security is actually pretty important. I do appreciate the Dependabot things. But for our, Lobster's is pretty deliberate as a code base about having few dependencies, which helps minimize how much security stuff we have to deal with. Boy, has that design choice paid off in spades over the years.

01:12:05So then the kind of question is like, well, how much do we get out of GitHub? And the answer is like, well, we get pull requests. Those are pretty nice, but there are other workflows and have the option of building something into the site because it is a web app. And then secondary is the main thing we get out of GitHub. hundred times more valuable than any of these features is the network effects the all the developers already have github accounts and i say all yes i am aware there is the i don't know call it hundredth of a percent or thousandth of percent of people who refuse to have github accounts they are totally entitled to make that decision But I'm okay saying like basically all developers have GitHub accounts and access to those potential competitors and comfortability for users to look at our source for transparency is incredibly valuable.

01:13:34But GitHub gets less reliable every month and less usable every month with stuff like this straight rate limits. And it can't even be straightforward about what those rate limits are. Oh, and I didn't even say like the third part of that post should have been, please explain roughly what the rate limits are because people are seeing them so fucking fast. but Microsoft didn't want to be involved in that conversation. So they ducked it. Anyways, that kind of stuff is really, really frustrating. And at some point it becomes a, how much benefit do we get out of your network effects? If we can get all that core functionality, if that stuff has a nice UI UX, if it becomes reasonably popular and usable such that it's not a jump through 19 hoops to see our source code.

01:14:52Just worth always thinking about what is the actual value we're getting out of these things and why. I feel that same way about Rails. of rising frustrations.

01:15:14And it's silly, it's these little things like, yeah, I've been using diphtastic for a year or two, and it's such a nicer experience.

...30So It's especially nice because it's basically instant and pull requests just get slower and slower.

...50All right, so she's done. Settings is such a long page. This page eventually is going to get broken up in some way. As it is, it's getting a little bit silly with how long it is.

01:16:10If I could just sucker some experienced front-end designer into helping out with the site. I keep tagging things as design, and I added a whole design tag, but so far, no luck getting someone who is good at design to contribute. That's okay.

...36Oh, she was thorough.

...50Yeah, so this is that ubiquitous change. Yeah, these colons. These were a lot of the diff I read the other day on Monday. All right. This looks great.

01:17:22Which bug fix? Oh, I must have accidentally dropped this in the revert. Good catch.

01:18:08know before i merge yeah okay so this is saying the labeled grid is its own thing and the site wide margins are separate and this will be inside them so we're not gonna yeah there was a particular UI bug that in my head was, I saw it on a submit form. And so in my head, it was associated with the forms rather than it's an overall page thing. So I'm just making sure that we're improving things. Oh yeah. pushcx This is Lobsters office hours, you can ask questions about the site anytime, or just watch as I work on things.
I should throw the, the bumper in here of the site any time.

01:19:53Yeah, more so.

01:20:14Actually, I had a comment too. chamlis_ sorry, just force pushed I spotted a tiny missed bugfix
Didn't have a... Oh, hey, Chambliss. I wondered why this started jumping.

01:21:18chamlis_ it was the :not(span input) -> :not(.link_post) change you made
That's been so I was skimming this. And then I went off on a very long digression about diffs and bad PR writing. But this is a great start. And I go see that not Oh, I remember this. Yeah.

...55Yeah, okay. All right, so this looks like it's good to go. As long as you're here, is there anything else to say about it? Because I assumed that you were breaking this out as let's do one or two pages, and then you'll do more iteratively. I would like to roll this out kind of one at a time. chamlis_ that's the plan, taking it slow seems like the better approach
Then you can get other forms as you want. Sound good? I kind of asked a hero, like, do you have a punch list or a plan or anything? Okay, cool. Well, then when this goes green, I'll go ahead and click merge. chamlis_ I haven't yet accumulated a full todo list for it, but I'd like to hit everything
And yeah, I've appreciated your contributions here. I'm really looking forward to it.

01:22:58Cool. Well, thank you. I would really love for you to take on that giant cleanup. You weren't here earlier, but I went back over the PRs of yours that I merged yesterday just to kind of show them off, because I really appreciate that kind of unglamorous maintenance work, like reading the whole RFC for more networks that we shouldn't allow people to enumerate. Thank you. Oh, as long as I'm waiting on this and talking through stuff I committed off stream, there is another thing. So first, ChamList spotted a possible fix to a bug. So we'll share this link.

01:23:57And I committed that. This is one of those you kind of have to be the admin to debug this because it's such a weird prod issue. And setting up to run against a Mastodon bot account is a chore. So I mostly just filed this one because I knew I would have to come back to it. Hopefully this commit does it. I haven't yet deleted something that made the front page. I did delete something about someone who was mad about a job interview, which is really not topical. I don't know. That one was marginal. I could see a case either way, but I removed it, but zero people upvoted it. And so it didn't make the front page. And so there wasn't a, the thing hadn't been posted to Mastodon. So it wasn't a prod test. The bug is low priority enough that I'm just kind of thinking about it. I haven't yet wanted to manually test it. So the other thing I did that I wanted to show, because it is the kind of infrastructure work that shows up is, where's the Ansible repo? Well, I can just show the deletion. so i have been moving things over to background jobs and solid queue and i wrote a bunch of ansible config to create a systemd service for solid queue because it has to have a worker process that kind of pulls the queue and runs recurring jobs and all this kind of thing and I could just delete all of this. I mean, this isn't a lot of code, but it took me a couple of minutes because I'm not super familiar with Ansible or systemd. Because again, I only know enough ops to be dangerous. And then it turns out that you can just tell Puma to do it and Puma will manage a worker. And as long as you have one machine or you want all of your Pumas running workers, hey, you're done. And I say, hey, you're done, because I did actually knock the site offline a couple of minutes and have to bounce it a few times because Rails doesn't really... There's been a bunch of marketing hype about Rails as the one-person framework. DHH gave a keynote about it. It's just one of his little marketing things. It's not...

01:27:01It's a nice theory. There is an issue with Puma where if you run your Rails app on one web server using Puma, which is by far the most popular way to run your Ruby workers, what you want is to be able to have some kind of phased restart functionality on deploy so that when you deploy, say there are 10 workers, it turns them off one at a time and starts 10 new workers as it turns off the 10 old workers, right? So you go 10 and zero, nine and one, eight and two, et cetera, until eventually they're all new workers. And it's sort of a blue-green deploy, but it means that you cut over without any downtime. You deploy without any downtime. And Puma sort of has functionality for phased restarts. But if you look at our config Puma, you will see a bunch of comments from me trying to fix things. Oh, the heap dump stuff is still have that leak going on. I got to come back to that. All this stuff about reload app. And then there's a bunch of stuff in the Ansible config. And then if you don't call preload app, long story short, Ah, fuck it. No. Long story long. If you don't call preload app, this solid queue worker doesn't work. And it fails with an obtuse exception about solid queue is not defined because Puma didn't preload all of the Rails app. pushcx https://github.com/lobsters/lob…
And you can sort of get it to boot the Rails app. And I just added this from an issue, but I suspect it is currently... breaking our phase deploys because I did a deploy the other day, like a very anodyne one, the channel says commits. And when I did one of these, someone in the chat room was like, oh, hey, the site is down. And so it looked like we were doing a hard restart. So anyways, yeah, the summary is, Marketing message about one-person framework aside, Rails doesn't really scale down to having a single VPS serving your app because every deploy becomes an outage. And the outage is 30 to 60 seconds, but that is certainly enough time that we have users noticing it. And it is frustrating that Rails can't scale down that way, you have to do is have two vps's have a load balancer in front of it have deploy wired up so that it restarts everything on one it waits for them to boot up it re-enables them in the load balancer then it does the same on the second like it's a lot of fucking complexity So if I had a magic wand, this is maybe the thing that bugs me most about Rails because we keep taking downtime about it. I have a ton of Ansible config to try and correctly kick off phased restarts. It doesn't work reliably, so sometimes... So I have to do the... The hard restart if we update anything in the bundle because the phase restart keeps the parent process around, I think. So it won't pick up on new libraries and gems and we get weird ass exceptions if we don't do the hard restart. The whole thing is so frustrating.

01:31:05But it's a nice little digression here. So let's, let's not. Yeah. Chem list. You, I see you did your thing of making very semantic PR. So I'm going to do that. Let's put an actual use whole message up top. All right. Suck that down. Deploy it. Actually, that would be a good one. As long as I am griping about the site goes down when I deploy it. Let's watch this one because this doesn't touch the bundle. So my Ansible should detect that it can do a phase restart and should attempt a phase restart and the site should not go down during this deploy. But I sort of suspect that that workaround for solid queue is breaking it.

01:32:41pushcx https://hatchbox.io/
there's a as long as i'm watching this let's talk about a competitor there's this thing called hatchbox it is a commercial service it's like 10 bucks a month and it would allow me to pitch ansible and i am increasingly tempted to sign up for it i've been listening to the what is it the ruby remote podcast somebody sent me a link to that a couple of days ago because they mentioned lobsters and they were like hey your thing all right so it thinks it can do a phase restart instead of a hard restart but it is doing a hard restart because i'm not getting sent to a worker

01:33:38And I got a 502. God damn it. Yeah. And you see that you didn't do a reload. You did a restart. So reload is the name for the signal in systemctl. And then... grayhatter_ > AWS
grayhatter_ https://gr.ht/i/trash.gif
So even now when I'm saying reload, it's doing restart. God. That's just a bug.

01:34:17gray header there's a lot of options and i try not to dunk on them like i am trying to give thoughtful criticisms of things i have been kind of grumpier on this stream than i usually am and snarkier but i don't know this is taking longer because it's

...48chamlis_ here's a tangential question: why did mockturtle only mention one of the two commits from that PR
solid queue is going away is stopping okay there we go so it's like you know what this looks like i think what's happening yeah actually so this let's put it in a scratch i'm going to want it again in a minute Why did Mock Turtle only mention one of the two commits from that PR? I have no idea. grayhatter_ ^ it's a low effort troll, I'll admit... but that's all AWS is worth, AWS is a good effigy of the worst parts of modern SWE and it feels ethically wrong to see something bad and not say something
Mock Turtle reads the commits RSS feed.

01:35:43grayhatter_ even if that something is low effort
pushcx https://github.com/lobsters/moc…
So Chandler, I get to nerd snipe you with a link. But here's the source code for Mock Turtle. It just reads this RSS feed and pastes them in. If that commit isn't in here, it's not going to be in there. That's all I got. And if we could find a Ruby IRC framework, I would love it so that I could add to... Mock turtle, it's written in JavaScript and I don't write a lot of JavaScript and that hassle of having it in another repo and another box and stuff is enough that mock turtle was missing some obvious features. Do we wanna start? Okay, we're back up. So clearly what's happening is when we send the signal correctly for phase restart, Puma starts that process It shuts down the solid queue because solid queue doesn't know how to do a phase restart. And so solid queue is deciding to kill Puma. And that's why we're taking outages.

01:37:02Let's see if someone's talking about this.

...09Yeah, Linux clipboard.

...47chamlis_ reading the code, I think it's because those commits happened in the same second
You think it's because those commits happen in the same second? Oh, I mean, that could be a bug in the code. I don't actually have that code loaded in my head. It's been a couple of years since I've touched it, I think. So I guess we get to file a bug. Yep.

01:38:19How do I get the hell back?

01:39:45Let's grab some grabbing that log off screen.

01:40:54that's true

01:41:25I should just delete that. I don't know why I'm in a very snarky mood this morning. All right.

01:42:05I'm going to say, all right, so let's link. Yeah.

01:43:07And where's my commit?

...19That's this one.

01:45:58Currently managed by second issue. Is that plugin deliberately crashes Puma during phased starts?

01:47:26Where's this shoe?

01:48:54All right. Let me jump over to the IRC channel because we were talking about the phased commits a bunch there.

01:49:25I'm saying that I figured out what's happening, that we're taking a bunch of downtime.

01:50:29jamesnvc Hello! I actually had a random "lobsters query" question I was thinking about the other day - how many comments have been posted that are just the text "obligatory (xkcd)?: https://xkcd.com/.*"?
James, welcome. You have a random lobsters query. Yeah, we can totally do a query here on channel. So I just finished reporting this bug that we're seeing in production, unfortunately. So now is a great time to change gears. Let me just get this link. Yeah. Man, we still haven't gotten to the fun thing I was thinking of. That's okay.

01:51:12So how many comments have been posted? Are they just the text obligatory? I don't know. So that's an interesting question. Can you link me an example comment so that I know for sure I'm talking about what you want to see? But we can write that. So one thing, and I'm poking in the comments model, we have a bunch of jamesnvc Hah, I see a bunch here: https://lobste.rs/search?q=obli…
very very opinionated validations on comments that prevent meme comments and the specific memes we prevent are people saying a comment that's just this tldr bump what is this one oh this is when people post like a.c.b.d so like just as a way of emphasizing things or saying literally me too or nice and then those kinds of memes were just like please don't leave those comments right like just upvote or just don't do that this one Man, deep cut. jamesnvc I love to see those 😅
This one is a Simpsons reference that... Simpsons is getting old enough that... Maybe I should link this one.

01:52:54Oh, man. A TV rip. chamlis_ I can truthfully say I've never seen an episode of the simpsons
How old is this? 14 years? Nice. Well, that's long enough they've probably forgotten their account. I do try and explain these memes. Shameless has never seen Simpsons. Yeah, there's, I was just thinking about this other day because I spent a bunch of time with my nephew who is a teen. He's kind of great. I love hanging out with him. But I have never spent so much time thinking about being middle-aged because I'll make one of these references and then he just stares at me like, oh, old person.

01:53:37So it's kind of funny that now that the site is, let's see, we started in 2012. So now the site is more than a dozen years old. We are starting to see people who don't know those cultural references because they're just younger. At some point, we are going to have someone mention in an offhand comment that the site started before they were born. TommyLuco stop being old
And I am going to dry up into dust and blow away like Thanos snapped at me. TommyLuco and heyho everyone
Right? See, there's a younger meme. I can do the younger memes. I am hep. I am with it. chamlis_ you know your skibidi from your rizz
Yeah, thank you, Tommy. All right, so let's go look at the prod database. So we are looking for comments for James. So James, is it? the whole comment is obligatory or they mention obligatory? jamesnvc I'd say just the first three there
TommyLuco and i'm in the same thing. when i can say oh i achieved that like 3 decades ago, still makes my spine shiver
jamesnvc Especially 2 & 3
So I guess another way to ask is, excuse me, I actually know those references. Just the first three there, sure. So Chambliss, I will say one of the best things about being an uncle, TommyLuco being an uncle beats being a father of a 19yo :P
is getting to slightly misuse slang in front of teenage nibblings because it drives them up the wall and you get to watch them bite their tongue. jamesnvc I cracked up a younger kid on a dive trip I was on by referring to something as "goated"
So I'll be like, oh yeah, if something is like not cool, I'll be like, oh, that's so yeet. And watching them sigh internally is hilarious. It took them, because it's especially funny from the ages of like, jamesnvc and a couple years ago at Thanksgiving dinner, goaded my dad into saying "deadass", which destroyed everyone under forty
11 to 14 because they don't know that i'm just messing with them and then somewhere around like 15 they're like oh he does this like every conversation this is a bit chickapa is very funny chickapa is the kanada word for uncle the specific kind of uncle i am so That's great. chamlis_ I recommend adding the knowyourmeme rss feed so you can keep up to date
God, it's so fun hassling the youths. There is a very teenager period where they take themselves very seriously and are shocked that adults can use slang. Oh, there's an RSS feed for Know Your Meme? jamesnvc My wife is a teacher, that's how I keep up to date
jamesnvc She gives field reports
That's kind of hilarious that as an old guy, I can be like, oh yeah, an RSS feed for the youths as opposed to... I don't know. TommyLuco i had to explain RSS to my daughter...
I assume the youths are getting it from TikTok and YouTube Shorts, but I get it from RSS. I do have a Firefox keyword shortcut to search Urban Dictionary. That's how I make sense of browsing Reddit. Although I actually have a personal rule for reddit which is i read until there is a post that is obviously by a high schooler and then i stop so some days i get like 10 posts into reddit and some days i get a couple of pages into reddit but always it's like oh you are mad about high school i'm done that's my stopping point all right so anyways Where do we want to start? Let's start with date, created at, short ID, comment from, no, substring, comment one, let's say 100, because that's plenty to see, comments where,

01:57:40What do we want to reach for? Do we want to regex this? I think we have to. jamesnvc maybe ilike?
Let's just find the ones that mention xkcd. jamesnvc ilike "obligatory xkcd: "?
Let's start here and refine. Limit 5. jamesnvc mixed case?
jamesnvc Isn't "ILIKE" insensitive?
only one oh it's the is it the capital o so yeah i didn't even turn up these first five what did i get wrong here oh you want to see just regular like well i think i don't remember if r like is automatically i know regular like is case in census let's say obviously Oh, I wrote obviously instead of obligatory.

01:58:56How's that? James, you are not a teenager. jamesnvc oh no, I am old!
Nowadays, the default is that like is case insensitive. All right, so here's this. Oh yeah, I was going to say order by created at desk, just to kind of see that I'm getting the same comments as the search engine. Yeah. Okay.

01:59:34All right. So yeah, it's funny the things that catch you out. Every once in a while, I'll say something about Vim and people will be like, it's had that feature for five years. And I'm like, ah, shit, you got me. So are you okay with like, can I do the lazy version of this? Can I just say, and the length of the comment, is is it length or sterling i think it's sterling clearly i should have i should vibe code this it's less than i don't know what 100 characters not sterling man it shows you that i learned php the same time i learned my sequel because i constantly confuse those all right This looks like roughly what you want. Let's see how many we got. Oh, actually it said four rows and set when I had limited five. So yeah, there are literally four of them and they're all on screen. Does that answer your question? jamesnvc Hm, I wonder if including ones that have a quote before them?
What if we say obligatory and then percent XKCD? Cause like that first one, Well, they could just say obligatory.

02:01:05Yeah. All right, so we got eight. So not a heck of a lot, honestly. jamesnvc That's good!
Maybe not enough. I don't know. jamesnvc I think other sites have it a lot more than lobsters
There's kind of a frequency times irritation factor that adds up to Should we write a custom thing to prohibit that meme? I do try and discourage meme comments. pushcx select date(created_at), short_id, substr(comment, 1, 100) as c from comments where comment like '%obligatory%xkcd%' and length(comment) < 100 order by created_at desc;
I don't know that this one rises to the level, or maybe it's just a meme I'm a little softer for. So let's grab this query. jamesnvc Thanks!
I'll stick it in the scratch so that anybody who wants to, here, if anybody wants to write an alternate version of that, and whoa, man, what the shit? chamlis_ do not adjust your set
grayhatter_ that's impressive
What did I even do to the terminal? Control-C is doing nothing. Control-Z. Man. Okay. jamesnvc Need to degauss your vim 😅
That's really something. So let's have a new terminal. I guess I'm just going to gun that. I think it got confused and thought I was scrolling up.

02:02:26We used to crash properly. How is that still in that clipboard? Oh, because the fucking the X clipboard isn't a separate thing. It's secretly a, oh man, lucky that I pasted that query over here. I would have had to go log into prod again to fetch that back.

...58pushcx https://lobste.rs/about#queries
Anyways, so yes, that is... Oh, I didn't do my little... I try and mention these things because I spend a lot of time explaining and re-explaining the site, but Degoss might have... Oh my god, I used to own a TV. Speaking of being old, I'm okay reminiscing about this. I used to have... Actually, the prior TV to the one I bought a year or so ago is... was a i don't know 13 15 inch tube tv so this is i think i got it in 96 ish had it till 2001 but the vertical hold was going so when you turned it on especially if it was cold in the house it would roll just like that terminal rolled other direction actually rolled up and then as it warmed up it would roll like two or three times and then stop but as it got older the vertical hold got worse and worse and so it would take 20 seconds of rolling before it would stop and it would be like like one of those what's the big wheel on prices right it was exactly like this do they have a video of it

02:04:27Make sure this thing is muted. Yeah, here we go. So there's this American game show, The Price is Right, where they spin a big wheel. We don't need, right? It's like this. So it was like watching the TV was like watching this for the first 10 or 20 seconds where it would just roll and roll and roll. And then it would slowly stop. And sometimes it would even like have to back up halfway to totally settle. grayhatter_ you've heard of CRT right?
grayhatter_ no
grayhatter_ cool retro terminal
it got worse and worse and worse over the years and it used to be by the time i got rid of the tv it took two minutes or so for it to settle but i was broke as hell so yeah yeah i know what a crt is i'd say tube tv because i'm just that old the tv i'm describing was a cool retro terminal I don't know what this is. Oh. This is... Oh my god. grayhatter_ I love the scan lines
This is very familiar. I have actually... Oh, it's even got the ghosting? The scan lines? grayhatter_ I want a terminal that does those
This is going to be like... implemented on the shader and require an enormous GPU, right? This is all some kind of...

02:06:10grayhatter_ this is gpu based, IIRC yeah
jamesnvc oh man, that amber is transporting me back to my parent's computer in like 1992
I have used these terminals. chamlis_ I've occasionally wondered about getting a vt220 but they aren't cheap
Like, these actual... Like, I have physically touched a VT100 terminal and it looked just like this. Although the one I used didn't have color. Or... bolding whatever is happening here that these fonts are slightly different. No, I assume they aren't cheap anymore. They were when I use them. That's why I use them. Yeah, classic green. I feel like they're slightly exaggerating the fisheye effect, but maybe it's just that it's edited out in my old man memory. This is beautiful. This is a wonderful blast from the past. Thank you. grayhatter_ chamlis_ there's some really good guides for making a [fake remake]
I have to grab this to share it with my fellow old folks, friends. grayhatter_ I'm sure one is using a raspi
We have a little social network, a group chat. Works by carrier, pigeon. No, it works by punch card, obviously. God, that's beautiful.

02:07:29Did they load? Yeah, it's funny. They should load WordStar in one of these. Or the QBasic editor. chamlis_ interesting! I don't know if I'd like that as much but I'll take a look (I'd want to plug it into my desktop and run weechat/mutt on there)
God, beautiful. So anyways, before I got nerd sniped there, which, thank you. Excellent nerd snipe. You got the shit out of me there. What we were doing with James was if you want to run queries on the prod database to answer fun questions we can and I usually. phrase it. Oh yeah I wrote. But not write them for you, I mean on office hours when we're just hanging out I don't care about writing them, I just didn't want people emailing me and being like hey why don't you spend 10 hours writing a query and revising it every time I think of something like I'm not signing up for that but. grayhatter_ but if I was gonna get one, I'd do that, because I don't really place a lot of value in the "legacy" part of it. I'd just want a cool looking thing on my desk, if it's modern or really old, I don't care, how it looks is much more important IMO
You know one like especially live on office hours yeah what the shit. So if you have questions about the site like. How many questions get posted per time of day? And short answer is everything is correlated with the US work week and traffic is half of on weekends. Anyways, we can do those kinds of fun investigations.

02:09:00grayhatter_ chamlis_ yeah, for some people, the history is more of the value, I totally get that
If anybody wants an ancient slash dot meme, I actually used that. Did I have a VT220 or did I have a VT200? VT220 versus VT... What did I use? Yeah, okay, so I said VT100. I have used one of these, but I was specifically thinking of... So I used a VT220 terminal in the late 90s, and here comes the slash.mean. to set up a Beowulf cluster of Linux machines. And that was the first professional Linux work I did, was I set up a Beowulf cluster. It was a proof of concept of four machines running Red Hat 5.1. Yeah, I remember updating it from 5.1 to 5.2. So we are going way back in the day. And it was going to do analytical, credit card transaction processing and it was going to get scaled out and then the project never went anywhere. It was for a consulting firm. I think they just didn't end up closing the contract, but that boss never told me what the hell was going on.

02:10:36chance somebody's seen this nope all right so yeah all right we're two hours in these streams usually run three hours so I'll do my little bumper that this is lobsters office hours you can ask questions like any time otherwise I work on the site and Oh yeah, the thing I had actually thought, oh yeah, I'll just jump into the issues real quick. And then, you know, two hours ago, type ID in Lua. So I have been tinkering with, I don't know why I got this idea, but one of the things, this is one of those Peter knows a little bit about ops, just enough to be dangerous, is

02:11:35I wanted trace IDs on requests. So we have, I have added, I did some of this work on stream. pushcx https://github.com/lobsters/lob…
I added recently type ID to the code base and type ID is, can I just find the way I did make a, I don't know about replace, but there's a GitHub issue about this that we have short IDs. They're nice. We are starting to see issues with not having enough entropy because of the birthday paradox. This is a very this is one of those things where I am trying to get way ahead of this because otherwise we're going to forget about it for five or ten years and then it's going to really be on fire. Because we have 580,000 comments and we pick up, call it 250 a day lately, we are starting to see collisions in the short IDs. They just don't have enough entropy. There is a nice project that is well-defined, that is multi-language. Stripe, please write up about... See, it's inspired by Stripe IDs. I will repeat the thing I always say, which is Stripe, please write up a blog post about Stripe token design and structure. It would be a canonically useful resource that would meaningfully advance the state of the art. In the meantime, I will use this thing that looks very much like Stripe tokens. I like that it has very clear way of creating these and a formal specification and it's implement you know it's just it's clearly written it's nicer than any of the other versions of this i've seen people have implemented it in a bunch of languages it comes with an official test suite there's just a bunch of smart stuff happening here so i started adding it to the site a bunch of our models, basically everything where you can individually address items, or I can foresee wanting to individually have URLs to them in the future has got a type ID and I'll add more in the future. And so I'm pretty jazzed about type IDs is the short version there. and a trace id i don't have a great link for this but a trace id is an ops idea where you end up with multiple services as much as i'm trying to shut down services and roll things together like that solid queue i saw earlier you always end up with multiple services so if you imagine you see something weird in prod especially a performance bug like This one customer complains that a page took 30 seconds to load. What happened there? Well, let's go look at the Nginx log. Can we see their request? Let's go look at the app server log. Can we see their request? Let's go look at the database log. Can we see their queries? Let's go look at, and you know, as you stand up more services, you can think of how many places you have to look and figuring out how to correlate the SQL slow query log With the nginx log, that's a hideous task. What you really want is a trace ID, which is whatever the outermost layer of your system is should generate a unique ID, like a type ID. And then it should pass that on, and every intermediate system should pass that request ID on to any services it contacts, and all of them should log in. So then if you are debugging something like that, you can see, oh, okay, so on request, blah, whatever that long ID is, I can just grep for that or do whatever kind of query for that in all of my services and I can trivially correlate things. And that kind of investigation is really obvious you get other kinds of nice benefits out of it like. Being able to do. keep coming back to performance things because it's so useful for performance where you can say like oh what's the average number of queries per request what's the average number of queries per request that takes more than three seconds to answer the user how many services does the average request touch or i guess another one is This user request sent was supposed to hit, I don't know, Redis or Elasticsearch. Does Redis or Elasticsearch even log that trace ID? Are these services talking to each other? Did we have some kind of net split? Is this an ongoing thing? I don't know. It's so useful for these things. And it is a little bit more than a little bit overkill for lobsters. but it was so fucking useful when I worked at Stripe that I want it, and I want it to be standard. I guess I could blog about this eventually. Let's see. So, if I want Nginx to generate type IDs, I have to have some kind of module. And I know if I write C, I'm going to end up crashing because I don't write a lot of C. I'm not going to get that pointer math right. So I started writing it in Lua because type ID is ported to many languages. However, there is not yet an nginx module for generating these. There is a... I did find. Yeah, so there's someone for creating version four UUIDs. And. This is Google ignoring what I searched for. There we go. I don't know what this one is. I don't know. Let me find out my personal browser. So it's just all over this repo. There we go.

02:19:21So there is, somebody made this a year ago. It looks like they just knocked it out. It adds a UUID7 variable or function for generating a variable. That's nice. That's most of what I want, but I wanted it to actually be encoded as a type ID. And I looked at this. I guess I could have tried vibe coding it, but I was like, yeah, if I try and edit this, I'm going to overrun or math it's not going to work it's the base 26 encoding that i know i won't get right and there wasn't already a vanilla c implementation a whole bunch of languages not c what's this guy's repo

02:20:25okay just some guy who implemented it in python not useful google there also is not a lua version but i know that there is a lua plugin for nginx that's in apt so i can just install it yeah that was the other part of even if i ported extended that guy's c1 this thing which doesn't even have a license file so i kind of don't want to touch it even if i extended that then i have to figure out how the hell i compile nginx modules and then i have to put it into our ansible tool chain and i just didn't want to learn that so i started writing a nginx module in lua and of course i ended up writing c anyways it's getting there so this this is bit shifting math i just tweeted about this last night man for a thing that gets embedded into video games all the time i was a little surprised how unnatural it was to express this bit shifting especially the fact that it starts its arrays, its tables, at index 1. chamlis_ ooh I fear you've nerdsniped me with typeid not having a c impl
I found this kind of kludge for starting my lookup tables at 0, which, oh man, thank heaven I didn't have to just shotgun plus 1 into all of this. This is pretty much a direct copy and paste from the Go implementation. Chamlus, if you want to write it, especially if you want to package it as an Nginx module, I wouldn't mind. This was just like a, I did it because it was fun. And then it was like, oh, there's just one little hassle. Like I have to index into the string one based instead of zero based, et cetera. And then I had a bug where I had a typo in here. You can imagine how much fun that is to run down, right? chamlis_ you should do your thing, it just feels like a nice small useful project for me to do
And then there was a limitation where The clock in Lua, os.time is, and Chamelis, if you have like free time coding, I want to nerd snipe you into more lobsters commits. Whatever, do whatever you want. It's fun. The, what is my warning here?

02:23:18chamlis_ my bank account stands at the ready ;)
I thought I kept this... I didn't want to pull in, like, big int. Yeah, I may have fucked this up. Your bank account. Yeah, I wish I had a budget for lobster stuff. As it is, part of the purpose of starting this office hour six months ago was... You know, I would probably have stuff happening better in my life if I kind of timeboxed my lobster's code work to six hours a week instead of letting it sprawl. especially after I left full-time employment, that became a prioritization issue. So I had to write C because the Lua function for getting the current time only gives you seconds resolution. There is not a way to get milliseconds. And if someone is immediately thinking of the OS.clock function in Lua, That allows you to count the number of milliseconds since your Lua process started executing. That is not wall clock milliseconds precision. So I wrote a dumb C function that pulls from, I guess this is, what's the name of the GNU C standard library? I've really shown off how little C I write. This is, I found like baby's first Lua, see plugin and then i just kind of hammered it until it seemed to work i think this is correct it's short enough it's probably correct and then i wrote a uuid function that says okay let's and again i am probably not expressing this well in lua this is probably none of this is idiomatic lua because i have no idea what idiomatic lua is i've barely written any i've just edited some and vibe coded some for my window manager i didn't hear because i still don't know the copyright status of vibe coded stuff so like if i had to throw away some of my window manager config i mean first off no one would care it's only on my machine but this if i put it up on github someone might care so here's where you need that millisecond precision because it goes into the timestamp the first part the first six bytes of the uuid7 are just a timestamp directly and then the rest is random bits and you overwrite a couple no the last let me say it right the first six bytes are the timestamp the last 10 bytes are random and then you overwrite some of your random bits with version flags for uuid and it's just in the middle probably to get to avoid some cardinal cardinality hassles if you are shoving these in a btree index that's just a guess off the top of my head i didn't even read the whole wikipedia page on uuid 7. so it's coming together the Typed script i'm not going to show it because it's not done yet, but I have a spec. And I. I mentioned that type ID comes with. Oh it's open in my my personal editor but it comes with a little test suite of things that are almost valid but not quite, and I think I have one bug left. Where. I think it's this test case I'm failing on. So I'm going to throw a few more at it, but I feel like I'm just an hour or two more of coding. Yeah, I had that. I totally nerd sniped myself with this where it felt constantly like I was like, oh, I think I'm like an hour from done. And then I looked at the clock and I was like, oh, it's midnight. I got to go to sleep. I'm an old man. I can't be staying up all night coding, even if it feels like just one more bug. But it feels like I'm just one more bug from done with porting this over to Lua. Oh, and then I guess I'm just one more little feature of I have to find baby's first Lua Nginx module and wrap that around it. But again, that feels like it's just an hour or two of coding, right? It's not much. And then I will have trace IDs that I will log into Nginx. I wonder if I could get Nginx logging in JSON. That would be such an improvement. So if I had that, then I can give it to Rails, and then Rails can log it. Then I can give it to the database and the database can log it. Maybe I can upstream it and everybody else can have it. I don't know. This is kind of toy code. I've been thinking a lot about Lua lately because it's a very neat language. Because it was designed for embedding, It has a bunch of nice properties that you only see in like OCaml and Haskell, where you can modularize code. A really common pattern in Haskell and OCaml is you make a module like, I think I do it in UUID. Yeah. So, I guess I don't do it here. Didn't I do it? I did it in type ID, and I'm not ready to show that because we never refactor so much of the interface, but there's an idiom in Haskell where you make a module. Imagine we make uuid7. Lua doesn't really have the types for it. But if we say that this is the only way to get a object of the type, a value of the type uuid7 if you make it so that the module see this just exports the whole module but if the module doesn't export a raw constructor it can export only a constructor that does validation so actually base32 is a better example here because like if you try to encode a string and you have a character this was a bug i had base32 only allows you these characters and so if you pass in a string that has Say, and I in it. Because that would be confused with a one or it has an uppercase letter in it, I know it's not valid right out of the gate. And I can throw it away. So if you had this concept of a base 32 string and we had the base 32 module and the only way to create a member of this type was to call the constructor that validates it. then you know every time you have a base32 value that it passed the validation rule. And so this allows you to lean on the type system and be expressive in a way that dynamic languages are not often. I swear I have a point here, hang in. So if you have a thing that says I'm a base 32 object and you know from the type it can only have come from this module that forces you to go through the validation, you know every instance of that is valid. This isn't like Active Record where you're constantly slopping around invalid objects and getting bugs and no method error on nil. And I say this because that pattern, I can do that in Lua, where I export this base32 object, and if I want a function to be private, I just don't put it in this little namespace to be exported, sure. But there's also a program-level version of this where, because these things run in a Lua is designed for embedding. So you can even have the idea of I'm going to load untrusted or marginally trusted code and then only make certain things available to it in its execution context. Like you get to use the standard library, but you can't pull in the socket library. You can't pull in these other things. I think that's really an interesting model. So like Roblox does this. It's a very popular video game with the youths where you can kind of script your own video games in Lua. And then they have their own fork or packaging of the language called Luau where, you know, their video games are not allowed to open arbitrary network sockets or browse the file system. I think that's a really interesting programming paradigm, and I kind of want to lean into it. And so that is a long way of explaining why I am tinkering with writing things in Lua, is it has this property that is sort of like a neat thing I know from functional programming and expanded from are we talking about values to are we talking about execution contexts. I guess what I'm saying is that Lua has solved the halting problem. So I can know that an arbitrary script isn't going to dink around and do things I don't want it doing and only execute for a particular period of time. I like that a lot.

02:34:46Anyway. That's my big ramble. That's the thing I wanted to show off that I'm probably going to shove into the site in just one or two hours. Just one bug fix. Just one Nginx thing. I'm almost there.

02:35:13I kind of want to get back to that instead of... carrying on here because there isn't there isn't so much to do there's no pressing bugs now that all the display stuff is closed and oh chambliss if you didn't see wasn't it wonderful that none of the display bugs got reopened in the last three days i kind of thought with my giant shotgun revert we were gonna get something but zero bugs reported from that that's actually And every bug I closed stayed closed. Nobody said, hey, this is still broken. chamlis_ I think #1544 was meant to close but you missed it?
So isn't that nice?

02:36:05chamlis_ but yes, good job on the big ol' revert!
1544. Oh, yeah, I'm not trying to pat myself on the back. I am trying to pat us on the back that that went well. Oh, this was the one around deleting. And then did you have the one that was around querying?

...34Hadn't you? Okay. I kept this open because of your note about a deleted comment. Yeah. Here's a good thing to spend a minute on as long as I've got, I don't know. 25 minutes left on office hours. chamlis_ sorry, wasn't trying to be sarcastic there - that revert could easily have broken a bunch of things
Yeah, I think what I wanted to do was check that we understand what's happening here, which is, yeah, threaded views shouldn't use accessibility user. No, I didn't take it as sarcastic at all. It didn't sound that way. I feel like I've got a pretty reasonable understanding of your writing voice at this point. Don't worry about it. So I think it's two bugs. It's number one, we shouldn't have this filter. Because it's okay to just have a tombstone there. And then second is... Is there something different about the way this page is rendered that instead of getting a tombstone is causing a bug?

02:38:09Oh, it's the... And you know, as soon as I say it, I get it. It's the... It's the jiggery-pokery in comment or in thread. Yeah, it's in thread. So this thing, it's comparing the depths and it's building up state. So there is, for anybody who's not chamless, we commit ridiculous crimes in the name of performance. And one of them is that we try to chamlis_ I think just deleting accessible_to_user will fix it. if you wanted another check you could assert each comment is at most one level deeper than the one before it I guess
only loop the comments array once and have it flat but we render nested stuff and so we kind of maintain the state and the bug is going to be right here where if a comment is deleted and removed by accessibility user we're going to go from depth one to two to three right and then if two gets deleted we go from one to three and then not having rendered the opening tags for two we render extra closing tags at three and that's why the page is up that's it that's the bug

02:39:45Yeah, I don't even have to add the assertion because I assert that I'm correct. So what that's saying also is because of this optimization, accessible to user can never be safe because we don't want to filter out those comments, we have to render the tombstone.

02:40:08So now I want to audit every use of it. So search renders comments in a flat, unthreaded mode.

...25So that one's fine. What's this one? Respreads. That's this same bug.

...48Who else uses it in the comments controller? Index. indexes of flat rendering so it's safe and who are you upvoted upvoted also is a flat rendering of comments so it's safe but user threads is nested so it's not safe okay yeah we just gotta drop that out

02:41:21And then it's okay to drop out stories because then you would just take the whole tree all right.

...36So this has a flat render so it's okay let's add this comment.

02:43:49There we go. So if I say that and I say this, this is also incorrect. Yeah. So who calls recent threads? Because I want to make sure I'm not going to break something. Yeah, the point of this

02:44:24chamlis_ recent_threads is only returning thread ids though, right? so maybe it's desirable to filter there
oh actually this is yeah so recent threads it is i was thinking about this because recent threads one of the reasons i put this here was imagine alice has gotten into a flame war and so if you pull up tilde alice slash threads And the first five things are deleted, deleted, deleted, deleted. You're like, oh, this Alice is a big jerk. And it becomes kind of a shaming thing. Whereas if those are just gone for other people, it doesn't look like Alice is a big jerk. It just looks like you see whatever the last anodyne conversation she was involved in was. so if we filter out those that are accessible yeah the way the recent threads appears is you see their comment first and it does show a parent if or no it does climb up the tree to show parents yeah this one's fine

02:46:10So yes, Janlis, you're correct as usual.

...56are not user threads.

02:47:10I almost want to change accessible to user as a name to express the idea that it's only safe in threads. All right, it's only safe in the flat context. But it seems weird to mix those two things up. I'm gonna just leave it, I think. I don't have a better idea off the top of my head. All right, so let's get that deployed.

...48Oh, I typed equal ansible instead of equals lobster. Why are we auto-completing not the deploy? yeah so i'm getting run down we're coming to the end of the stream here it's the last couple of minutes so if you have any i should turn off the the footer so people don't show up just in time for me to end the stream i try and turn it off a couple minutes before i start or a couple minutes beforehand see how tired i'm getting i knocked the site offline because now i can't deploy without outages The solid queue folks seem to be pretty responsive, but if that doesn't get... And gemless... I know you heard me thank you here on the stream, but I'm also doing it in public because I want people to see that I appreciate your work. All right. So they already closed my, did they fix it? Oh no, this is the, this is the one I linked to. I was like, wow, did they already? Which bug is mine? Why am I doing it? Wouldn't mine be linked in here because I just linked to it? Nope.

02:49:47There we go. So anyways, the deploy finished. The site is... Back up. It bounced. But it's back up. I'm really... If I can't... They seem to be on top of issues because for a major feature, 44 is a low number of issues to have open. So fingers crossed that they can help puzzle this out soon. Maybe I'll look into the solid queue source so that it can support phased restarts. That would be a neat contribution. But maybe I'm going to revert to... perfectly working systemd service I had for Solid Q. We'll see how quick this gets attention. I don't know if it's going to be simple or giant. I can't speculate. Well, I can speculate. That's my job as a Twitch streamer. I wonder if the person who wrote the plugin wasn't familiar with this phased restart feature. And Because that's really what this feels like.

02:51:12That they should decide to crash the supervisor process because the worker went away. That also is an odd choice. Why not just restart the worker? I thought you were managing it. I guess they didn't want to get into should we be in charge of yeah what if the worker itself has some code monkey patched into it or has a bug and the worker is crashing on boot well then you are re-implementing systemd's service monitoring yeah that gets complex fast so okay that that makes some sense from this perspective but it shouldn't fire during a phased restart

02:52:02We'll see how complex that fix is, but it's going to hurt the heck out of me if every deploy is an outage. I have gotten very comfortable with being able to deploy three or four times in a row and having no interruption. Zero downtime deploys are super great. Alrighty. Well, this has been the Lobster's Office Hours stream, and I have been Peter, and I... I appreciate folks who tuned in to talk about the site, to run fun queries, to look at my silly ideas of writing things in Lua, or to help improve the site's guidelines so people are cool about things and understand how it should work. Thanks for helping build the community. Thanks for helping fix bugs. chamlis_ thanks for the stream!
That one's directed at Chalice. Thanks for... hanging out as we code. The next scheduled stream is going to be Monday afternoon, Chicago time. It's on the Twitch schedule thing, and there's a link to that from the stream archive. See you again on Monday. Take care, folks.