No, it's a different difference

01:43deploying the whole like sales, what it is site. If you look at the domain, I've had a archive page up here for too long or not an archive, just like a placeholder marketing site. Like here's what it is, sign up to the newsletter. So you hear about the announcement and that is now replaced with, I'm using the jumpstart boilerplate by the same person who made Well, I guess it's a team, same team that made Hatchbox, which is what Lobster is deployed with. And if it wasn't pretty obvious, also that Reach Excel site will be. So on sometime between now and the next stream, I am going to... Where's my tools? Scene switcher.

02:52Did this work? That worked. So sometime between this and the next stream, I'm going to start that beta. If you are interested, well, you know what? There is not really a way to sign up for the mailing list right now. So you'll just have to already have been on it. Where to go? And then I think next Monday, I will probably do a demo of adding the gem to Lobster's. pushcx This is Lobsters office hour, ask questions anytime!
All right, so this is Lobster's office hour. Ask questions anytime. All right, so what else? Otherwise here, if folks don't have live questions, I'm going to work through pull requests and issues.

04:00I had hoped that would get me issues by last updated, which is what this sort should be updated dash desk. Okay.

...16Did I close any pull requests since the last stream? Why do these have the notification bar next to them? Let's review. Oh, this one. This one was a really interesting interaction. So MarkerRoth dropped by the last stream. We talked about HTML. He ran his tool on the Lobster's repo and found some issues like pushcx https://github.com/lobsters/lob…
missing open and close tags kind of stuff. So if you look at the PR, you will see it is not huge. None of these are like hair on fire emergencies, but it's nice to tidy all of these up before they become the most weird bugs.

05:14And then let's see. I've got to find, let's see, this guy, Marco. Here we go. Here's one of them. pushcx https://bsky.app/profile/marcor…
So Marco and I, a day or so later, had a conversation on Blue Sky where he said, what are some of the most annoying things you see or do yourself in HTML and ERB files that you wish a lender formatter would catch for you? And He had a bunch of issues queued up. And then, why'd you go way over there? Where did you go? I don't want tab groups. I want the tab you just threw away there, Firefox. All right, here we go. So I chimed in with a bunch of stuff that we have seen on lobsters. I think there were two tweets. I only see one. Yeah, when you're logged out, you see less. But all of these things are bugs that we have seen in the Lobster's HTML and ERP or that have caused bugs. And this is just what I remembered off the top of my head. I didn't actually have to. You know, if I was really answering the question, I would run Git log app views. and look for stuff, but this was easy enough. So I don't know why I can't see my second message, but I also mentioned it would be really nice to have context-aware linting where this tool prevents XSS by knowing that an HTML attribute needs to be escaped different from something inside of a tag, different from the actual tag content itself. And I am aware, what did I say? I'm aware it's huge, but any incremental help would be appreciated because there is the dirty flag, basically. I don't know what Rails calls it under the code. Maybe it calls it HTML safe and you can escape. You can call .html safe or you can call raw something. twitchtd hi pushcx
And it prevents the worst of the foot guns, but Not all of them. And then Marco picked these up. pushcx https://github.com/marcoroth/he…
And so if we search for is issue push CX. Yes, he is an issue. There's this whole set of issues that he got out of. Oh, Hey Thomas, that he got out of my suggestions. And I think the open close tag thing was already in there because that's what he ran to create that PR. But then the rest of this, like this CSP is for that pull request that's open right now. And he wrote these up pretty comprehensively. This is a significant amount of work because it's not, you know, my three words, it's here's this. And then he correctly understood from my quick notes, what squeezing shit in the 300 character limit of loose guy. And he expanded that into a clear rationale and The examples are all exactly what I would have created had I actually done any work instead of just listed some bugs we had. So this was actually super nice to see and makes me more excited for Herb Linder. My fingers are firmly crossed that this helps us with all these kinds of bugs.

09:19Let's see, what else do we have? So Thomas, I'm looking at your PR, and it has a notification. Oh, I hadn't seen this. Ah, and then we picked it up with, in your other thread where we talked about, say, a validate false and wrapping a transaction around the whole thing. twitchtd that's an old comment by now, looked at it in the other issue
Yeah, the lifecycle hooks. It's my big gripe about active record objects having the ability to be... having the ability to be in an invalid state. It is one of those original sins of rails from Caius. All right. So this is just me catching up on notifications I missed. Oh. Huh. Nav mic. I thought I learned view assigns from nav mic. Where did we do it? I guess we did it in the home controller spec before he did it.

11:08Hunter did it. Aha. So we both have to thank Hunter. Okay. That's 355E3B. I'm not on the site for anybody who doesn't recognize. In case you never picked up that wonderful pun, that is the hex code for Hunter Green.

...32So I only missed a thank you. Some of these I saw in email, so I'm moving pretty quick. Yeah. And then Thomas, this is the PR you're thinking of where we were discussing or the issue where we were discussing your script, create a bunch of fake data.

12:05Skip slow queries for flag warning. Oh, Thomas, this is super roundabout. It's... And it's also wrong, right? And it just flickered. Why is Firefox jumping around? That's obnoxious. Please don't jump around. Jump up, jump up, and get down. Let me double check this message.

13:22Alright, dev flag warning.

...50so this is the thomas this is the skip logic here in possible flag warning if you're in development or test mode instead of running this expensive query you just get that little partial saying we didn't run the expensive query and that flagged commenters query aggregates like twitchtd ohh
every comment and vote on the site and is not fast. I want to say a couple of seconds to run typically, it's been a while and on the new production server. So yeah, so let's grab this name and go over here.

14:55twitchtd I thought that it was referring to the replies
no no so it's just it's just that thing above so it has slipped off of my current projects but i have a half finished thing i started on stream where i was updating this but the gist of it is When people get a whole lot of flags across a bunch of stories, and I think off the top of my head, you have to get at least five or 10 flags across three different stories from multiple users. And you have to be, I think, the top five most flagged users in the period. you get a warning message on a couple of pages because this is super unusual. In a given year of posting comments, the median commenter gets zero flags and the average commenter gets, it's been years since I ran this stat, but years ago it was like 0.5 was the average. And it is lower since then is my hunch.

16:23twitchtd ok, thanks for explaining it
So I've been revamping this because this message is kind of heavy handed. It was written in a period where I was kind of cleaning up a bunch of trolls. So let's talk about

...54Frici Morning 👋
it does still serve a purpose but i am trying to make it a more general purpose standing tool oh hey friji saw you on blue sky the other day there where i was what was that yesterday where i was joking about the slop guy who bent my ear and then god tried to get his followers mad at me like tag me into a conversation so they could post mad things at me. It was a brigading attempt, but it didn't work. I think he got like three replies from very confused people who were like, yeah, posting slop is great. No, it's not. All right.

18:0625 what is it oh 6 26 oh man lying through the year you know we're only a week or so out from the lobster's birthday where's my calendar i wanted to stream recheck on monday but is that gonna be Oh, okay, so that'll be July 1st, and then July 3rd, Thursday, will be Lobster's birthday. So, okay, we'll have a special, like, birthday-themed office hours. I wonder if I can get, like, a terrible image into the background of the terminal, like Hardy Hat or something. I don't know. If you have ideas for Lobster's, it'll be our 13th birthday. Site started in 2012.

19:02Thomas, did you have any other questions here that I can help with? No. Okay. Oh, new pull request from nav. Mike who's continuing to chew through story repo. And then I think. Both of Chambliss' PRs are stalled on me, and because I'm running shorter streams, this is my mistake. I'm dropping the ball. All right, so NavMic has done a bunch of these, so I'm not... I'm expecting this to be pretty easy to merge. Story, news by user, yep, yep.

20:01Deleted, mod preload, yup. Hmm, that's kind of an interesting question. Should stories that are filtered out by your tags be hidden from a user's newest stories? I lean towards no. It's such a specific thing of show me the last 25 stories that this user has submitted that yeah yeah i really think the tag filter should just filter the views in home controller so i mean i know this one is in a home controller oh this is nice i like that he mirrored my style i appreciate that pushcx https://github.com/lobsters/lob…
When I say mirrored my style on the. I want to say after his last pull request I. tidied up the tests, a bit because we don't do a one assertion per test and I wanted a clearer helper. yeah so this all got tidied.

21:39But we lost.

...50Yeah, this I actually do want to keep this test unless it appeared somewhere else. Oh, OK. It moved over to the scope. That's fine then.

22:05Good thing I scrolled down. Okay, great. So this is very thorough.

...15It's a clever little helper. I like that.

...55So I'm going to go ahead and merge this.

23:04I'm just going to squash it down. It doesn't need to be separate. Cool. And then let's deploy it.

...26you know, I don't actually need to fetch before I do it. It's just such a habit where otherwise I end up with surprising little merge conflicts on my local. The deployment from Hatchbox happens directly from the main repo or from our production repo here, the master branch. And so It's set up like that so that I can't forget to push a commit publicly as I deploy the site.

24:07Because it is a bit of belt and suspenders for me, because especially if I changed some bit of mod-related functionality and hit deploy and then failed to push it to GitHub, which would be a really easy step to miss, even if I scripted things, it would just look super bad for transparency. So this one, Regis and I talked a little bit more about this on IRC. And then, yes, he left this wonderful long comment explaining what these different settings are, and this is great.

25:12All right, well, let me, so I don't usually have IRC up when I'm streaming. don't see Regis talking, so I don't feel the need to bug him, because I am just going to say, great, I understand. This is doing and have. And I think that will clear up.

...55pushcx https://github.com/lobsters/lob…
let me skim it one more time but i don't think this needs anything else so this is i should share the link this is the postfix config the server gets spam because of course spam bots try random addresses at something else and hunter's theory is that since we do username pages as like So a couple of years ago, just as a cute little Easter egg, I switched to this tilde whoever instead of slash you slash username. And this is a callback to old Apache mod user der.

26:49Excuse me. And Hunter's guess is that we're seeing more spam because spam bots never really stop trying strategies. They've just been around for decades. And they know that if they see mod user dir, there is probably a user named eHomburg at this domain. So we're seeing spam because I put an Easter egg in the URLs. I don't know. Easter egg is... It's really not, because it's not hidden, and it's not about getting credit. I guess if I was the movie detail subreddit, I would say it is a subtle nod. It is a reference humor. I don't know. All right, so all of this looks good. And this is how things tie together. It is a lot of config to say something small, but that's boost fix for you. All right. Cool. Let's merge.

28:11I have to wait on the workflow to merge. That's fine. All right. GitHub. That was so tacky. I click merge and clearly it enqueued a background job, but then it went, oh, there is nothing blocking anymore. And the whole thing disappeared. And then it came back. GitHub is so here. While we're waiting on this, do I still have this screenshot? I try not to like roast people in public because things are hard. Yeah. All right, so that's a different bug. There it is. Can I just make a tab and drag this over? Yeah, so yesterday I saw this bug two or three times separated by many hours, like eight or ten hours on GitHub, where something went wrong loading the css for this thing as i navigated around repose and then things were doubled which i don't know this is such a single page app kind of bug and watching this stutter through inconsistent and invalid states

29:49I don't know, GitHub has gotten picked up so many SPA bugs in the last eight months, and they're They're not huge because they're all just visual jank. Very few of them have actually blocked stuff. And if I do a hard reload, this stuff just goes away. But it's a surprisingly, I don't know. GitHub was rock solid for what? twitchtd oh man, that ui XD
15 years? More? And then all this kind of jank started happening in the last six months. When did... Yeah. When did Microsoft announce that they were forcing all their... Well, not announced and not forcing, but that they were encouraging and incentivizing all of their developers in a slightly deniable way to use Copilot all day? Because Microsoft's style is they just like... They're like, oh, we just... Frici recently-ish
think it's great and that developers should want to mention on their perf that they did it. But what that actually means translated by employees is if you don't do this, you're going to get bad perf.

31:17Man, why is this? Oh, it's the slow task. I forgot that this runs it. twitchtd posted a comment to https://github.com/lobsters/lob… with additional questions rather than twitch so we can keep context
yeah our test suite is getting a little bit slower the last year or so 26 seconds is longer is there just the one commit two commits redundant entry all right so if there's just a tiny fix up i am going to squash and merge why is this co-authored by this is authored by let's make a merge commit i don't know github

32:05Oh, look what important word I forgot.

...15Gotta say please and thank you, right? If someone is going to be generous enough to both use their expertise to improve things and explain it to me, Captain Dummy, Oh, thanks. Let's take a... I'll pull this up because it's, you know, still in my head.

33:4012.

34:21The so Thomas how about this inbox tabs for various users.

...37

35:15called inbox count. I think it was. Yeah. And then I think also replies count. Yeah.

...40No, it's just unread.

36:00Thank you.

...50twitchtd How do you feel about polymorphic associations in rails?
So one sec. Got a cough again.

37:06Sorry.

...13I am.

...21I try to avoid polymorphic associations and some of that is they were a little rough for the first year or two they were in and that's when I first tried to use them. And some of that is as a consultant, I saw a bunch of code bases that used them when it didn't need to and had Perf issues and complexity issues based on it. So my gut feeling is pretty negative on it, but they do, sometimes they solve a problem that really nothing else would. So it's possible in my opinion there, that gut feeling is just outdated at this point. but I do try to design away from them. We have, so this is one of the oldest parts of the code base. Actually, in a way it is the oldest part of the code base, but the moderation log, rather than have a polymorphic join, has optional foreign keys for all of the connected tables. And I think if I go down here, there's a validation that you can, yeah, I added this. I want to say one foreign key present.

39:04So I look at this one, and especially as I've added things like category and domain and origin,

...17This design feels less and less defensible compared to polymorphic associations. At the same time, the table is not big.

...37Yeah, 12,230 mod log entries. And then this thing is three lines of code.

...57Yeah, you're kind of hearing a lot of silences. on it because if I don't have a strong opinion, I kind of recursively go like, okay, well, what are the criteria by which I would form an opinion? And off the top of my head, it's things like, you know, I always go for reliability, clarity, low likelihood of producing bugs.

40:28and these two approaches on moderation at least are basically equivalent you know for for 12 000 records oh wow so we allocate an extra honestly it's not even an extra four big end columns because in the polymorphic version we would have the string for the name of the associated table and then you know two lines of code that don't I think I wrote this once. And then I only update it when I add something. twitchtd the foreign key constraint is definitely missing for polymorphic types
So like there's a little more potential for a bug here because you have to know that the list of keys matches this list of keys. Which I guess mildly means a polymorphic association would be better. I really don't have a strong opinion here, I guess is what I'm saying. Foreign key constraint is definitely missing for polymorphic types. Yeah, I do like that a lot, actually. Because Rails is not great at foreign keys. At least it doesn't error about them anymore, but that's been a long time. That's just me, old man, griping. Let me be reasonable here. rails doesn't lean on the database enough and this is one of the ways i guess if i can get a get recheck shipped in the next couple of days, that really doesn't matter, because I can write a checker for that and know in prod if it's actually a problem and know the instant I create a problem, right? So why don't I solve my own problems?

42:36twitchtd hah, ok, fair enough
Recheck, honestly, the first seed of it was a talk I gave at RailsConf. Wait, was that one RailsConf? or was it Kansas City Developers Conference? Either way, it was a talk I gave nine years ago about some problems in the active record model.

...59Can I meaningfully advance either of these? What's going on with CSP headers? Oh, okay, this one's actually not waiting on me. Channels, any chance you're around? I know you sometimes drop in for these Thursday streams. If you are and you need anything more on the CSP pull request, I'll take a look at it now, but I don't think so. twitchtd thanks for giving your feedback, seems like I still need to generate more data to see perf issues and think about how to unify the notifications and pms
there's just the thing about the login page and if we replace the markdown helper there's nothing yeah not login page sign up flow you know what i mean and then this one common marker this is one of those that tickles at the back of my neck you know like i'm out walking around running errands and i'm like When is Common Marker going to break? Because we pinned it two or three years ago. And this kind of slog through 100,000 comments is actually not something I really have the attention for. 2017. pushcx https://github.com/lobsters/lob…
So the first we can so channels did another great spelunking job here let's share this one. yeah you know Thomas. If you look at replying comments. This is the slow view right, this is the definition of the view. and Part of the slowness is just how many predicates are conjoined. And I don't immediately have a gut feeling for where your synthetic data is meaningfully different from broad. But definitionally, it pretty much has to be here of, The number of replies or the number of flags or. yeah if it's a cardinality issue like the distribution of flags, this is going to be real Harry for you to reproduce and I don't know how to give you more feedback. twitchtd yup, the view is very slow when you count(*) (like 20 min to query it) but when you add the user_id + other filters, it's actually very fast for me
Do you need to reproduce it, though, because. Part of the design goal of... Yeah. Part of the design goal for redesigning notifications would be to materialize those notifications as a model, and then we don't have a view anymore. Can you just believe me that the view is slow and is ready to go? It's fair if you don't, but the other way, how big is the, what's the cache? It's just a, it's a read through cache here somewhere. twitchtd I was hoping to repro to make sure I can repro the issue so that when my I build a solution I can see it fixed.
twitchtd But I can work on belief and.
yeah the other way for you to do this because i believe you're a occasional commenter on the site is leave a comment or two wait two minutes and then you will see it in broad because the page you load more than two minutes later will bust this cache and hit the view

47:29Yeah, but I mean, the view works so fundamentally different that versus we insert a notification based on when some porting of this fires, right? Hang on, let me say that very differently. That came out confused. So the way the notification model sort of has to work is when someone posts a reply, we run some version of this of, okay, well, the replier didn't flag this person and the comment being replied to isn't heavily flagged and the thing isn't deleted, right? Okay, let's insert into the notifications table. But necessarily there, you are selecting so many fewer rows because you were looking at the individual reply and the story it's on rather than like all of our biggest tables stories comments read ribbons

49:01twitchtd Makes sense
And I guess the other reason I'm sure it will go away is if that's slow, or even just prophylactically, we can move that calculation and insertion of a notification into a background job. So the user posts a comment, we insert the comment, we serve them a 200, we enqueue a send notification to op, background job, and then, you know, even if that horrifically takes two or three seconds, that's fine. User doesn't care. They got a fast page. Having, man, solid queue, it's funny. I want to shift so much of our work over to it just because of how much performance benefit twitchtd true, background job to gen notifications seems like a good backup in case things get slow
can get out of it by moving work out of line on controller actions. And I just started this morning. I haven't finished it, but what was it called? So Rosa Gutierrez is the primary author of Solid Q. Rosa Gutierrez. Watch me fail to spell names. Where is this new podcast? pushcx https://onrails.buzzsprout.com/…
So the Rails Foundation has started a podcast and the first episode interviews her. So let's share this link.

50:58I've really been... more impressed with solid q the more i've used it and it is i can't even count how many years but this is the best feature in rails in years because having an integrated background job and recurring job that is rock solid and only a very small extra moving part added to our production setup and I haven't yet managed to do something so weird that I get nonsense out of it. Or, oh, you didn't consider my use case. It has every use case I've wanted to throw at it already. And it is just works kind of software. I'm really impressed by her work here. And so it was really interesting to hear this, the process she went through of... building it by absorbing use cases from hay.com one at a time while live migrating the whole thing over from rescue which woof that's a that's a hairy kind of migration to do

52:26twitchtd resque was like years not updated years back when I used it :(
So I'm really, I'm really enjoying the more I use it, that the happier I am with our code base and the other nice thing, which, so that was Hannah, Vivian Shaw, Viv Shaw on GitHub took on, on jobs. most of these things to move our cron jobs over here to background jobs ah it's so nice actually having these all wired up with telebugs because now when things fail in the background they're failing individual entries you know so it is one job one sync of a mastodon user rather than This whole script wedge and you have to rerun it from the middle. That's a pain in the butt. Oh, and you know, I had mentioned on an earlier stream that. With one of these. Right. Maybe it was like the mastodon sinking or it was the web mentions, but. Yeah, actually, I think it was the web mentions. we i kicked around my unfamiliarity with how solidq wants to be used and background jobs in general when i said like do we want to directly port this where we hit we have a job that runs per comment and then it sends all of the web mentions or i should say per story and sends all of that story's web mentions or do we have a job that runs per story per web mention or per story per mailing list subscriber right how do we break down the jobs and gutierrez here oh the i is after the t that's where i was wrong Based on her experience said what you want to do is decompose the jobs pretty much as much as possible, because the experience of resuming a job or trying to get a job that half you know failed halfway through is. Not an interesting chore and generally not worth doing that you want to just decompose your jobs as much as possible. and given the. The very high quality out of Solid Q, I'm happy to just take that on faith.

55:32Anyways, I've still got like a half an hour on this podcast, so I'm really looking forward to it. This is kind of a recurring theme of these is Peter is impressed by Solid Q.

...50Why don't they build the whole framework out of Solid Q?

...59All right, so let's look at Chanless's other PR then. twitchtd I haven't used solidqueue much, I used to use goodjob for everything in the past
So this was 2017 4.12.

56:22twitchtd but will check it out
Yeah, I've used good job rescue. What was the other one delayed job, sidekick, of course, and a bunch of others. But none. None where I wrote a significant amount of code, you know, like I touched them, I continued the existing pattern. But I didn't write a hell of a lot. So have this in history and we're looking for comment out where created at. What is the way to say greater than with dates?

57:20Oh, it's with a range, but I can't make a range because I have a string. Let's just... It's funny, I reached for the Django solution of, oh, you call it, create it at underscore underscore GT. And it's like, come on, I haven't written Django in a decade. Why did that float up first? So 2017.04.12, right? Yeah.

58:01Give that a second. So this has ridiculous n plus one queries, as you can tell. But this is a one-off that I'm manually running in the terminal. It doesn't have to be terribly performant. would be nice if it finished sometime in the next couple of seconds but you know if you look at our our stats graph okay so here's 2017.04 is here so all right what's the area of this triangle What's the area under this? Most of our comments have come since that change. But this did just finish. So let's take a look. Short IDs count. 26.5. That's promising. Because what did I fail? 110, 112, something like that. Okay.

59:31You know, this is the kind of thing that I can have the LLM slop a one-off script for. Load this list of comments. Ignore this particular change. So like the change between turning the m-unicode... Excuse me. God, I don't know what this is today. ignore the change between the m-unicode character and the m-html entity filter all those out show me five do it again and just kind of chew through the differences as opposed to this is yeah where's my

01:00:59the other hand just eyeballing these things is not so bad in the terminal it says smart quotes again no it's it's no follow yeah so that is at least part of the difference

01:01:38Okay. So the thing is, what I would really like to be able to do is just say like where markdown comment doesn't include nofollow, but I can't quite do that because maybe there's some other difference here that I'm not immediately spotting because I'm just eyeballing instead of even running diff. I really don't want to build infrastructure.

01:02:11I also really don't want to play whack-a-mole.

...24Oh, wow. Thank you for that autocomplete.

...31So this is also a nofollow difference.

...39And it looks like that's probably the only difference. And in case you're wondering how I'm spotting these, what I'm doing is just mentally I'm zipping up and down to see if the lines end at the same point, because once we've addressed that issue with smart quotes, that's some comments on the PR, if you look. Anything that adds or remove characters tends to stand out alright so let's let's try that it's just filter goes out. Which is where not.

01:03:35And I am aware this is gonna under select comments, but that's okay. It's enough to make progress.

...52Hey, wait a minute. You know what I'm doing? I'm not on the branch. I didn't switch off a master. So I think I added nofollow a couple of years ago, didn't I?

01:04:14Didn't think that alarm was on. Yeah, there we go. Eight years ago, we added it. Why do we ever change things?

...502000 what in the spacing why do you render that different what is the source on this the source is two spaces something compacted the space because html doesn't care and it runs multiple white spaces together that's pretty minor all right so let's get over to the branch which JJ bookmark list all right we got a few and these are not in a useful order

01:06:07There it is. Weird that it didn't show up in the list. So let's jump over there. Bring back up the Rails console. Run the latest version of this. It would actually be super promising if we got the same number as on master, right? What was that number? 2050 something? 2096? Almost a round number.

...58this going slower because oh yeah so every time it's doing reprinting this it's going into the next thousand comments so there are few enough we can actually see the progress and then in between there are like three of these things rather than 300 so this is actually really promising

01:07:35What is it? There's roughly 600,000 comments in. 586. Okay, so we're coming up on halfway.

01:08:05actually this full text search is slow enough it probably would be faster to do on the real ruby side i don't know so if we get a number in the neighborhood of 2096 we immediately have pretty good confidence that this branch is correct

...39So while watching this, I've been chewing on this idea. Do I want to build infrastructure around it? And as much as I don't, the comments are the heart of the site. They are the thing I'm most paranoid. Well, after security, they are the bugs I'm most paranoid about is losing or breaking comments in some way.

01:09:07So I think that means I really do wanna build something that is gonna compare all half a million comments and be smart about, let me know what each individual diff is so that I can, when I say, oh, the nofollow change is fine, acceptable, it ignores that change and not that whole comment. What does that look like as an algorithm? Can I regex it? I mean, I know there's that classic stack overflow answer of parsing HTML with regex.

01:10:02pushcx https://stackoverflow.com/quest…
Yeah, this is the one.

...08Classic bit of internet lore. Let's throw it in the scratch.

...26twitchtd I've parsed html with regex before, but your html has to be predictable for it to make sense imo
PudottaPommin Good afternoon VoHiYo
And I'm actually not gonna scroll down and spoil this one. You should read it on your own. However, it is okay and reliable. Oh, hey, Pudota. You're coming in right towards the end of the stream here. PudottaPommin Nooooo archiv32Uhoh
But since the output of Markdowner is so regular, I'm not trying to use Regex to make a general purpose tool. I am just treating it as really a text blob and looking for individual things. So how do I do it? PudottaPommin I was fixing sink NotLikeThis and missed this
Do I treat it like the Stripe API upgrade thing where the things I want to ignore are a series of regex transformations of the HTML? So if I, fixing a sync, oh, that's kind of a fun puzzle. No worries well, there is the archive right if you haven't seen that it's linked under the. PudottaPommin I will have a look then
video it's push dot CX slash stream and I typically have the archive videos up in an hour or two it takes a minute to transcode them and sometimes I wander away and don't push the deploy button. But. PudottaPommin Im curious about this comments lore
yeah. So. If I take the markdown comment, and then I apply a series of regular expression G subs to it. So like in the case of these links, it would be update the link to include that nofollow attribute that I added, or I'm sorry, no, that contributor added eight years ago. And then we compare to the output of markdown. common marker rather. PudottaPommin Whats the lore behind "markeddown" word
Yeah, then if they're the same, we can ignore that comment and I can do the same thing with smart quotes and the dash encoding thing. And it seems better than trying to say that kind of sort of equals with two strings, right? Because otherwise I would have to say, okay, do my sort of equals, show me the div, the lore. Youth detected. My nephew does this to me. He's like, oh, a new uncle lore just dropped when I mentioned some story from 30 years ago. It is past tense. We took... the markdown that the user entered, we rendered it and we shoved it in a database column so we never have to render it again. And that is why it is past tense. I probably would have called it rendered comment. But I didn't name it. PudottaPommin Lore sounds might sound less agressive over "hey, why its named XXX" :D
At least it's clear. PudottaPommin I see, ehm, hear
I just never managed to type the two Ds correctly in the middle. Lore sounds less aggressive. That's fair. I do a lot of these like let's talk about why something is odd in the code base. So one of the things you missed was Thomas asking a couple of questions for his work related notifications of why do you call this thing slow? What's up with this warning? Is this dead code? You know, doing these streams for what has it been eight months, something has really knocked down any sense of, oh my gosh, they're going to see, I make bugs. They're going to see, I make typos because you know, after the first thousand, it kind of loses its sting. All right. Home stretch here on the comments. And I haven't been counting. but I do believe we're gonna come in significantly above 2096. Just kind of hunch.

01:15:19Yeah, there's a lot of these. And it also feels like there have been more of them The later we get the more recent comments have had significantly more like the average number of lines between these blocks is increasing. which could just be a style thing like people have started, including more links or started at mentioning people more. wish I could have run a poll about this. yeah 24,000. Alright, let's look at the second one. Because that's what's in the history. That's... Whoa. Oh, some kind of code styling. Interesting. Why does this... Huh. I wonder if the first one is like this too. No, it's a different difference.

01:16:39And I don't immediately see it peeking at the line endings. So is it a, it is probably something like this hyphen or this colon or something has a different code point. So if I took this and I said, come here.

01:17:10is dot code points dot join dot code points dot join okay so that's more apparent the second one is just a little bit longer two or three bytes well two or three code points and the difference is somewhere in the third line

...47PudottaPommin 46
PudottaPommin 101 100 46
32. this is the thrilling part of maintaining a social site 60 what the hell is ascii 60 maybe it's an angle bracket but

01:18:20Huh. 46 versus 101, 146. PudottaPommin You found it :D
PudottaPommin Before checking chat
Yep, you got it. There's an extra. What is this, 6098? All right, so. What are these? 60. 98. There's an extra bold tag. How am I not seeing that? That's pretty apparent. Is it? Oh, it's an extra. Block quote move. Wait, so block quote modular slash block quote. Block quote modular block quote. The BR. How did I not see that? Huh. I mean, the second one is wrong. That's not a change we want. Okay, so. All right.

01:19:58Do you have any last questions? questions about the site, now is a great time to do it. Because otherwise, I'm going to wrap up after this comments.

01:20:11Yeah, extreme. What did I say today was 0629? 26. Actually, no. But nofollow is my own difference. Not mine. Why do I think I introduced nofollow? Oh, cause I touched it on. I did the way I'm confusing links. twitchtd I'm going to logoff, thanks for office hours!
So the nofollow, I keep thinking it's mine because I edited the story links so that after the first few minutes after they're posted, they don't send a refer so that marketers can't get good analytics out of our traffic. PudottaPommin I have question
that's why i keep thinking oh you did an anti-spam thing with links yeah but it's different in those two ways cool thomas see you later and let me know if you need anything else with the notifications i will keep trying to find a couple of minutes each day to peek in at your pr so if you have any questions that you're not blocked for terribly long padota throw it in it's gonna take me a second to write this comment

01:21:56PudottaPommin How do you handle cron jobs - within app or its linux cron jobs
What was that? Comment ID? Comment ID was...

01:22:30PudottaPommin I seem someone dissing cron jobs inside app in favour of using linux cron tab
pushcx https://github.com/lobsters/lob…
we actually have been moving all cron jobs over to solidq i would dis linux crontab and i say that having used it for almost 30 years and it's not that it is bad it's just been surpassed by alternatives and in this case PudottaPommin I do prefer if possible having those jobs within app
The things I like about Solid Q are we want to have background jobs anyways. Having the background job system also handle the scheduling and recurring jobs means we have one fewer moving part. We don't have to install cron. We don't have to wrestle cron syntax, which is also a product of its time. because it's all happening in the app job context it's correctly wired into our bug tracker because cron by default you're supposed to make your cron jobs produce nothing to standard out nothing to standard error and if you do it gets emailed to you which is not how i want to handle that kind of bug reporting and also That precludes logging, like then you have to shift to disk-based logging, which means you also have to find that log. And this is beyond lobster scale, but it means you don't want to have ephemeral servers anymore because now there's valuable information on that VM. And then also, cron is not as expressive as what we can do with recurring.yml in Rails. So we can express things like the second weekday of the month if we wanted to. That's not a thing that we care about, but it might be for some business cases. Anyways, I have a lot of I've gotten a hell of a lot of use out of cron over the decades and I still have I don't know 20 cron jobs running on my personal machine under my user but. For Apps I would prefer not to use it at all period oh yeah the other hassle is. PudottaPommin If possible, Im never again touching cron tab
unless you are real deliberate about how you wire things up that crontab file isn't going to be in your repo and now you have code changes that are outside of your regular version control and deployment process that's no bueno yeah

01:25:35So let's grab... I need to grab the... I'm doing this off stream just so I don't accidentally pick up an extra thing.

01:26:12And then there's a change to clutter oppression.

01:27:12I think a lot of people have had that experience with cron. So this one's not the code block. Was it the second one? Second one, yeah. So let's grab that short ID. And then I'll grab that source.

01:28:30PudottaPommin One more - how is short_id generated, custom algo? Or just random string
short ideas, a custom algorithm that is a random string, and then it has some logic around retrying and inserting. Where is that generated? Because there's like a helper for it, right? Is it in extras or is it in the monkey patch? It's not in DB. Not in routes.

01:29:13Worst example.

...44Huh. I don't remember offhand where that code is.

...52PudottaPommin Ephereal code Kappa
It's some kind of like monkey patch into string or something that generates a random thing and then has to pushcx https://github.com/lobsters/lob…
search for it because of course it has to handle collisions so you might be interested in me starting to you know this says replace it is augment add on we are starting to see collisions in short id and this is like a realistically, we have a number of years before this becomes a significant problem like a decade. But it would be very easy to forget about this for a decade, and I would prefer not to. PudottaPommin Influx of comments will shorten it
And at some point, the birthday paradox means it tips into being a performance problem if it has to try and retry a number of times before it can generate a valid new short ID. pushcx https://lobste.rs/stats
So Yeah. And if you've seen slash stats, our comment rate is slowly increasing. We get more. You know, it's not a monotonic thing. We are not a startup, so we're not prioritizing growth. I would actually be alarmed if the growth rate was too steep. because we have to acculturate people and handle things. But yeah. All right.

01:33:33PudottaPommin Moderate people archiv32Uhoh
It's not showing.

...47I just have to.

...55All right. All right. Are you suggesting I pull a Thanos and just delete half of new comments? PudottaPommin One possibility
God, that would... There are not a lot of things that can kill a years-old community that is as big and healthy as lobsters. That would be up in the neighborhood of the self-destructive kind of things one could do to a community. Woof. So let's not do that. Let's instead maintain things and fix bugs and generally be chill. So I am going to have an early lunch and then code recheck so I can give you all a demo on Monday during the next stream. PudottaPommin Have a good rest of day and lunch archiv32Glowstick archiv32Shy
And then think about Lobster's birthday wishes for our 13th birthday, which will be on next Thursday. And I will be streaming. So obviously that one can't be a short stream. We've got to have a party, right? I'll put a paper party hat on the cat. God, he'd be so mad. PudottaPommin Cat cam
He'd bite me. We'll see if I can do it. No, actually I can't. I don't have it in me to be into the cat. He wouldn't understand. PudottaPommin bamo16PEDRO
yeah when he so pedota another reason to look at the archives is he does show up on cat cam occasionally i have a webcam and when he sits in a particular place up on a filing cabinet we can show him he was just on two streams ago i think so yeah maybe three four sometime pretty recent so anyways all right well thanks for tuning in folks and of course thanks for contributing to the community and hope to see you all on monday and around the site in general take care