I don't know what I need a book to crash rails

01:39Hello, it's Thursday. This is Lobster's Office Hours. This is Lobster's. Oh, boy. So, I'm a little fretful today, so I will tell you, like, first sentence out of my mouth, that this is going to be a shorter stream than the usual three-hour chunk. We'll see if I make it about 90 minutes. So, the reason I'm a little... twitchtd hi pushcx :) here early today
was a big server move over this weekend, and as I just put it on a Rails discussion chat, errors that only occur in prod make my hair fall out, where I grab it in chunks and yank. So, so, so, so. twitchtd your audio is going in and out
On Saturday, Hunter and I spent a good five hours or so working on the server move. And then I came back in the evening and I spent a little more time and more on Sunday and a bunch more this morning. But I'm not looking at the issues first. The gist of it is for folks who, oh no. Okay, there we go. twitchtd ya sounds better now
I should probably sound fine now. Thank you, TD. twitchtd tom
You're Ted, right? I know it's TD, but I don't remember. Yeah, it's the volume workaround that I still haven't put more time. Excuse me. Oh, right, yes. And frazzled and starting late enough that I forgot to open the workaround. So yeah, put a ton of time this weekend into the server move. You may notice the, we are on the master branch and the hatch box directory is merged down. twitchtd oh nice, is lobsters now running on hatchbox?
So Hunter and I have replaced our, twitchtd damn, I didn't even notice that
ansible script that i was poorly maintaining for years with hatchbox which is a nice deployment script yeah so it has we are now running on a server on digital ocean still that was provisioned and deployed by hatchbox and it's pretty nice yeah so we were in read-only mode for maybe a half an hour to an hour on Saturday morning, which is when traffic is pretty darn low. Sunday mornings are a little lower. But I wanted to have Sunday in case anything serious came up, so we did it on Saturday. Otherwise, we tend to do server maintenance-y stuff on Sunday when it's quiet and, you know, don't have jobs distracting us or anything. So watch this. twitchtd how are you not taking downtime with 1 machine?
lobsters deploy i have a cheesy script so look that kicked off a deploy and the production site is live and we are not taking any down time look at that see this i can reload the site over and over and oh look now we're into the meat of the thing and it takes a second or two to reload but hatchbox does a whole clever thing yeah so that's hatchbox being clever this is why it is worth paying somebody 10 bucks a month there it is now it's now it's into the meat of the start so the puma worker is currently restarting in the background and hatchbox does a thing called socket activation which is a feature i hadn't been familiar with but basically It has systemd hold open and proxy the connections between caddy, which is the reverse proxy out in front, and then puma, which is the Rails app server. And systemd hangs onto those sockets and just sort of is like, hold on, hold on. I'll come back, I'll be right back. I promise I'll be right back. And That means hopefully nobody sees 502s or 504s while we are in the middle of deploying. Epic_Ninja_Elephant Oh hai.
We've been doing this for a couple of days now. twitchtd B)
I have deployed dozens of times. Sometimes I'll just pull up the dashboard because they have a cute little web interface and I'll just hit the deploy button for funsies just because it doesn't cost any downtime. Hey, Ninja Elephant. Nice to see you again. And I had to put in this timer. Deploy took 39 seconds. So this is, if you have watched the stream before, you have seen me run just the Lobster's role of the Ansible playbook. And that alone used to take two and a half minutes. And then after that two and a half minutes, it would take like 90 seconds of downtime. Epic_Ninja_Elephant i used to be able to warm a cup of coffee in the microwave while you deployed.
Epic_Ninja_Elephant What do I do now, huh? Have you thought about that?
So this day-to-day experience of the one core thing of I push the button to get a deploy, having that just work and just zip along. Yeah, I'm so sorry. It's like when your tests run too fast. Epic_Ninja_Elephant YOu RUINED my workflow.
It's rough as a developer when your background jobs and everything run too fast. Speaking of which, we've been making really good progress on that. twitchtd do you have access to the underlying filesystem on the machine in hatchbox? wondering if sqlite is still a future possibility
I don't think I have another pull request from Viv Shaw to merge, but she's done a bunch of that. Yeah, she hasn't opened another, but she's been doing these jobs to merge more and more of our cron jobs over. yes we do have access to the underlying file system and the thing i like about this setup is hatchbox is basically a script that someone is maintaining to do the deployment we own the vps we have root on the vps is under my account on digital ocean It's our, I mean, I almost want to say hardware, but obviously it's a virtual server. And so if I show off this code, which, you know, when I open it in Vim, I've been linking people to the, oh, you are the, this should be in the, let's get everything sorted out. All right. So

09:07So I talked to Hatchbox support and he was like, no, you can't really run scripts as root when you're deploying. What's that Michael Jordan meme? And I took that as an insult. And I was insulted by that. So I have rigged up, because I have no principles or morals, I have rigged up this system so that when we deploy, we can run this script as root. And this is kind of the things we want to do as Root to finish out, to round out the provisioning. Because out of the box, Hatchbox, it's sort of, I think Hatchbox's ideal customer is a small business. Because a lot of stuff only makes sense where you assume they have someone technical to debug things and then you assume that they want everything to work by paying for third-party services so like there isn't a checkbox for install postfix on this server so okay we can call apt install postfix here and Then this script does the configuration of OpenDKM as a dependency of our postfix setup. pushcx https://github.com/lobsters/lob…
And then it does the postfix setup and then wires the two together with a couple of config files. If you're curious about it, you can browse here. I'll share the link. kazaii626 yoyo programmerz.
You can browse this directory because all of the config is in here. I have actually... I have actually... twitchtd I personally like shell scripts as config, does the shell script have to be idempotent or does it use some kind of fresh image each time it deploys?
I can't read the tiny type. I've been staring at the screen for all weekend. This repository was archived. It is now read only because all of our provisioning deployment stuff is right here. No, this is one server that is being maintained. We did actually... like start the deployment and then restart with a new VPS. But the idea is you just have one VPS and you maintain it rather than like you throw it away and you roll a new box. So like if I, oh yeah, you asked, do I have prod? Root at level four. Hey, look, here's root. I like the cute little message. I've left it in. and this is probably my bug i'll fix that later and then you can deploy at web04 and then like here's our deploy user if we look in the lobster's directory like so repo is the git repo and then it takes checkouts to the releases directory every time i push I don't remember if this is every time I push deploy or this is every new checkout for which I push deploy. twitchtd capistrano style
And then current is just a symlink into the current release. And shared is the stuff that's shared across releases. So that's things like storage. That's our queue. Yes, very similar to Capistrano. And Etsy is the directory our script adds. That has our, I can show you the file names. That's not. But that has our couple of extra production credentials that I didn't want to move into the Hatchbox environment variable system. So this etc directory is on my local machine as a private Git repo because you should version key config files. That's just how that should be. Oh yeah, that's the other reason. When I look at Hatchbox, I think it's for aimed at small business because it's not just that it sort of assumes you don't want to script anything yourself. It also, can I show that? Hold on, let me see if I can. So it has a web UI for stuff like, yeah, I could show this page. So light mode warning here. So it has like a page for editing cron jobs and it has like a little web interface and this is cute and all, but if I edit this cron job, what exactly was the schedule or the invocation I had last week? And the answer is, you don't know. There's no history on this thing. And here, I got to pull this off screen again because Epic_Ninja_Elephant crontabs are not versioned?
The UI, the web UI is nice, but it is a little cavalier about putting up API keys. So there's this settings page where you can turn on a maintenance mode, which is nice. Right. Correct. It's not versioned. And so like you can add a pre-build script. So this is after it does the checkout before it starts running its build, you can run a custom script and it's sort of, I think There's no placeholder. I thought one of these had a placeholder. Yeah. So like the little placeholder here is like kind of nudging you to write your script right here. But if I write my script here, it's not versioned, right? And so for these, it was really easy to write a script and just say like, okay, the script is going to live in the repo. And the only thing that this thing does is run the script out of the repo. Great. The caddy file, like they sort of, I don't think they really expect you to customize that much at all. I customize that a lot because our full page caching is very important to our standing up against traffic. And,

15:51This chunk of it is as small as I can make it. And I copied and pasted it into our repo. But there's no direct code link between this file in the repo and this form field. It's just literally, you know, the manual processes copy and paste. Because it includes these pre and post, like these live here, and you can look at the commit history of me sorting out various caching and rewriting and cookie things. This is great, but this is not the out of the box experience of Hatchbox. Hatchbox really doesn't operate with that level of rigor. I'm trying not to insult it because it really is useful and it's totally worth 10 bucks a month, but like not being able to version stuff is sort of a recipe for pain. I don't know. And I understand that if this, if all of these fields get versioned all of a sudden, like I imagine like, okay, well, what is it? Where are they stored? Are they stored in a JSON file that sits in a Git repo? And then at that point, you are reinventing Ansible with JSON instead of YAML, right? twitchtd maybe they're versioned on their backend which support could access but not the user
So I understand why the service doesn't want to head down that road too far, but some of the answers to questions I got were they really assume that you're not an engineer, because I don't know, like, I learned some CVS in the mid 90s. And then I learned subversion later. And like, right around that time, I learned subversion, I got in the habit of even for toy projects, you make a repo, even for toy projects, and, and tinkering, you make You initialize the version control system and you checkpoint occasionally. And this became a lot easier with Git. And it's been so valuable to me for so like two and a half decades that it's very odd to me that Hatchbox doesn't try to lean into that with here, put some files in your repo and we'll pick them up. Epic_Ninja_Elephant I feel terrible when people want me to work outside of version control.
I don't know. It is possible their version on their backend. Yes. I have no idea. Like this could be an event source system. This could be a database. This could be stored to a Git repo. I have no idea. twitchtd src, src_copy1, src2, src_backup0
That kind of is in the, I don't have to think about that level of detail. And it's nice. I am very happy to not have to think about The Ansible repo. Yeah, you know, I have a consulting war story of a Fortune 50 company that I worked for. And I'm not going to tell you their name because I'm going to tell you the funny thing instead. Not funny. Ha ha. But they are a Fortune 50 company. You probably have one of their products in your home. Epic_Ninja_Elephant backup_final_final_v2
You almost certainly have used one of their products in the last year. And at least if you're American, actually, no, they are international anyways. So they spent something like, I was called into work on this internal system that was used by accountants to plan the R and D budget. And they called us in because it was slow and we were like, okay, what does slow mean? Does it take like five or 10 seconds to load a page? And the answer was it takes eight to 15 minutes to load a page. twitchtd @Epic_Ninja_Elephant I see you have experience with unversioned code :)
And that tells you two things. Like number one, something is very wrong on the technical side, but number two, there is enormous business value if people are willing to start loading a page, go make a cup of coffee, come back 15 minutes later, and then use the page, you know? That kind of big putting up with pain tells you the system has a lot of value to it. And it had zero tests. Epic_Ninja_Elephant @twitchtd I used to work in a CMMI5 environment. Before that, I experienced a lot of pain.
and I asked how often they deployed and the boss said, oh, we deploy four, sometimes even five times a year, a year. bsandro VoHiYo ehlo
And of course deployment was, he would manually SCP individual source files into place and then try to remember, cause he didn't know cell history, but he had a written down, this is how you restart the Django process, note card. And they did not use version control. Like there was a version control system, but they didn't actually commit to it. They only worked by emailing files to each other. Hey, be Sandra. I'm telling a horror story. And then, the worst part of it was, so here what's her move first and then, consulting or story second. Cause that's the topic I'm rambling about now. So they were emailing files back and forth and then manually copying them to the server. And the system was used to plan. I was never given access to the production database because it was too sensitive, but I could look in their 10K and it was used to plan something like $5 billion a year in spending. That is billion with a B. $5,000 million a year in spending was planned with this system that was two junior engineers from two different consultancies managed by a project manager from a third consultancy. That PM had five projects. IT hated this system because it had been, as you can tell, snuck in the side door by the fact that it wasn't Java, you can tell. And... It was invaluable, so they couldn't get rid of it, right? Planned billions of dollars of a year. Where'd my scratch go? And I looked at that and I turned to my boss after the first meeting where we saw the full scope of the problem. And I said, you know, if we keep working with this system, we are going to find a bug that has misallocated more money than you and I will ever earn in our lifetimes. I guarantee it. That's the scale this system is operating at. And he kind of like, I mean, we're both pasty nerds. We kind of turned a whiter shade of pale. And then he went and checked on his errors and emissions insurance. Then we fixed a lot of things for them. So I look at, you know, had to commit a bit of a sin to hook deploys so what it does is when hatchbox deploys it kicks the puma workers and they all start and so i configured them so that when they come up they touch that symlink because there's a bug in how systemd looks at symlinks so it doesn't have to be the the current one it could be its own file but I was down the rabbit hole on this one. So when I deploy, all that stuff runs as the deploy user and this script fires because of chicanery. And then it does an idempotent install or configuration of the last handful of things. And I don't love having to tie this on the side and then just know how to make it item potent. You really don't want to roll your DKIM key every deploy. But at least it's pretty small and straightforward. And it's like, take this file out of the repo and put it into place on disk. It's not so complicated. I feel pretty good about this. Grab a backup with slash etc, because why would you not want one? It works. We're working.

25:32I didn't turn on the footer here. Where's my little tool?

...45How about now? OK, cool. My integration has broken.

...56Oh, it's broken because we have the new server. That's why. And the file lives in a different directory. So there's the little banner that I'm streaming Office Hours that hasn't appeared. That's because I broke that script because we have a new layout. God, I don't even remember where this script is. Everything is tied together.

26:40Oh my God. So you can't see it, but I'm tweaking the OBS setup here off stream. The new server, new thing lives in deploy lobsters current TMP office hours. Okay. And

27:11I guess I'll have to remember that when I deploy, I have to switch the scene to itself to make sure the banner reappears. That's kind of fine. All right, there we go.

...46Fun with plumbing. So let's see where we're at.

...57twitchtd question for you, have you tried vibe coding or are you actively avoiding it? just curious, nothing deeper than that
So that was my big project for the weekend. Have I tried vibe coding? No, I have actually. And I have... pushcx https://push.cx/stream/2024-10-…
even here on stream. So if we go back to... Yeah, this one. So I gave a little... Oh, man, it's hard to believe this is eight months old now, almost. But I gave a demo of this tool, Ader, that I've played with, and I've continued using it. That little... Actually, you saw me run that lobsters deploy script here in the terminal. That was written by Ader, because it's... Not something that's going in a repo anywhere, not anything I care about. It's a wild loop around. It hits one endpoint to get JSON. And then it here, I think I can just, well, let me open that off screen. Cause it's gonna have a key in it. Lobsters dash deploy. There is a, What do you call it? A deploy key in it. All right. So if I change that to just say key. All right. So here's this script. This is pretty much straight vibe coded. I did this in about two minutes this morning with Ader. I told it, write me a Ruby script that hits this URL. And I gave it the script with the actual key in here. And then it just kind of vibe coded this. I actually, this even meets Simon Willison's definition of vibe coding because I didn't read this code. So we were reading it together for the first time. So yeah. it fires up the ruby http library it hit starts the deploy and then it watches the you get back in i don't know where i put it up again there i guess so it could rescue it yeah so it comes back with a json that has an id in it and you take that id and You get the key a second time, so now I get to roll a key. Man. Hard streaming. Please no one kick off a deploy in the two seconds while I roll this. Where is this thing? It's here, and rotate key. Cool. Key is rotated. So... twitchtd what's aider? is that some kind of a local LLM or is that a proprietary llm from openai/ms/etc?
Then you just pull this other URL with the ID that you got from the first call. And so I put it in a little, I told it, make me a loop that prints a dot every second. pushcx https://aider.chat/
Yeah, so Ader is, I guess we would call it an agent now, but it's, come here. It's like a chat bot where it works on a Git repo and you just tell it what you want and it can talk to, I don't know, nine kinds of LLMs, whether that's a local one or a paid service with an API like, I think I used Claude Sonnet 3.7. I don't need an animation. But it basically looks at your code base, kicks limited parts of it up to keep the context window from blowing out, gets back code. It's got some built-in prompts that are pretty smart about giving instructions for reasonable code. And so I think I told it. I think my prompts, I think I had three. I had one that said, hit this endpoint to start a deploy and parse the JSON you get back and print me the ID. And it did that. And then I told it, okay, take the ID and hit this other URL. And it did that. And then I said, now put that in a loop and pull it until the, I think it's the state field. Yeah, it's called state. It says processing for as long as the deploy is running. And I told it, print me a dot and sleep. And then whenever it changes from processing, print me out the final JSON, because I wasn't trying to make anything pretty. And if it says completed, show me the elapsed time. twitchtd damn that's impressive
And honestly, I told it this last thing for elapsed time just because I'm so elated by not spending 15 minutes to make a deploy. So this was like three prompts and something like 90 seconds to two minutes. And I am aware that this could be a one-liner in the shell where I call curl, and then I call awk, and then I pipe that to a bash, like I could write that, I know how to write that. And this is not the tidiest code, but I'm gonna run this for, I don't know, probably five years before I care to open this thing up and look at it ever again, right? Like this is the kind of, so I talk about it more in that, In the stream archive of what do I see these as useful for and it's not really prototypes it's this kind of glue code one offs and I just mentioned willison right. So he had a blog post he just published. Here, in the last couple of days that was called like 105 tools I made where to go now.

34:27Hold on. I know I saw it on Monsters. I saw it off of Blue Sky first, but... Show me tools. pushcx https://lobste.rs/s/h72rfa/105_…
Here we go. 105. Oh, it's on tools.simonwillison.net. Okay, so this one... This is exactly in that vein of none of these are programs that would take an experienced developer. Well, I say none, but I only read like a third of this page because there's 105 of these. But these are all these kinds of like, yeah, you could hack this out in an afternoon kind of script. Or you could get it out of a coding tool like Ader or Cloud Code or Copilot or one of these other ones. You can get it out of these scripts in a couple of minutes. You probably... There's a benefit. The coding tools, the ones that are explicitly coding tools, they have clever little prompts And they produce better code than if you just open up ChatGPT or Claude and you say, write me a script to blah. You will actually get better code out of these specific programming tools like AIDR because they have specialized prompts. And if you prompt these things better, you get better answers.

36:14it's interesting. So like this is two prompts. Your browser supports pass keys. Okay, great. Let's let's create one. I'm push CX. No, did it not work? Oh, script has a bug. What a shame.

...39The platform in the current context, possibly because the user denied permission. I wonder if I have some security setting cranked up here.

...52OK, we got a timer. Great. Yeah. So like you look at these things, I can write this code in my head. You know, I know what that was a a dialog element. This is just centered and then we do a countdown. There's a set timeout that fires every thousand milliseconds, you know? But I know if I started writing this by hand, I would run into some fiddly little like, how do I horizontally or vertically center a div? Why is this weird and hard? You know, like that kind of nonsense. Or like if I reset twice, the state gets funky because I attached it to a DOM element and I have to do it a second time, right? Like, you know how this happens when you write small scripts. Even if this is 50 lines, you run into one of these little annoying things. And when your entire development interaction, this is interesting that he's put like a prompt or two in. So you can just kind of see yeah some of these it looks like he's got the prompts yeah like this is this is not bad at all these kinds of things so my big asterisk on vibe coding is we have no idea what the copyright status of the resulting scripts is going to be There's a bunch of pending lawsuits. United States copyright law is very complicated, let alone international copyright law. I can't guess at that. I don't know where a court is going to land. If I had to throw this script away, I don't care because it took me two minutes and like 15 cents in tokens. That's fine. If it turns out that the output of all of these LLM coding tools is legally speaking toxic waste, I can throw this away. So I don't run any of these tools on a code base I care about. I'm aware of some people do. And I know that saying, I'm waiting to hear how the copyright things shake out means I'm waiting realistically five or 10 years, right? That's how long that takes. I don't have a better answer there. That's just not a risk I want to take with things I care about, like the lobsters code base.

39:44I don't know. twitchtd ya, thanks
I hope that answers your question about vibe coding. Yeah. Let me create that. I think over here, because I had, you know, I try and make these nice little stream notes. I do think it is worth pretty much every developer's time to spend a few minutes playing around with one of the agenty tools, agentic tools. I don't know if they count as agents, but that seems to be what we're calling them. Where... Rather than just being the chat box, it is a tool that is aimed at programmers and I don't, I have really only spent time with AIDR. I don't have an opinion on which one of these is the best or which LLM is the best or other stuff. twitchtd I've been playing with cursor personally, so far mixed results (it uses claude 4)
Like I couldn't tell you more than that, but if you want to try AIDR, it takes two seconds to install with UV and It's worth playing around with. You know, people talk about model collapse. Okay, cool. People talk about model collapse. You know, if they never train a better model than Claude 3.7 or Claude 4, it's useful. It's useful right now. It can knock out scripts in a minute or two. I can't do this kind of thing because, you know, it's going to pull in weird context and it's going to assume all kinds of You know, whatever is, it's going to assume I'm running on an Ubuntu desktop because that's what most answers about Linux are about. You'll get nonsense if you try and write something like this. It won't. And it would never, never in a million years be able to figure out this kind of like, let me wire together four features that aren't designed for it. But hey, the one-off script to pull an endpoint Got that pretty easy. Speaking of things we probably couldn't get out of LLM tools, although actually security research is kind of iffy, like in the middle. pushcx https://lobste.rs/s/82zdak/low_…
I've heard people say they're searching for stuff. I don't know if Chambliss is here. She hasn't said hi today. But... pushcx https://www.naff.dev/blog/lobst…
Chamliss, who is a regular stream chatter and repeated contributor to the codebase, maybe call it two weeks ago, I'd have to look in my IRC logs, found a bug in how we authenticate keybase accounts. And she made a great little write up here. Let's get that in the scratch too. There we go. But the gist of this is part of the URL that gets constructed is under user control. So if Alice has authenticated her Keybase account to lobsters, Bob could copy that and Bob's profile would show that he has demonstrated he controls Alice's Keybase account, which is a particularly, like it's not huge severity in part because people didn't use this much and Keybase was kind of fading out. It's been getting minimal maintenance for five years. But also it only allows you to copy something. kazaii626 it was a good catch & a good read
But it is bad that A thing that is supposed to be demonstrating you own an account was vulnerable to having that attestation copied yeah yeah actually it was a great write up, I thought this was really clear sometimes I struggle to read. security write ups, I am not a security guy but. She did a good job explaining that heck I even understood it when she explained it to me on irc. So I wrote up a comment here. just endorsing what she wrote, because honestly, her blog post was very complete. There's not much more to say besides I tried to be a responsible administrator by making sure that it hadn't been exploited. As far as I can tell from the logs, it hadn't been. I talked to other users to be like, hey, does anybody actually care about Keybase anymore? I said it a lot nicer than that. That was kind of the gist of the question. I think I showed this on the last stream where I pulled Keybase out, but their last blog post was the day they got acquired. That's almost the date of their last commit. You could see it on the chart here when they got acquired, right?

45:07I look at this this chart and the fact that there are these like there's a couple of there's this steady trickle of commits and what this tells me is. Someone got it written into their contract that they get so many hours a week to maintain it and. I would this is, I say this, knowing nothing at all nothing internally I don't know any of these individuals I don't know even the username of who this is writing these commits. But I would bet that Zoom gives them exactly that contracted hourly time or number of commits or base level of functionality and not one second more. So the service is still on. You know, the lights are on, but nobody's home because there's very clearly just one person maintaining this, which is a bit of a shame. It was solving an interesting problem and people had some hopes for it. pushcx https://github.com/lobsters/lob…
If anyone would like there is a feature request open for supporting key oxide, which seems to be the Community chosen successor. I would be happy to take a PR for that nobody has put their hand up to say they want to work on it, although this person here Victor K i'm guessing it being an Eastern European pronunciation on this name. Oh yeah, find Polish name.

46:44Not Chicagoan enough to pronounce the Polish name properly on the first try. twitchtd kvapeeshyevich
In any case, perhaps he will implement that. Or perhaps he will, dear reader. So yeah, that's what's been going on the last couple of days in the code base.

47:12That is not how I hear the WICZ pronounced in Chicago, but it is possible that we have a local, I mean, it's certain that we have a local accent. I don't know. I was just at a wonderful little, not quite a beer garden, but a patio. And I guess it was a Polish neighborhood haunt, even though I sure wasn't in a Polish neighborhood. But the table next to us was two little old ladies and an older man, and they were speaking Polish for the first hour. We were sitting there drinking some beers and having a burger. And then the next table that sat down was four young people, also mostly chatting in Polish, but switching to English sometime. And then there was another table that I'm pretty sure was chatting in Polish. So I don't know. It's nice to hear in the city. I've forgotten almost all of my Polish. I never had much to begin with. All right. Yes, so I looked at this one. This one is waiting for Shubham to make some revisions. And then I think this one, I peeked at it, and it's probably ready to merge. because Nav Mike here has made a bunch of contributions to this refactoring. I really appreciated these. pushcx https://github.com/lobsters/lob…
So story repo repository div. Oh, and I will share that link over here. Oh, you know, I haven't even given my standard spiel, but Since this is Lobster's office hours, you can pipe up with a question about the site or the code base anytime, as you can see that, you know, Tom asking a question got us onto vibe coding for a few minutes. We're here to just chat about whatever, because some people find it a lot more approachable than emailing or IRC or messaging on the site, which are all valid ways to reach me. And then otherwise, You know, it is funny. I've said that one of the reasons I do the streams is to kind of time box my time, but then also I'm a little tired because I spent a ton of time on all this server setup stuff in the last couple of days. That's okay. All right. So let's look at this. This looks pretty straightforward. Yeah. So the story repository, for anybody who hasn't poked around or seen one of the previous streams, is it predates Rails having these very nice composable scopes. I mean, maybe they're roughly contemporaneous with the introduction of it, but scopes were a little bit rough that first year or so. I mean, that's all code, right? Okay, filter, and then... Maybe one of the nicest things about nav mics contributions is he really nailed the style of our tests and it's doing kind of the exact right amount of tests.

50:43Usually we do the. stub login as.

...56prime_coder Hey @pushcx i lost the link for the ruby on rails book u recomended.Can u like it maby agian
pushcx https://pragprog.com/titles/rai…
prime_coder ye that one thanks
wonder why is there a difference there oh hey prime coder i for rails that was probably there we go this was the rails book and hey if you didn't see it the the

51:29pushcx https://pragprog.com/titles/rub…
prime_coder oh nice
The Pickaxe book got updated. And my usual go-to suggestion for Ruby is the well-grounded Rubyist, but it's been just a little too long since it's gotten a new version. And this one is up to date. prime_coder shall i buy them both
So I have liked Noel Rappin's other books. I haven't yet read this one. I picked up my own copy since I mentioned it on stream a week or so ago, but I haven't yet gone through it. If you don't know Ruby, I am sure it's going to be a solid book because Noel's written a bunch of good stuff before. I guess I should make this a fact at some point, right?

52:12All right, so spec helper. Where's mine?

...28twitchtd oh another question about the hatchbox migration, did you also migrate the mariadb server to hatchbox or is that somewhere else?
spec support okay so stub login as is actually doing that which hmm yeah the cookie thing is probably easier and faster if we could just get the Probably need a different one for... That's why he did it this way is because it's a controller spec rather than a feature spec. Sure. Okay.

53:30Great. This is great.

54:26So I was going to say, do you want to make another one for controller specs? But they don't maintain a request like cookie jar along the way. So there isn't really a helper to be written.

...59No, the MariaDB, so it was interesting that you mentioned SQLite. MariaDB is still sitting on its own VPS, and that had been managed by Ansible. It has the unattended upgrades, Ubuntu package installed and configured to install security updates and bounce the box. pushcx https://github.com/lobsters/lob…
We have a... an open issue here where I said, Hey, maybe we should start thinking about getting off of MariaDB because like, so I made this issue. Oh, wow. Almost seven years ago. And then it came back up in the last year or two. Where are we here? Yeah, because someone dropped in with more good info, and then there was news that a private equity firm had acquired MariaDB, and that doesn't bode well for its future. So it was, you know, it didn't... turn into a screaming emergency but it is a like okay we should start looking for the exit and slowly start walking towards the exit because realistically even if they said no more open source MariaDB it's now ten thousand dollars a year is our lowest tier and you have to you know or it's call us pricing and we'll negotiate with you and see how big the barrel is that we're shaking you over. Even then, we would still have six months to a year to, like, mosey our way off before something important broke.

57:10And they haven't done that yet, to be clear. But, you know, K1 is private equity, that's... It would be unsurprising if they did something like that. kazaii626 The latest humble book bundle includes Ruby Crash Course & Ruby by Example. More recruits on the way, at the speed of reading two textbooks :D
we talked a bunch in this bug about do we want to move to postgres and rahul gupta made some experiments with it the code base is used by the widget team oh that's interesting And the YJIT team using the code base, actually their fork of the code base has ported it to SQLite just to have smaller dependencies. And we've talked about SQLite a bunch. And so you, TD, asked about, could we run SQLite? And the answer is, yeah. Yeah, we probably could. I've been trying to find production numbers for pushcx https://www.humblebundle.com/bo…
kazaii626 yep, second one. Kaz-aye
kazaii626 no big deal
sequel light i have not found enough people running it in production that have shared numbers to be super there we go so there's the bundle that kazale is talking about kazai that is two eyes isn't it kazai excuse me sorry i keep calling you the wrong thing

58:55So I don't know. I haven't heard of either of these books to have an opinion on them. I don't know that I need a book to crash Rails. I can crash it all the time on my own.

59:19kazaii626 The Python Crash Course book by Eric M is highly regarded. UNsure about the Ruby ones.
So maybe TD, maybe here, once I... get some time for the Hatchbox migration to settle down. prime_coder @kazaii626 <3 <3
And there's still one more thing I'm chasing around with Action Mailbox not being quite wired up correctly. But once that's done and I see everything be chill for a couple of weeks, I'll start thinking about, do we want to move over to Postgres or SQLite? I don't know.

...59Oh, yeah. You know what I just did? So I just merged to PR. Actually, merged to PR for the first time since moving over to Hatchbox. So let's see. How do I? kazaii626 :D
I don't remember how to do the pull rebase in Jujutsu, which I've started using. Yeah. So jj git batch, I believe. And then, yeah, jj edit master origin. That's fine. Did you make me a new one? Or where did you leave me? So it's complaining like, hey, don't try and edit a commit. Yeah, all right. So it wants me to say, give me a new commit on top of master. Yeah, there we go. How did I not get that pager sorted? Yeah, so now we're on top of master. Great. Let's run my deploy script again, right? So I hit that button off screen to roll the token that I accidentally flashed up when I was showing the script. So let me get the new token into that script. So let me edit bin lobsters deploy. You know, I could probably tell it to extract that key to a constant. But instead, I'm just going to do it myself.

01:01:45It's a little funny working with the tools because. Kind of by default. You end up with like junior code. And it took me a second, but it makes sense for the way that LLMs get trained, right? They get trained and they're sort of representing the median code available on the web. And there's so much more tutorial code than there is senior developer code online. So yeah, I showed you that Lobster's deploy script that I vibe coded. All I did was pull that key out into a variable, string interpolate it back, copy out of the thing. I could probably have made Ader do it even faster, but then I can't really show that on screen, I guess. Because again, you'd see the key and I'd have to hit the roll button again. 29 seconds to deploy. Isn't that beautiful? 29 seconds. We're going to have a couple of weeks where every time I deploy, I marvel at it and I'm smug about it.

01:03:15Yeah, look at that. The site is running so smoothly. Oh, I can show you that. Oh, that was the thing I was going to show about the move. Yeah, so here, where was I? All right. Light mode warning. On DigitalOcean, we have little performance graphs now. This is at 100% because there's a whole thing happening with the mail queue that got backed up and paused over the weekend. It wasn't wired up correctly, so two days of mail went out. But, you know, you can see like, oh, look, this is where Peter deployed because the RAM graph resets. Disk IO, you can see this is where I kicked the mail queue. I bet there's a bump in, yeah, there's a bump in outgoing bandwidth too. So anyways, we have these nice little charts now. Monday is our generally busiest time. I would bet this spike is Calvin's WWDC thread that's on the top of the homepage now. twitchtd I was asking about mariadb because I might be willing to try to migrate to sqlite/postgres (whichever you would want to try out) assuming the existing developers that tried to migrate in https://github.com/lobsters/lob… don't want to take on that work.
But anyways, kind of neat to have this stuff. How's the mail queue doing, actually?

01:04:45that's really generous of you to offer td huh so i'm just thinking off the top of my head but why don't you leave a comment in there and so i'm a little bit nervous about sqlite numbers because i really only got good numbers from one production site and they were comfortably in our neighborhood on smaller hardware so i think we will probably be fine on sqlite and i would really i want it to work you know that that's what it is is i want it to let us simplify our hosting and simplify our deployment strategy and just have the one database type, I worry a little about it.

01:06:03So I guess my hesitation towards leaping on your generous offer or trying to talk to Rahul or someone else who talked about SQLite in the thread is If we try SQLite and it can't stand up to production traffic, that's a pretty sizable amount of time that goes into a project like that to waste. And I'm loathe to risk to waste contributor time like that. But I think that's our best bet, probably. I don't know.

...49we're down to 2800 when i kicked this off we had i want to say like 21 or 22 000 males this has been running for a couple of hours maybe two three oh looks like it's at a pretty flat point otherwise it was going down as fast as i could run that command again all right

01:07:20twitchtd got it, I'll leave a comment on that issue and see what if any I could help out with
just realized that's what that source is i know why the cpu is at 100 because this this does that touch but everything that starts like a rails console or anything is going to boot rails even even if you don't start a web server you sort of boot rails What I want is for this to be in the post deploy script. Yeah.

01:08:01We only need to do that once on deploy. Yeah. So let's do that there and then. So.

...16Here we go.

...29Yeah, sorry, I kind of lost the plot there because I realized why I had seen the CPU graph up so high. I saw that a while ago and it was on my list of things to investigate. And then it just clicked with me as we were talking about stuff.

01:09:31So I'm going to run this and that. Good. And now let's deploy. Look at this. It's going to be done before I can sit up in my chair. I shouldn't sit up. I shouldn't slouch. Yeah, so I guess the end of that thought is, I worry about wasting somebody's time, but I think it would probably be pretty great if we could move over to SQLite. I think that's our best bet because we will probably be totally fine scaling on that for a while, and I've been nervous Nancy for nothing. Nervous Nelly? Now I can't remember the saying. Or the aphorism.

01:10:49All right, so that was the only pull request hanging out to merge, right? Everything else is waiting on. We had a little discussion of this. I haven't seen the last comment or two. We had a little discussion of this in the chat room and stuff.

01:11:14Is there another comment? Because I saw the one at the bottom.

...35All right, since this is security sensitive and talking out loud as I do on stream takes out a big chunk of my intelligence, I am going to have to come back and review that off stream when I can give it my undivided attention. And then Anybody broken in issues? Oh, there was this painful bug. I hate any bug that can lose a command.

01:12:20Somebody's talking on the mock turtle. Yeah, somebody volunteered in chat to do this, Wallaboo, and then I haven't seen them since. five minutes before I left this comment. So I kind of tapped Church on the shoulder. Yeah. And so he's picking that up. And Caius also has been picking it up, I guess. So hopefully, hopefully we get the chatbot ported over to Ruby and then everything can live in the repo. Actually, it's the silliest thing, but I really twitchtd lobsters mono repo dream
like having this system administration stuff right here, because coordinating pull requests and commits across the two repos has been a minor annoyance, even at our small scale.

01:13:22Yeah. Yeah. Working at Stripe really converted me to mono repos. Stripe. Somebody was talking about it recently, right? So I don't have to mentally review an NDA. grayhatter_ are you still at stripe? or is this an old lesson?
Who did I see talking about it? Is it on HN?

...54No, not 10 months ago. Well, maybe 10 months ago. I thought just a couple of weeks ago, but time flies.

01:14:08pushcx https://blog.nelhage.com/post/s…
Yeah, I think this is the post I was thinking of. No, no, no. This is an older lesson. I have been out of Stripe for... Oh, gosh. Just over two years now. Yeah. I want to say I left May of 23. I would have to, like, go back and look for my last pay stub or something, but I want to say May of 23 I left. My LinkedIn is probably not updated.

...44grayhatter_ still a recommendable place?
But anyways, I was there for four and a half years and, and Stripe kind of still a recommendable place. Hmm. Greyhatter. grayhatter_ I applied to a job there and I'm wondering if I should pay close attention
I have nothing disparaging to say about Stripe. You applied to a job there and you're wondering if you should pay close attention. My answer to that would be, I have nothing disparaging to say about Stripe.

01:16:05grayhatter_ I appreciate the feedback :)
The feedback? You can ask me more questions if you like.

...23Let's see. So I didn't make the transition here, but there was this bug about losing comments, and I thought I might immediately be able to spot where it is because I remember what this code is.

...50I was over here, other tab. So yeah, I thought this code right here in 27 was exactly the code that was going to avoid losing the comment. It must be in a... No, it can't be because this actually works. So when the story isn't valid,

01:17:27We were under new.

...59Here's the resubmit. Then we have a textarea tag for the comment. So this is in the... It might actually get tagged into the story object. Let's test it. I think I can repro. So let's get over to localhost. And then I need a link that was submitted more than 90 days ago so let's go just look at the top for the last couple of years. And we will grab this link. submit it. There we go. So if I inspect you, you are named. Just comment okay. So test comment one. And now I expect it to fail because I don't have a title or tags. And my comment text is there, so I have failed to reproduce the bug. I wonder if it's because I did fetch title first. Let's try again. Submit, paste that. Oh, and as soon as I unfocus, this fires. Hmm. So if I said preview, no. Test two, preview. Aha, I previewed and the comment was lost. That was the bug. All right, so let's go. Oh, preview doesn't know about resubmitting, does it?

01:20:03Yeah, okay, so there's the bug.

...43So let's extract this off to a method. And down here, QRS TU. These guys are out of order. Not surprising on the god object controller. We will stay out of order.

01:21:22Let's see if that did it. So the steps were form, URL, fill something in. No, only on three. Preview. Okay, and it's still there. And now, ASDF, and I submit. Ooh, and it disappeared again. Okay, so attempting to submit That gets me off of new and onto create.

01:22:04Did I get a, no, I didn't get a 301. I didn't get a redirect because I would have seen that in the get parameters.

...40There's that same logic.

...59Undelete. All right, so there's something here.

01:23:18What do we see here?

...32So it sees that I posted to create. There's post. There's action create. There's the title, the empty tags. I don't see the How did that post with an empty comment?

01:24:03And this one's probably the preview. No, that's the post to check URL dupe. Huh, the post to check URL dupe had the test three in it. And then what did it do? That rendered. How did you get into post URL dupe?

01:25:30This doesn't feel like it should be it. Test. Test for. Submit. OK, still present. If I comment that out. Whoa. All I did was click on it and it's wiped out the comment. What happened? What do we think happened there? Did we just get a? Some bit of Ajax in there. Test five. Submit. And then something flickered. I think checkurldoop dropped it.

01:26:24Yeah, so it's rerendering that part of the form. That was it. Okay. Because that's part of the form errors. All right. That's where that went.

...47Oh, it says it's Ajax. We don't really have a good way to test it because we don't have any front end tests because we only have a couple hundred lines of JavaScript. We are slowly losing, I say slowly, wrong tense. I think we have lost that bet that there is so little JavaScript, it's not worth adding a front end test suite. I think we're seeing that, lost that bet. What was the, 16, 15.

01:27:38Don't lose resubmit comments on preview. Let's change. I want to see it with it for two more seconds. Now that I think I've seen it, I just, I want to see it again. All right. So yeah, now I can click. Nobody goes away the moment I'm clicking in or out on any of this.

01:28:12then preview still there change six to seven it's not reverting this is kind of half-assed testing like i am sure a proper software qa person would think of some nasty set of circumstances i just wanted to see it one more time because I don't know, I feel like losing a comment is one of the worst sins, because that is, you took the time to write something custom, something special, something that contributes to our community. That's the beating heart of the community. And short of a security bug more serious than the key base imitation thing, that's really, I think, about the worst bug we can have.

01:29:11Oh, and you know what? We're right about an hour and a half. I think we're going to deploy, and in like 30 seconds, we're going to be done. So get in your last questions, because I am going to just make this like a halfsies stream. We'll see how I'm feeling on Thursday morning, which is the next scheduled stream. That's 9 a.m. Chicago time. Probably that will be the regular length. I don't know. We'll see. I have a bunch of non-lobster stuff I'm hacking on. including breaking our bug tracker. Cool. Deployment took 30 seconds. Oh, man. One second more than our other deploys. What a slowdown. It's worth it. Castle has been worth it. twitchtd I posted a comment about sqlite at https://github.com/lobsters/lob… but don't feel like you need to answer it on stream.
Alrighty. So this has been a truncated Lobster's Office Hours stream. Well, I'm going to peek because it's in my head. Can I just yoink?

01:30:24Okay.

...34Yeah, I don't know that I have anything to add on stream, so we'll see. Oh, you know, one thing is I believe SQLite, and I know Postgres, have a better array type than is available in MariaDB. And I think in SQLite, it would end up being a JSON array. But this whole confidence order hack can go away because it's buggy. Like Doge's Health says here that it's mostly likely buggy. I am saying our prod implementation on MariaDB right now has bugs. There's an open issue about it. And that is doing a clever thing to get performance out of MariaDB because... And I think that just goes away with SQLite. Does that make sense? Am I making sense? I don't know. I'm a little fried. Otherwise, we'll see what the comments say. I will keep an eye and reply before the next office stream. twitchtd ok, thansk
But otherwise, the next office stream will be Thursday morning. grayhatter_ cheers dude, hope your monday is a good one!
Yeah. Thanks for dropping in folks. Hope to see you next time. Take care. Oh, it is a good one, man. You see how fast the script runs? It's a good one.