Recheck demo / a correct but wrong description

03:11chamlis_ I think you're muted?
How about now? Okay. Yeah. Now I see the, I didn't watch to see the, the thing burst or the thing bounce. Thank you Shamus. So let me jump back and redo my intro. That's not awkward. So this is Lobster's office hours, where we talk about the site and the community and the code base. And I am happy to take any kind of questions and demystify anything. We can do that live. You can just pipe up in the chat room anytime you want to talk about stuff. And then otherwise, most of this stream today is going to be demoing Recheck, which is a database tool that I made inspired by one that I used at Stripe. definedecision calvin here: i just joined and missed the pizza discussion, but is tavern style the true chicago style?
I was showing stream pizza there when I was accidentally muted because I try and start out with something a little bit light as folks filter in from Blue Sky and Mastodon announcements that I'm getting started. And... Folks are probably very familiar with the idea of high defined decision. And like, so two things, number one, Calvin, how many nicks do you have? It's different everywhere. definedecision i do know my other friend from chicago prefers detroit over chicago style
And yes, that is exactly where I am going with this. Everybody knows Chicago style pizza, this big deep dish thing. And I was even just as I realized I was muted that I could actually identify some of these, like this is unquestionably Lou Malnati's pizza. definedecision i hear Pequod's is great for detroit style in chicago. i forgot to try it when i was there
And then there was a Giordano's pizza here somewhere. Huh. Weird that they don't have a Pequod style. Maybe that bottom one. So this is a matter of some religious fervor. definedecision i had lou's when i was in chicago b/c i was obligated
I don't know that I would call Pequod's Detroit style. Maybe Roots? I don't know if that's Detroit style or Quad Cities style. jangomandalorian 👋🏼Hi everyone!
Yeah. It is an obligatory thing for chicago tourists but yes the other kind is and and i mentioned that deep dish is a sometimes food in the same way that for cookie monster cookies are sometimes food hi jenga welcome back and But like, it's so big and so harding and so valuable to belt manufacturers that this is not something that we actually eat on the regular, especially as we get older. Chicago tavern style is the other kind. So this is, this is the not so famous Chicago style. And though Calvin knows it right off, most people have never heard of it, but it's a thin crust pizza cut into squares. And classically, the idea of this was, imagine you are a mid-century factory worker and you get off shift and you stop at the bar on the way home, you have a nice thin slice of salty pizza and a beer before you go home. Grabowski. There's a fine Polish name. So the pizza I had for lunch, was Chicago Three Ways. It's tavern style. Second one is here on screen. You see this, the standard, let me put this in the notes, because this is very important. Oh, and I need the definite article. definedecision sausage > pepperoni for sure
The standard pizza topping, the default. crumbled sausage italian sausage i know in a lot of the countries the default topping for pizza is pepperoni no absolutely not and i realized it was so weird for me as a kid when i grew up and i started traveling because here the default pizza topping is sausage like you'll get a cheese and you'll get a sausage and then if you have enough people to justify more pizza you will get different kinds but sausage is the default not pepperoni no so I am sure there are people who are fans of pepperoni. I'm not going to judge them in public. But sausage is the default. And then the other thing that was on the pizza, oh, thank you, Google, is giardiniera. Oh, yeah, you want the marconi. Giardiniera is another food that's only available within about 50 miles of Chicago. It's kind of a weird one. It's, yeah, maybe the homemade picture is better here. It's pickled chili peppers, carrots, sometimes cauliflower, sometimes broccoli, just hard, often root vegetables, pickled, and you can put it on anything. definedecision ok not pizza but also chiraqi: what's your opinion on italian beef
fast food like pizza and hot dogs and such are the most popular but giardiniera is a irregular oh god i can't spell it right even looking at it it's one of those words where the spelling of it is giardiniera okay that's how you say it and that's how it's spelled it's kind of like french room know the place where your parents entertain and maybe your grandma has plastic on the couch that's all the way in the front of your house that is the french room i should have had i could have made it four ways but yeah the french room is that place where your parents entertain so very important what's my opinion on Italian beef. Generally positive. chamlis_ in the UK this is what I think of for Chicago-style, from being in every supermarket: https://chicagotown.com/the-ran…
It's not a thing that I actually have a strong opinion on it. I've had plenty of Italian beef, but it's just so damn messy that it's not a regular thing for me. chamlis_ look forward to hearing that's an abomination
What do you think of for Chicago? From London. You think of cookie settings. Come here. Deny. Is this a chain in the UK? Oh no, it's a... in the supermarket. Yeah, that's only about half the thickness. It kind of looks like a cartoon. small version of it compared to the, it's not an abomination. definedecision OK one last chicago food question: the hot dog. do you?
It's just, so the pain in the butt about Chicago style pizza, because it's so thick, you have to have a special pizza oven to make it. Chicago style pizza, so like most pizza ovens, oh, I'm gonna misremember the exact numbers. Most pizza ovens will get up to like 700 Fahrenheit. For Chicago style pizza, you have to get up to like 900 Fahrenheit. So you really don't see good Chicago style pizza outside of Chicago because they have to install special ovens to make sure that this like lovely, big, thick baby cooks all the way through. So the thing that jumps out at me on this, because it's just made for, you know, I'm guessing this comes frozen. You stick it in your, your range and you warm it up is because your range probably tops out around 500 or 550 Fahrenheit. It's just gotta be thinner. There are a couple of these places that will ship you a real Chicago pizza with dry ice around it, but I don't know that any will ship across Atlantic to you like that. Jeez, the price on that would be incredible. Yeah, last Chicago food question, hot dog. Yes, absolutely. For anybody who doesn't know, the Chicago style hot dog, ooh, it has a Wikipedia page. definedecision cesium salt 😋
It is a hot dog with seven different kind of toppings, and it is kind of an homage to Waves of Chicago Immigration. It was deliberately designed with ingredients from various backgrounds. Cesium salt? No, come on, celery salt. I hope that was autocomplete. chamlis_ trying to juggle 9s, 5s, and 32s in my head with these temperatures :p
The nuclear green relish in here, though, that's where I would start thinking of radioactive materials. Yeah, very important culture note.

12:09dpk0 moin
yeah i would give you the numbers in celsius but i don't think of baking in celsius so i have no idea none at all all right so a few more folks have shown up so i will remind this is lobsters office hours if you have any questions otherwise i am going to use the lobsters code base and dpk0 oh no, i missed the pizza discussion. will have to catch up on that
a, I don't know, a couple months old backup of the database to show off Recheck, which is the tool that's shown up a few times here on stream, but I don't think I've ever really stopped and dove into it. I know I showed a little bit of the code once a few weeks ago. Yeah, well, DPK, good to see you, and that's what the archive is for, for all that vital pizza-related content that I use to drive away Twitch viewers. So... A thing I saw over and over in consulting. So I did consulting mostly on Rails apps, a little bit of Python, but Django and Rails are like cousins for how similar they are. And we would see these weird bugs where like one customer would complain, they can't load one page or we'd get, you know, the five hundreds in the error reporting thing. And in the middle of debugging, we would do something like user or whatever the God object to the system is. Cause you know, in all of these rails apps, it's always user as a God object. And then whatever your app is about. So for us it's story or comment, right? But like I would run user dot find 1, 2, 3, 4, 5, 6. And then somewhere along the lines, I would call .valid on it. And I would get false. And this is ridiculous. The data that is straight out of the database before I have had a chance to apply any changes to it is like the archetypal, well, this should be valid. And the first time I saw this I was like that's weird and kind of impossible, and then the second time I was like that's weird. And I guess I have to admit it's possible it's not a one off right. And then, like the 10th time I saw this I was like all right there's no such thing as a 10 off this is not a one off this is not a two off this is not a 10 off this is. a fundamental issue happening and I don't know what I'm seeing. And it was really frustrating because nobody was talking about this kind of thing. Oh, my other go-to example would be working on an e-commerce where like, I don't know, people order widgets and they have orders and I would find something where like there's an order and the state machine, like the state field on the order, If I said order.status, I would get something like open or unplaced. You know, whatever is saying that it's open and the customer can keep putting things in their cart. But then the weird thing would be like I would find a... You could say like order.shipment. You know, say that's your... your active record model. And it's like, oh yeah, here's how does, how do they style it? It's like shipment one, two, three. And it's like, wait a minute. How does an order that has not been placed? How is it already shipped? And then I'd look at the shipment and I'd be like shipment dot status. And it would be like delivered. Lol. Like This is an impossible combination of facts. It should not be possible to have an order that is unplaced and yet delivered. And sometimes this would be even more complicated, like an order that is unplaced but has been delivered or It would have like a shipment estimate and no delivery. And then the customer would have a review of items in the order where they're like, I love my t-shirt. It's great. My dog likes to wear it too. And it'd be like, but, but you never placed that order. And we only have half of a shipment order shipping record that says it didn't actually get shipped, but clearly you got it. Like what, what happened here? What combination of facts happened? Cause none of this should be possible. And so I spent a little time pulling my hair out. pushcx https://push.cx/talks/kcdc2016
And eventually it became a talk. So I'll throw the link here. In case you have not heard enough of me talking, you can see the video. But then the the interesting thing is the slides is I found basically six classes of issues in Rails apps and in ActiveRecord itself that lead to these kinds of impossible data.

17:41And I'm gonna like grab this. So like these two, ah, just grab all of it. These are the two examples I've found most useful for explaining what's happening here.

18:07So I would bet that everyone who is a developer, who's listening, who's had a database with just ballpark half a million rows, you have seen this kind of thing, right? state machines in impossible states associated records that are missing or present when they should not be possible or just basic validations that don't pass and it led me to give this talk probably if you want to skim ahead i'm not going to go super deep on it but Or is that overview slide maybe I didn't talk about show the overview until about slide 50 where's the. So I found these six ways that things can become invalid and stored in your database and when I say invalid, I mean. invalid for the purposes of your application. You cannot retrieve this data and operate on it normally without getting 500 exceptions because your functions at some point have to trust some of the data in your database. And this is all even if you don't call validate colon false. So there's like in Rails, there are explicit ways to say, I want to save some data without calling the validations at all. Rails provides that escape hatch. And the funny thing is like you will have these other issues and end up writing code to update records that has to call validate false. And I'm pretty sure we have examples of this in the Lobster's code base.

20:08Let's see. Validate. Cooling? Yeah. So, and of course, it's our got object comment, right?

...22So, yeah. This is... I got lucky, not lucky, I got experienced here where eventually because you end up with this invalid data in your database, code that updates existing records starts sprouting validate false because you try and run a migration to fix stuff or you have like the admin interface for editing a record or you have some utility method that like regenerates all the markdown. there's weird ass data in there and you're like well i just want to knock out this feature so i'm going to move on what were the other ones oh yeah so if you delete a comment we don't validate deleting the comment that's a weird one what's the other one undeleting yeah undeleting and there were three so this kind of thing where uncommon and administrative code paths start sprouting validate false is a sign that this is not an isolated issue this is a regular thing that happens and to give the i'm not gonna like deep dive this whole talk because it was a what 30 35 minute talk and i had to kind of run like hell to fit all of it in but the the very short version is these are easy bugs to make most of them are actual bugs some of them are like partial rights they're kind of they're a uniqueness or not i'm sorry not uniqueness they are a sort of a transaction level issue where rails you don't want to take a transaction for long enough to read and then do back a partial write so there are like yeah some of this is race conditioning kind of stuff and before anybody can make the objection yes almost all of it you could make sql constraints for but sql constraints are very very painful and nobody in practice nobody does them at this level because They're code that lives in the database and they don't have a great UX around knowing which ones are going to run, knowing when they ran, being able to version them, distribute them to developers, know what's running in prod and the developers are using the same thing in prod to revise these and deploy them in a lockstep fashion with your backend and your frontend, which is how you have to use them. The median case for all of these, so I've spent most of my career over in dynamic languages like Ruby, Python, PHP, even some Perl, where the various ORMs and database libraries just do not make use of database features like SQL constraints. Rails only, well, it's not recent anymore, but for the first few years, Rails was suspicious of even using foreign keys in the database, which was not a design I would have made. And I don't think, I think if anybody says, oh, you just have to use SQL constraints, I think you should stop listening to them because that's not a reasonable solution. They are not a cheap enough tool for fixing this stuff because this stuff does tend to be

24:22issues where you have hundreds of thousands or millions of records. And sometimes it is literally a one in a million data bug. Like the race conditions, the ones where I was debugging race conditions, the thing I eventually learned

25:05The thing I learned was that race conditions happen at the top of the hour because cron jobs are so often scheduled to run at zero minutes past the hour. When I saw bad data come from race conditions, it almost always was like 2 p.m. and three seconds because it takes a minute to start the Ruby VM. And then if a user is clicking around in the app at the same time the cron job is running in the background, that's the only time you really see these race conditions until you're at truly staggering scale.

...48But they happen all the damn time with background jobs and external APIs. So it's background jobs, external APIs, and privileged users. These are not big, complicated bugs. A lot of them are really simple and small. But there are design issues with all three of these things that lead to these bugs. With background jobs, when things crash, no one is present. The thing I have seen over and over is that there is a background job queue shurizzle tavern style, crumbled sausage pizza and giardiniera. So Chicago is south Italy LUL
And when it crashes when it has a minor bug and it gets halfway through editing a record or halfway through editing a clump of records that makes up some meaningful business entity like an order and its shipment. There is a 500 that happens. and no one is there to get it and maybe it goes into some exception monitoring system but those don't often get treated with urgency and deliberate attention and so they just fall off the end of the world and you end up with this kind of impossible data where the state machine that is supposed to mark orders as completed once a day as it sees their shipping records broke in the middle of the night at 3 a.m and now there's just some bad data in the database one with the external apis and this also touches on background jobs but every time somebody has an external api pretty much the only way that the pattern i see over and over is you have some record and you it has like nine fields that you fill in from your app and then that last field is Maybe to continue our example, we have an order, it has a shipment, and then for that one field of shipping cost, we say, okay, hey, FedEx, what is your estimate for shipping this? And the way that always gets implemented is you persist the record with a null, and then later a background job or some other thing comes along and fills in that null record. And if the remote API breaks or any of the background job stuff happens, it just doesn't happen and you end up with nulls in your database.

28:45I hope that this echoes something you've seen. If you have seen a production bug that looks like one of these two, I hope this is all very familiar.

29:11Well, not fix, but work around errors in code and in business processes. So when I say privileged users, I often mean customer support who don't often have a lot of training, but they are also not incentivized to make sure that the database is in a consistent state. They're incentivized to make sure that the sale closes or the order goes out or the shipment gets packed. or that they fix the other bug and they don't notice the one they are inserting. And so privileged users need these escape hatches that bypass validations, or they are using cobbler shoes kind of tools where the validations aren't quite there. Because, you know, the internal tools always get made a bit in a hurry and then you have these impossible states in your database where the CS agent saw something weird and they tried to delete the shipment but they only got rid of the shipping record not the whole order and that's how you end up with like these are not exciting kinds of errors I was on leading up to the stream I posted on Mastodon that I was gonna talk about this a bit and I I don't know how to pronounce it. Aphyr? A-P-H-Y-R. You can say hi if you're present, but he responded. He is the guy who has done Jepson, which is...

30:50pushcx https://jepsen.io/
This is a wonderful bit of tooling and research. I will throw the link over. And he does a ton of work with distributed systems and multi-master databases where he has found not injection issues like Jepson. Yeah, I'll say it like that.

31:30Jepson is great for finding out that when your database talks about how transactions are propagated between multi-master setups, Jepson tells you that it falls a little bit short of those docs and those promises. These bugs are not really that. They are not exciting computer science bugs about the dining philosophers and the Byzantine generals or formal method things about what can you prove and what's your actual state machine. They are really unexciting bugs. They are nulls. They are customer support deleted something, but not all of it. And they are mostly rare.

32:30So the thing about rare things is that when you start operating at scale, rare things happen all the damn time. When I joined Stripe, I saw a neat tool they had for introspecting on the database, the production database, and saying, hey, did this bug recur? You know, if all orders have to have a shipping record to be considered delivered, do they? And it was a way of cheaply making sure that bugs didn't recur and other guarantees were followed. And after I left Stripe, well, when I first learned about it, I went, oh, my God, I can think of three bugs off the top of my head that this would have caught just in the last six months or some of these really fiddly bugs with background jobs where it cares about the order the background jobs run in, it would be really nice having fixed something to have the confidence to say, oh, yeah, this bug is not going to be accidentally reintroduced nine months from now. And if it is reintroduced nine months from now, it would be nice to find out about it before a user finds out about it.

33:54So with permission, I have implemented my own version of that system. And I'm starting out. I've said Rails, Rails, Rails, and ActiveRecord. But this is every database tool. And I will be, if I get any reasonable amount of engagement and use out of the Rails-oriented version of this, I'm going to come back and immediately port it to other languages. So if you see this, please do not run off and throw it up on NPM. Give me a chance. And let me just kind of... I've been talking for a while. Let me just show what it does. So I'm here on the Lobster's code base. Why don't I have commits that... Get into that later. Let's make a branch.

34:55And I am going to, well, it's something to run in prod.

35:14I think this is the right syntax. We're going to find out real fast here how fast I remember. gemfile syntax but I think that'll install off of the dev version that's sitting as source on mine looks like yeah so let's bundle exec recheck setup and we get a whole bunch of output so The big value in writing these checks is making sure that you don't have regressions or that things actually work, but out of the box, I can give some useful stuff. Like everybody's got DNS, everybody uses HTTPS, everybody has Whois. So I can check a couple of production things about that. And then also I can introspect the app So it detects that there are active record models, and then it says, okay, for each of these active record models, let me go ahead and set you up with a default check. I'll talk about what that means in a minute. And there's one in here that's special. This one was a bug fixing session, yay for alpha testing. But if it sees that an active record model is read only, it knows it's probably a view or some other kind of uneditable record, like maybe a join record. So we just skip trying to make a checker for that. I sometimes lap into the Stripe jargon. I'm trying to call them checks. Inside Stripe, the system was called Checkers, like the board game. I called this Recheck just to get away from it, both because it's an independent reimplementation with no code or data from Stripe, but also because trying to show up in a Google search against a children's game that is popular across the entire world is not a smart move. And then, oh, this was a fun one. I felt so smug about doing this because I wish generators did this natively. But I also have it detect that you have a linter installed in your project. And if so, it automatically runs it so that the default checks just get slapped into your linting style because why wouldn't you want them in your linting style? It also does, if you don't use standard RB and you just use RuboCop, it detects that. So we're gonna, I'm just gonna follow these instructions. Yeah, I did mean with two dashes.

38:25Initial setup. And because this is a demo, I'm gonna sneak one piece of code into it.

...38Default reporter. So I'll start with the thing I'm adding, even though it's not the clearest, but I don't wanna be like, then there's some secret magic code that happens, cause that's no fun, right? Who needs that? So default reporter, When Recheck runs, it has these reporters that tell you about things being broken and you can customize them and display data in lots of different ways. So offhand, like right off the start, I have a default reporter that prints to terminal and I have a JSON reporter that gives you a JSON output so that you can dump it into a database or something else. As this gets into a beta release and then a full public release, a big valuable thing is having the reporter go ahead and send notifications over to, or I'm sorry, having a reporter that persists to a shared database so that you can see things like trends over time. The point of these checks is that they run against your database over and over and having some kind of history is valuable. So for commercial projects, hopefully. So this bit of custom code that I pre-wrote is, as you familiar viewers can tell, this short ID is very Lobster specific. I try to avoid showing tons of bare record IDs from Lobster's data on stream because there is a weird cache to, are you user four in the database? Or are you user four million in the database? And I don't want to start those dynamics because I saw how toxic they were on Slashdot back in the day. So I wrote out this whole thing that says, yeah, if the record has a short ID, we'll report that. definedecision you should just turn all the IDs into UUIDs
If it has a regular ID, we will just show the last couple of digits of it because this is a demo rather than I'm really going to use the tool. I could turn all the IDs into UUIDs That is a bigger project than I want. And I'm not sold on UUIDs for tables. And there's, what is it? Type ID? Someone has made a project. pushcx https://github.com/jetify-com/t…
Yeah, type ID is almost the one you could talk me into using. So there's UUID v7 that this is based on. where first off instead of just being hexadecimal it's base 26 yeah base 32 fine and if you've worked with the stripe api you have seen that they have a hungarian notion notation prefix on keys so you won't see like ID 123 for a user, you will see this string, like user underscore some opaque identifier. And that prefix is super useful when you are a user of the Stripe API. And I say that having worked there, but I thought this before I joined Stripe, and I wish everyone would steal this. Yeah, we could talk about that a little. The reason I haven't done it on lobsters is, so this project is inspired by Stripe APIs.

42:35Hold on, I gotta think through this sentence in my head before I say it out loud. There are more interesting things happening in Stripe's IDs They look like this, but they are implemented very differently. And I really wish that Stripe would write a blog post talking about how it thinks about record IDs and API keys. Because I have seen the internal versions of those documents and I wish Stripe would talk about them because they would meaningfully advance the state of public use of these kinds of things. And that gap between what's here and how I know things could work is endlessly frustrating to me. So please, Stripe, update your dev blog. I know it's kind of quiet and you're always looking for topics. I don't know if anybody from Stripe is here. I know there are some other xStripes who are probably watching, but I don't know. Every six months or so, I like hassle someone I know who works at Stripe. And I'm like, how about that blog post? Fingers crossed. All right.

44:06So there's that demo reporter. So I'm going to just run recheck. actually why don't i show what we have here so yeah i'll show it and then i'll run it and feel free to pipe up at any point with questions i know i've been talking continuously for i don't know like a half an hour about this but anytime you have questions or anything just feel free to throw in a message or if you want to see the code i am happy to so the basic idea is the heart of it is if there is this invalid data in our database We need a really cheap, affordable tool like testing is to check that our production data is still correct. And then we run these in the regular on the background. And the demo I'm going to show you today is just me running it from command line. The production version of this has a whole service that runs continuously in the background. And then You can imagine that if you have hundreds of millions, billions of records, you have to be very smart about your strategy for running these, where you don't look at every record in your database. You look at all records updated in the last, well, like once an hour, you look at all records updated in the last day, because sometimes those timestamps are part of the bug. And then once a day, you look at all records updated in the last 15 days, say. And once a month, you're going to look at roughly every record in your database, but you probably just do some kind of slower scheduling of that and once you get up into the billions you have to be smart about stochastically sampling your records ideally with a i keep coming back to a time component of being recently updated because it is your recent records where that you are most likely to introduce bugs especially once you've been using a tool like this for a while and have cleaned up your data but then also You have bugs where the code is running and changing, and that's going to be your most recent records. So let's look at one.

46:33Let's look at comment. I think comment is going to, not the comment model.

...43So here's a pretty simple checker. Check, oh, Peter. The core of a check is let's query some records out of the database and check if they're valid. This is the heart of it. Every check is really cheap to make. So it's one query each and then multiple checks. So the simplest version is, well, we know these active record models have a .valid method, let's call them. So let's run that.

47:34I go back and forth if the directory should be named recheck or just check, like it's specific to this tool, I don't know. So it's gonna go, Oh no, it's going to have a bug? A bug in my demo? Oh, fun with live coding. Did I break things before my demo? I'm going to off-screen look at the code base real fast.

48:15What if I... one of the nice things about using that gem with a path is i can just tweak the oh no i'm in hell i'm in hell because i broke the code and now i have to debug it if i'm going to show any kind of demo of it actually running

...55so so actually i think i can just throw away this stat and be fine because this is just counting statistics cool i ran faster than i needed it to so This noisy line is that heinous inline partial in lobsters that we delved into. So it loaded up the comment model, and it found that the first 20 comments all failed the checker. None of them are valid. And it printed out their IDs, but Recheck has a safeguard where if the first 20 records it examines are all failures, either you have a ton of bad data or you wrote a bad check so it's not worth checking the whole table. So it just kind of bailed out there. But that's weird, right? There shouldn't be comment records in the database that are invalid. And this one, it's not traversing in any particular order. But we know that MariaDB likes to give things back in insertion order unless something odd has happened. So

50:30So let's take a look at that comment.

...39So that's weird, right? Like I pulled this comment out of the database, I asked if it was valid, and I got false. How is that possible? What are your errors?

...57dpk0 heh!
And the gist of this error is, a very common situation with validations on Active Record. Yeah, I didn't remember that this example was in here. I just figured if I run on comment, which is our God object, we'll find plenty of examples.

51:24Let's go look at that validation, because something about a validation, right? So story is no longer accepting comments.

...40So it asks the story, hey story, are you accepting comments? I know how this is implemented, but I'll show you. Accepting comments is, dpk0 so it fails validation only because the story it’s on is too old, even though it wasn’t too old when it was originally submitted?
the story isn't gone so like it's not deleted and you're not previewing and either this is a new record or the story was created in the last what is this 180 days yep dpk you got it it fails validation because the story it's on is too old even though it wasn't too old when it was originally submitted So most of these ORMs that show up in dynamic languages, this is not an active record specific issue. dpk0 so you shouldn’t use ORM validations for that particular case
They have the validations on the model, but then validations get written for the code path that is in use at the time. dpk0 i guess is the solution
dpk0 ?
dpk0 right
So they are used to validate a form, but they live on the model. Well, you should be able to use ORM validations. And I think this is probably hmm i think there is probably a larger design issue with the active record pattern where validations are a thing that live on the model instead of you are validating a change because if you are validating a change it fixes a whole bunch of stuff like where are they

53:22Oh, does comment not do it? We're not even using the Rails feature that works around this in a weird way. Do you have it? God, objects are always so weird. Yeah, so in Rails, you are expected to take like a story object and then say, which attributes is the form allowed to update? And then also here's kind of where we do the auth check of, well, if you're a moderator, you're also allowed to update these other fields and then you just kind of slap them onto the model. And then the model has to inspect all of that stuff and say, is it valid or not? I think this clunky expression would be better served by saying, using like the what is it called the unit of work pattern to say we are about to do a story edit so we are going to pass in the story the edited params and the user which are the three inputs to this are you allowed to or not validation rather than the validations living on the model so it's this validation it's this validation that's not actually quite accurate. If we're going to do this on the model, which is, I guess it works. Not every Rails app has split out their own form objects and put validators on them, although many do. What we want to say is, let's see, we want to say if we are a new record, Then we need this.

55:25And new record isn't precise because really you could be editing a record in the middle of the edit window, like the first hour or two where you've posted a comment, you can edit it to fix typos. This is fine because that will generally always be in the story window. People are rarely posting new comments right before a story stops accepting comments. But it does point out that this isn't actually an issue in the database. This is an issue in the validation. So having fixed that, and I hope that YOLO fix is correct. Let's see if this check wants to run longer. Oh, undefined method new record. Comment, oh, it should be self, right? Yes. Self.newRecord. Did not YOLO. There we go, it's actually working now. So comment, because it's validation goes and pokes other tables in a one plus N kind of way, that is fine for individual records, but the opposite of fine for this bulk sort of thing. This is running pretty slowly. Each of these indicators is checking 1,000 records in the database. Lobster's has, what, around 600K?

56:59Oh, in this copy of the database, only 518,000, but I think in production we're closer to 600,000 comments. So this is going to take a minute to run. I'm not going to run the whole thing. I think I still get the good stats if I cancel out. I recoded that, right? Oh, no. I dropped the stats on the floor. Let's put that on my stream.

57:35Still alpha code. So here's another table I'm going to pull up real quick, just because I know it runs faster. So messages, I'm not going to pull one up on stream because these are private messages. But similarly, you can kind of see, oh, OK, so we have 13,000 records in the database. and 99 of them are invalid for some reason this shouldn't be possible like oh what is that one in a thousand of them have some kind of issue i happen to know this one off the top of my head it's another issue with validation it's a model message this one is dpk0 what’s the condition again where it decides that the validation itself is bogus? where all of the first twenty fail or twenty out of some other number?
When you send a message, you can use the hat. So if a user has a hat, like whether that's Lobster's moderator or... The condition, oh, the condition is called blanket, like blanket failure of all of them are failing is if literally the first 20 in a row all fail or throw an exception. And that's kind of an arbitrary number, but in practice it works fine.

59:07so on that that bad edit that i had the hot patch here on stream it is supposed to say hey because the first there's an extra message that's not printing i better put that on my to-do list here too

...38Like you can tell this is in progress software. I am aware of the irony of having bugs in a software for finding bugs, but hey, that's life.

01:00:02So we didn't trip it here, but if each one of these indicators is 1,000 records, we print a dot if all of them pass and an x if any of them fail. And I'm reaching for our god objects like story and comment because they have the most complexity and they have the most opportunity for either validations to be edited in an inconsistent way or to be written in a way that they only work at the time a record is new and not if it's being later edited. I'm going to stop this. That's not useful to watch. But the first kind of runs of these are very similar where then simpler objects so like here's an invitation record we have 10 000 in the database and four of them fail this one's probably not a validation and if you look at the model invitation the validation is very simple it only checks if an email is valid and this is only Facially valid. Is there an at sign? Is there only one at sign in it?

01:01:29It's possible that we tweaked the length of one of these fields in this validation and not in the database, but I don't think so. So we have a little mystery there. Four in a thousand records. considered invalid that's four and a thousand or four and ten thousand excuse me where they are probably all used at this point because invites don't tend to last too long and I'm not pulling them up on screen because I don't want to pull up somebody's email on stream that's rude this pattern of the earliest records in the database because you see they're all in the first 2000 that's real common so i'm going to show a different style oh well let me show a a bug we have in lobsters and write one yeah so there's a an open bug right now where No, it's not threads. Which one of these bugs is it? Got a little busy right before the stream, so... Maybe it's... Have we been just commenting on the user threads? Here we go. So there is a bug that jmiven, say hi if you're present, you were here earlier, reported where on user pages, some comments are appearing under the wrong parents. That's a pretty concerning bug. I have been tinkering with this one. It is apparently related to the code we have for threading, but I haven't been able to run it down yet. And this kind of integrity bug is where checkers really shine. Let's see, did I make a... Had to make a regression folder.

01:04:12Let's see, user... threading1236.re.

...31And I call it that because that was the issue number over on GitHub. And a big, this is a tool for developers. So if I can find If it spits out a failure, the person who is using it is a developer, and I want them to have a connection over to the history for it, especially for regression checks like this. So what I want to check is that in the database, and I'm going to write this one. You don't have to use the ActiveRecord interface. You can, in fact, return any enumerable from query. So I'm going to say comment.

01:05:39Yeah.

...45What I want to know is, is this a presentation issue or is this a internal issue where the comments have the wrong parent ID? And I know it's not that they have the wrong parent ID, but I don't know it unless I query the database. Actually, let me do that because it's easier to iterate on these things here in the MariaDB console. So let's find comments or ID from comments where comments C join comments parent on C dot parent ID equals P dot ID, right? So we're finding all comments. We're joining them against their parents and So this is one of those places where it's difficult or kind of crosses against the grain to express in SQL, because SQL especially doesn't love tree structures. what we're saying is when we join comments against itself so join every comment against its parent they should be on the same story and i'm going to just also make sure that i'm saying that where c dot parent id is not null and i think that's implied by this join Yeah, actually it's implied by the join. That's fine. I always have to think through ternary logic. C dot parent ID. What is it? Parent comment ID. So this one's useful. There are no stories in the database.

01:08:25So this is a slightly different style of check where we're saying, look, if you can find any comments where the comment is replying to another one that's on a different story, that's just bad data. That should not be possible. Something is seriously wrong there. And as straightforward as it was to write, it's the kind of bug or it's the kind of query you have to write when you're investigating a bug like this. You have to go and check in the database. And even if you see an issue, you have to write this kind of query and say, are there more? And so it is. basically no work to take that query and say, you know, this should never be the case. We should never write a bug that write code and store data that violates this constraint. And so I've called it regression, although this one is more of a constraint check where we are checking a constraint that is difficult to express in the current code. or is not yet expressed in the current code, or has been violated in the past, or we think may be violated in the future, a check is a really high ROI, hopefully, tool for developers to get database integrity. Yes. I could write the active record validation for that. Yes, I could write the SQL constraint for this. But if I am bug checking, I got a whole lot of assurances for copying and pasting one line of code.

01:10:20And it's a line of code, a database query that I had to write anyways.

...32That needs a better message.

...56Yeah, so I wrote my little note to myself, but because this constraint check found no records, it was valid. I could give better output for that.

01:11:17It's also worth checking. This is not super obvious from the UI, but, ooh. Whoa, that is impossible data. I totally thought that was gonna run clean just like the previous version of it. So internally in lobsters, there's a thread ID and the thread is a unique integer for each top level tree of replies. So let's go look at a story with a couple of comments on it. And I will point out what I mean by thread. Sure. So you see the tree of replies. There is a top-level reply here from Fan with, what, a dozen responses? And then a top-level reply from Technomancy, a top-level reply from Mparam. So these are three different threads in the code base. And it's really just an internal tool, but because there are no features for moving comments around, it should not be possible for any comment to reply to a comment in a different thread. So in the two seconds it took me to flip this one over, we just found an actual legit bug in lobsters. That's probably never going to be reported because it is literally one comment out of 600,000. So let's investigate this one, right? This one's not going to be a validation that's wrong. There's, I don't believe in one and a half million, not for our code base.

01:13:15Okay. I'm always curious with bad data. How old are you? So this is really conspicuous. This is early in the site's lifetime.

...33Yeah, the site was about a year old.

...42All right, so is the comment valid? Yes. What's its thread ID? Okay.

01:14:00Man, that is weird. Let's go look at it, right? Oh, actually, I don't even need the short idea of the comment or of the story. So let's grab that. That was the comment short ID. See, there's where it would be nice to have those Hungarian prefixes. Firefox, why did you give me the settings? So this comment responding to this message has the wrong thread ID. Oh, and look here. We see JCS, who is the guy who started the code base. The comment ends with an edit saying this was supposed to be a reply to Kyle. JCS seems if I click Reply and then Preview, my comment loses its parent. So oh, man, what a great demo. I swear I didn't plan it. This is an example of a privileged user. A privileged user has come and edited that comment to fix its parent, but he didn't remember to fix the other attribute on it.

01:15:19What was that ID?

...33So it should be 2290. So 12 years ago, the code base had a bug with replies that prompted one of the first messages, because this has got to be a fairly early comment, that prompted one of the earliest comments to not have a valid thread ID. because it was manually fixed. And this bad, invalid data has been lurking in the database, and it could lurk for years more until some other tool comes along and blows up in a weird way. Or the other tool, like the thing you want to write on top, cannot be as reliable as you want because you find yourself saying like, all right, well, we're going to loop over these comments and we're going to loop over their threads. And now I have to add an if because my process crashed and you don't know if that's a one in a million kind of thing or a one in the whole database kind of thing or a common error because you're trying to accomplish another job. So right there, I'm a little smug about that actually, that a consistency check just immediately found a real error. So this style of, if you can query for it, you know it's gonna be bad data is really powerful and it runs fast. Like this one takes a second or so where running the whole suite I think I ran this the other day and it took maybe 15 minutes. Oh, and it's gonna start with comment, which is too darn slow. Oh no, it started with category. So like all of our categories are valid, great. So this is why checks are useful to run in the background and say, do we have any new data? And, Yeah, I'm not gonna... Let me show that one here.

01:18:06Where did I put it in the code base? Oh, it's just in the README, isn't it? So I've started the README and I'm not gonna show the whole thing. But there's an example in here of the structure of a check is that you might have a setup, you query for records, you write one or more checks for them, and then you can have an optional notify block for production. So in the case of that issue with if there's a comment, you know, imagine Lobsters is the size of Reddit and we get 10 million comments a day or whatever ridiculous number they get because they absorbed every forum on the web. Well, there's going to be some commenting team or internal issues team. And so when you run in production, you can say, all right, I want you to do the production notify, which is for this check, probably go file a ticket in their bug tracker. So you can break up checks by team very nicely. And however that team tracks work, should this bug get reintroduced where a production comment has the wrong thread ID, it will immediately start filing tickets or otherwise notifying the team. And it's just plain Ruby. So whatever that team does of Jira tickets, GitHub issues, emails to a particular inbox, you can just code it. Oh, typo.

01:19:53Still going. That's going to be a minute. So I'm going to show a different style of check here. Eh, not style. What do I call it? Site checks. So some of the things you can get out of the box are stuff like you can query. Oh, I called it email records. All right. So I generate this check for you. We have two domain names on lobsters. We have lobsters and then we have a backup domain, lobsters.dev. So by putting them as the query, if we return any enumerable, we can have checks that all of them should have MX records. They should all have an SOA record. They should all have an SPF record. And if I go to recheck site, DNS or what did I call it? Let's do TLS.

01:21:15We can check other things about, so you've probably seen production tools where there are various SSL validity checkers. We'll just start another one.

...39Man, that was like Jordan Eribert.

...46I have a little bug in this. So checks can also be against things that don't feel like a database, like our list of domain names. And they can check for things like, is our domain about to expire? Is our SSL cert about to expire in the next 30 days? Are we using old suites that we shouldn't allow because they're insecure? There's not a lot of magic to checks. Ideally, this feels a lot like the mini test API where it is straightforward that you do your, what is it? Act or arrange act assert. You do set up query and then check, check, check, check. dpk0 is the stream broken for anyone else or just me?
then the magic happens when after an initial run you clean up your bad data and then you catch stuff as it comes in in the future so i can keep investigating that threading bug and say okay well these comments aren't actually on the wrong story they aren't actually on the wrong thread this is a ui issue and for the amount of work it took me gtfrvz nah, strim down
to do the quick check of the data to make sure that the data wasn't getting generated and inserted wrong, I have an ongoing value. As long as I maintain these checks, and they're pretty cheap to maintain, you know that you have good data in your database.

01:23:45That's about everything I wanted to demo with Recheck. So if you have questions, now is a great time to pipe up. So I have been, as you can tell by the bugs and typos, I have been very actively developing this. And I kind of alternate between working on it and then running it against Lobster's data and And it's actually kind of funny to me that I was able to immediately find a new bug in Lobster's data by writing one here, because that's the experience of it. If you don't have these kinds of constraints, you just get one and a half million kind of issues. And you get like one customer who writes in and says, I can't load this one page of the admin panel. You're like, what in the hell?

01:24:50The other way to express this validation would be to say, like, new record, or you're in the edit window. That would maybe be what I end up committing here.

01:25:06I've joked that the stream's motto is that if something is worth doing, it's worth doing badly. The gist of that philosophy is Why I keep coming back and talking about SQL constraints here is, yes, you could write SQL constraints, like to make sure that no comment is inserted or edited to have a different thread ID than its parent. But in practice, we never do that because it's painful to write and it's painful to maintain. It's just not worth it. But being able to knock out a check, a one-liner check that runs a query that takes, what, a tenth of a second to run? Most of that runtime when I ran it here on the console was booting Rails and then exiting.

01:26:10That's a big value. And it's valuable, especially when are operating in a business there's my my sales pitch if you make money on your site you would really like to have checks and catch issues before you're paying customers and subscribers find it or having found one before more people do hey nagwe welcome i'm kind of winding down i have done the demo talked through some of the philosophy I will show the little website I made for it. Actually, I'll also show the readme. So I made a website and it's just a kind of a sales page that introduces what this guy is and says, hey, please sign up. So please do consider signing up if you are, especially if you are willing to beta test. nogweii oh, hello there!
So I have been alpha testing it on lobsters and I would really, I think I'm going to have these last, like these things, the print better error messages. These are each just a couple of minutes kinds of bugs. So I'm going to iron those out either this evening after the stream, if I don't have dinner plans or tomorrow morning. pushcx https://recheck.dev/
And it's just about ready for. a user who is not me. So if you think to yourself, boy, this is awfully familiar, especially this early part about finding invalid data in your database, please email me or sign up for the email list because I would love to have some beta testers. And I think I'm going to do a staged sort of thing where I invite literally one or two people and do probably do pairing sessions to get people started because I think immediately people are going to run into issues that I don't see because I'm so close to the software having touched it almost every day for the last couple months but then if it doesn't match what someone comes in with or it doesn't explain itself or it's not clear how to get started I gotta talk about that some more So this was... Here's my draft readme. I've spent a lot of time polishing this. And my Markdown style is to write one sentence per line, because in most Markdown engines, that formats into one paragraph. And it looks a little funny on the screen, but I get much nicer diffs.

01:29:02But the gist of it is... This intro I gave, have you ever pulled a record out of your database and been surprised to find that it's not valid? You have probably seen this where production data is inconsistent or impossible. And when I say it's not just your code base, it happens in every database at scale. I say that because I have had this conversation a lot with people, both when I gave the talk, gosh, eight years ago, seven years ago, and no, eight. And also when I've started talking to folks about recheck, people see these issues and they assume it's just them, that they have done something wrong. And it's not. People need to know they are not alone. These kinds of integrity and validity issues are common and they are survivable obviously the world is not actively on fire because of these issues but it is possible to have a heck of a lot less hassle about these and while I would very much like to write the Ruby version of recheck get a couple of customers, and then port over to, I think in order it's gonna be JavaScript for popularity, and then probably Python, because I know it at least a little bit, and then probably .NET or Java. And I really hope that I can popularize this idea of checks, because they are a tool for production systems in the same way that, I don't know, in the late 2000s, everyone came around to the idea of automated testing, whether that's unit tests or integration tests or integration tests that call themselves unit tests like in Rails. One second, I'm gonna clear my throat.

01:31:20But there was a big shift in the industry that automated tests were something that basically every app would benefit from. And as a consulting customer, I have certainly seen customers who did not write any tests, but they knew they were missing out by not writing any tests. And we got a lot of value by starting to add some to their projects. And I would very much like this idea of writing database checks to be popular in that way of, oh, this is just a straightforward thing that roughly every app is going to do so that it can immediately catch issues before they escalate. Because when you start seeing these issues, you don't know, you know, we saw one that was one comment and half a million was wrong. If I had found that from a production 500 or from a customer email, I would have had to write that query to know, is this literally this one record? Is this one in 100,000? Is it one in 10? Is it all comments since last month when I tweaked the threading code? And since you have to write these queries anyways, wouldn't it be nice if you had that certainty very cheaply the time it takes to copy and paste into a check? You have that certainty that your data is correct on an ongoing basis.

01:33:05That's my big pitch for recheck. It would be nice if there were fewer bugs in the database. Man, that run's taking forever. A first run is like that.

...29So I've shown you this kind of basic output of when you run it on the command line, you get some stats on pass and fail. You get the ID numbers that have failed. I think I tweaked this to put them on one line. I should probably update this.

...55There's the basic It works like you would expect from a command line interface. It's very similar to RSpec or Minitest, where you can list the files you want or the specific numbers. Oh yeah, DPK, you had questions about the co-op. I'm happy to dive into that. Yeah, let me finish running down the readme, because we're about halfway. So writing checks, you've kind of seen these two styles of, if there's any bad data, this is like that regression check I wrote for user threads. If you can find anything that's invalid, that exists, it's invalid. dpk0 so maybe you can talk now about the co-op you’re setting up to … commercialize it? offer recheck consultancy on how to use it on your app? what’s the plan there?
And then longer, I wrote an example of, let's say we have users, we have an LDAP service, and we find users. And for all of our users, Everybody should be synced to LDAP in the last minute. And there should be exactly one, no more and no less, instances of a user record in the LDAP database. dpk0 okay cool
And it's not a heck of a lot of code to write, but this catches like nine different kinds of bugs where a user could be duplicated, two users could have the same ID, a user didn't sync properly. et cetera, et cetera. We changed their email and they didn't get over there. You could also check other services like that their user avatar appears on the S3 bucket. I threw in some tips for writing checks that basically they're cheap.

01:35:47Oh, and I didn't talk about it. I should stick it in that regression check I wrote, but it's great to just include a link to your GitHub issue or other issue. So actually, dpk, there's a really nice transition here at the end for running in production. I mentioned those notify blocks for filing issue tracker tickets and such. So when you are running in production for real, you'll want to run with dash dash notify. I'm going to go with an open core model. I'm strongly inspired here by Sidekiq, which it's funny that when I was showing what threads were, so I noticed that one of them was Mike Parham. He is the guy who writes Sidekiq. he has very generously let me use his commercial license so the core of it everything that was demoed here and more including you know bug fixes typo fixes is going to be in the open source component of it and i really hope it does well the back-end service so it's two components number one it's strategies for running so this like Find all records and iterate all records. This takes a painful amount of time in production. And especially once you've worked through these kinds of issues through an initial run, this is almost certainly a bad validation if it's so many records. Once you've worked through those, individual runs tend to not show up any new errors because introducing these bugs are pretty mild. And then, once you have just staggering millions you know hundreds of millions billions of records of data, you want to be smart about how you run them and I gave that rundown earlier. That kind of stuff that's useful to big business will be in recheck pro. You can manually hack it out yourself that's fine i'm not offended by that, but. The other part is an admin panel, a nice web UI. You run it yourself on-prem. This probably isn't SaaS because it has to talk to your production database. So everybody gets real picky about that. And as one guy, HIPAA compliance and trying to be like SOC 2 type 2 is just an incredible amount of work for low benefit. So I think it's going to be... a docker image of a rails app that you can stand up inside your data center because you're recording info about your data and that lets you track things like one of the ones that gets you is especially for the regression checks i didn't mention it here state machines that measure their transition times in business days. If you have a state machine like that order method I talked about, or order model I talked about earlier, and it measures where it's in the state machine by how many business days have passed or it takes days to run, You have bad data in there. I have never seen one of these with non-trivial usage that takes days to transition and there is no invalid data. It's just when there's so much time and when there can be code deploys in the middle of a life cycle, there is always invalid data. And that's not bad. It's just we need better tools to support what we're doing.

01:39:54So, DPK, to your question, I am starting a worker-owned co-op. and worker-owned co-ops have been around in the states for like 100 110 years something in that neighborhood and the really simplest version of it is a business oh the illinois definition is a business where the employee owns at least 51 of the equity

01:40:40But more often, it means a business where employees own 100% of equity. And just to 101 for a second, in the standard equity model, businesses use ownership and Let's just say profit sharing. are controlled by shares even divisions of a company so.

01:41:43dpk0 yeah, i wouldn’t consider a non-equitable corp cooperative 🤨
so a worker-owned co-op usually gets away from this standard equity model and it let's say in contrast

01:42:12let's say capped so in recheck just starting out it is just my spouse and i who has a bunch of experience with pr some of this readme is well it's not my editing it's my writing but not my editing oh man having a an editor is so good and if i can get recheck pro sales to happen and i can start hiring people just my personal belief is pretty strongly that there are problems with the standard equity model it produces pretty serious principal agent problems where employees are not incentivized to do the right thing for the business they are incentivized to do the right thing for their performance review for the boss Wrong key.

01:43:26Principal-agent problems is another way of saying it. So if you are the principal, the owner of the business, and you have an agent and an employee, this is kind of an old-timey term, principal and agent, you might say, employee, I want sales, and I am going to pay you for the number of sales you get. And then the employee goes and says, well, if I'm just getting paid for the number of sales, well, let me give all the customers incredibly steep discounts. And then I, the owner, am mad because I had a hundred sales and I'm losing money on each one. That would be a principal agent problem. Alienation, yeah. That one's a different one. So that's different than misaligned incentives. Misaligned incentives and principal agent problems are dpk0 alienation would be the Marxist way of saying it :D
do the actual employees have a reason to care about the success of the business that is not, that is broader than their individual success? Because if you are the employee and you are only incentivized to close deals and you close 10 deals that are money losing, you are very happy and the business is very unhappy.

01:44:42What is it?

...48So alienation of labor. So I'm really, I don't get into politics much on stream or in public. That's kind of a long standing thing. There's a lobster's comment from like 10 years ago about it. So this is a pretty charged way of saying it. Another way to say it is who's entitled to profit? Where does the money come from? So if a business makes a million dollars, and let's think of a small business like the sandwich shop that's on the corner of your block. Yeah, that's me talking as a city kid. The sandwich shop that is nearest to your cul-de-sac in the suburbs, wherever the sandwich shop is. If the sandwich shop takes in a million dollars this month and it spends half of that kind of money on rent and taxes and supplying. dpk0 yeah, okay, so that example is not really alienation, i see how you mean now
The profit left over is a half a million dollars. Who gets it? And in the standard equity model, the answer is whoever owns the shares.

01:46:08And it makes some amount of sense. Well, you know, well, that owner presumably, they put in all their work to start up startup hustle or early resources and investment you know who bought who rented the sandwich shop who built the building who bought the bread that went into the sandwiches this gets a little harder to justify when we're talking about something like the consultancy in the office on the corner and i spend a lot of time at consultancies where people are very directly paying for the consultant's time and there's very low overhead in terms of material investments. You don't need to build a thousand pound dye press to do rails consulting. You need a laptop. Maybe you can even get the employees to bring their own laptops when you are getting started as a consultancy. So why is the owner entitled to the profit? It feels less justifiable even if you are seeing it from the perspective of a red-blooded American capitalist. The Marxists call it alienation of labor, that the profits they have earned are being alienated from them, appropriated by the business owner. Another way to look at it is

01:47:54especially the startup hustle. One I've heard in conversation with friends is, you know, oh, the owners of the business, the owners of the startup are entitled to it because they did all of the work and they took on the extra risk to start the business. But I wonder about ROI.

01:48:19So if you start a business, like, say, Recheck, and it's Just the two of us starting it and we own the whole business. Okay, fine. So year one, we have a hundred percent of the equity and we have a hundred percent of the sweat equity, right? The work put into it. And maybe year two, we hire somebody. And now if you total up the, what is that? So say there's, call it, the work year is roughly 2,000 hours. So there's two of us. So 4,000 hours. Say year two, we hire somebody. So okay, we're now at 8,000 hours and somebody has put in 1,000. but we still have 100% of the equity. Things are going great. We keep working. Year three, two of us are at 12,000 hours and there's more employees and they're up to, I don't know, seven, right? You see where I'm going here. Every year, we can only increase yeah okay I'm working nights and weekends blah blah I have the grind I live and breathe I work hundred hour weeks but you can see at some point the employer are gonna start outnumbering you.

01:50:08At some point, there will be more employees than the person. And if we think of, business doesn't give out equity

...43Another way to think of it is if you invest in a startup, you get some ROI, right? Like you get an IRR and maybe it's like, well, I put a hundred thousand into this building or into this business. And then 10 years later, I have made an inflation adjusted. So let's just say

01:51:13hundred k and the internal rate of return is what is that like five percent ish i don't want to break up my calculator please don't make me do the exponents actually yeah we can do this easier rule is 72 right so if we say it Doubles after 10 years, then 72 divided by five is around 12%. Yeah. So arithmetic is one of those things that's hard for me to do when I'm talking out loud and retaining a monologue like I am on stream. So please don't beat me up if I'm making basic errors in arithmetic and exponents. But if you continue to own all of the business, At some point, your startup hustle is a very small fraction of the success of the business. There's also like the great man theory. So I said I worked at Stripe.

01:52:30So I have met Patrick and John Collison. I have had I had a couple of conversations with Patrick, I don't think. Oh, no, I did have one with John. Yeah, not like super deep, I wouldn't say I know them on a personal level. And I would say they were smarter and more experienced than me, especially running a startup at that scale. Would I say they are 10,000 times better than me? Probably not, but they were compensated with 10,000 times more equity. And sure, I would believe that they are twice as smart and twice as good at startup executing and finance company executiving than I am. I would believe they're 10 times better. I would believe they're 100 times better. But it is another big step to get to 1,000 times better, 10,000 times better, 20,000 times better. and the outcomes are really that unequal and i you know i don't mean to pick on the collisons i mean it in the other direction if i hire employees maybe i will be better than them maybe i will be smarter than them maybe i will be more experienced than them i've done a pretty bad job at hiring if they're if all of those things are true to frankly but I won't be 10,000 times smarter than my employees. And I just can't justify it.

01:54:13So in recheck, all the founders get one share. All employees including founders, get one share on the anniversary of starting. All employees are capped at... We've gone back and forth on the number, but it's gonna be like six to eight shares. Oh, and then a Illinois requirement is that

01:55:08What's the right way to put this? I'm trying to recall the legal jargon, and I've been talking code so much that, you know, all the legal jargon is evicted from the L2 cache. But when voting equity, votes must be, it's redundant with requirement. There we go. Equity votes are one member, one vote. You'll often see it abbreviated OMOV. It is a requirement for the Illinois entity type we have. Illinois calls it a, oh God, it's such a painful name. Limited Worker Cooperative. Let's see, the bill is Act Agreement Association. Yeah. So I call it, the company is formally named Recheck Co-op, but in Illinois, you know how you have to include INC or LLC after your name? We have to include LWCA, which is just such a ridiculous mouthful. I hate it so much. And I realize it is like the most bike-sheddy, meaningless thing, but I am irritated every time I see LWCA. So DPK, did I answer all your questions? I've kind of been rambling about what worker co-ops are, but many co-ops experiment with management structure as well. And they say things like, we will not have any managers or executives, or we will be led by a rotating committee, or we will have something like circles or holacracy, or one of these other experimental and kind of unpopular and maybe not valid above 50 people kinds of management structures i am not planning to do something like that so i don't have a lot of experience or opinion there for me it's fundamentally it's just a business that has it's shares distributed in a different way. For recheck, it's just trying to make this very simple. Oh, the point of shares if we have to be OMOV was profit sharing is when the company makes profit and pays out profit sharing, fingers crossed, right? That'll be a quarterly process dpk0 yeah, well, i know what a co-op is 😉 i was interested in your motivation for doing this, what your business plan is, etc. and this was interesting. thanks!
pushcx [post-stream note: I did all this basic intro because I assume coops are unfamiliar to a general audience. dpk0 clearly knows all this already but I was braindumping notes for how I'll explain this on the coop site in the future. So this came out as disconnected from her questions and experience. I'm sorry to be rude.]

total profit divided among members a member is an employee who owns a share so in the case of recheck that's employees who have been at the company for a year by wages times shares so there's a whole bunch of stuff around how do you distribute profit fairly and If profit were evenly split among employees, it sort of becomes an incentive to not hire anybody. Even if there's a waiting period, especially once software businesses can be fantastically capital efficient and run with like an 80% margin. It's kind of incredible and wonderful if you are a business owner, or I should say a business member, right? A co-op member. But taking someone on, especially that first employee or two is like, wait a minute. If this person has just started, why are they entitled to 33% of the profit sharing? And so poorly structured profit sharing. And then co-ops often, what do they call it? They call it surplus. There's a, because worker co-ops are often created for ideological reasons. As I noticed, Marxism has already come up in this conversation. They sort of have all of their own jargon for everything. That took me forever to learn that they don't actually have a different definition. They just want to save profit without feeling dirty about it. I went to, what is it? So it's funny, I talked to, you know, your standard tech startup bros. And they tell me that I must be a socialist because I want to do this slightly weird equity model thing. And then I went to the 2024 worker co-op conference. dpk0 surplus is the very socialist word yes
There is a national, what is it? United Federation of worker co-ops. Yeah, it's an interest group and lobbying firm for worker-owned co-ops. I mostly got into it because they have very nice benefits. If you were starting a startup in the US and you need health insurance, it is an enormous pain in the ass. We would be able to get it through the Federation, so that would be very nice if needed. But this meetup was here in Chicago. So I went to it and I was considered hilariously right wing where, you know, I don't think anybody actually called me like a capitalist running dog lackey. But when I talked about this stuff in the way of, I see this as a, an alternative equity model, people were mad that I was not revolutionary. Oh man, it was kind of, It was kind of hilariously sweetly lefty and not just, you know, like white people with dreadlocks way, but like speakers who sat down because they didn't want to speak from a privileged privileged position over the audience. And I was like, that's really sweet. But the reason we have a lectern is that so the audience can see you. If you sit on the floor, we can't see you. And seeing people is an accessibility concern. dpk0 haha
It was a very, very funny experience that I am getting called extremely right and extremely left over the exact same topic. Anyways, for me it is just, it feels right. I can justify this profit sharing. I can say, yeah, the reason to wait by shares, which for me just means time, is that it doesn't disincentivize hiring people. And the reason to wait by wages is it doesn't disincentivize hiring people because the fully loaded cost of a senior developer is, I mean, for market rates, if we're talking like 90th percentile, we're talking something like a million dollars a year. If we're talking about market rate for an administrative assistant, dpk0 i mean, i explicitly identify as anti-capitalist and find your explanation perfectly in line with my … ideology, i guess?
we're talking like a hundred thousand a year so if there isn't a wages waiting then it says never hire an admin assistant which is dumb because then all of your highly paid workers end up spending their time doing admin assistant tasks but that's the kind of misaligned incentives so yeah dpk i can see how you got interested in this stuff as a as an anti-capitalist you know i think i don't think you are weird to be anti-capitalist but pro-market economy i think capitalist is not a super great term because the way it mostly gets used

02:03:54dpk0 (but i’m also weird in being anti-capitalist but pro-market economy, so like you, i’m considered heretical by both left and right, i guess)
The way it mostly gets used in... Man, how do I close you? That's not helpful.

02:04:07The way it mostly gets used in leftist discussions, and I say leftist discussions because they're the driving force behind worker co-ops. So I've spent a lot of time talking to leftists and getting called corporate. God, that's funny. If you knew how much struggle my mom had getting me to put on a tie for my first job. Oh, khakis. Anyways, deeply funny to me that I get called a corporate shill. Capitalism is not a super well-defined term of... Like... So the Wikipedia page is like, here's nine things or 19 things. And if you have about half of them, we feel good calling you capitalist. But also people just use it to mean a market economy or they mean it to mean economy I don't like. And without getting into what is the single morally correct and effective economic system for everyone in the world, I think we could do better on terms. So you'd define it as where the surplus or profit goes to the capital owners and not the workers. Yeah.

02:05:30dpk0 well, i would define it as where surplus goes to capital owners and not to the workers who generated it
Yeah. Cause even the folks who talk about being anti-capitalist and I've talked to a lot of these folks, some of them, like they would. dpk0 but yeah, it’s basically used to mean ‘thing about economy i don’t like’
really object if you got rid of customer sovereignty. Some of them wouldn't, which is a little wild to me as an American, but, or a voluntary exchange commodification. It's funny commodification. It is so hard to take criticisms of that seriously. dpk0 yeah, it’s not a very helpful situation
I'm getting into political statements, but the short version is like, I have seen people rant about commodification. And at the same time, they very much enjoy the product of commodification in ways that are, it's so ubiquitous, it's hard to notice. I don't know. I'm kind of doing that comic. What is it?

02:06:32Ah, that's not it. Where is this comic?

...40All there are is edits of this comic. I can't find the actual... Oh, it must be this one. I am aware I'm kind of doing this comic. I get a big kick out of this one.

...58So the person is unhappy that their AirPods are expensive, but the workers don't get it. And someone pops up to kind of gotcha them of like, oh, but you participate. That's kind of what I'm doing when I gripe about people who are complaining about commodification.

02:07:18Which is to say. This is why I try not to get into, well, one of the reasons I try not to get into politics too hard. pushcx https://thenib.com/mister-gotch…
So we've touched on a lot of politically adjacent and economic topics because they are, but they are very much the same thing. But that's around where I am. Oh, I think I am. All right. So. dpk0 yeah. very tiring argument
dpk0 nah, i don’t think you are
I guess my politics are, I'm okay saying that they're fairly pragmatic of, I am trying to do a thing here by starting a workaround co-op that will improve the chances of success for the business, improve the benefits to the customers of that business, and especially improve things for the employees of that business. Because I generally think the world could work better, right? That's not an uncommon political or economic sentiment. And even if I am not storming La Bastille or, I don't know, calling out the Pinkertons just to pick options from different sides of the aisle, this is the thing I'm doing. And, oh, one more note about co-ops is... Businesses have articles of organization. These explain the capital and equity structure. In a co-op, that's often called the articles of agreement. It's basically the same thing. It tends to be a little bit less boilerplatey. than articles of organization are. At this point, articles of organization tend to be, and I say this having seen many of them. So I used to work on Stripe Atlas, which is a tool for starting Delaware C Corps and LLCs. So I have seen a lot of articles of incorporation. both inside of that project and outside, and they are almost pro forma at this point because people write really general versions that leave themselves with maximum flexibility while meeting their legal requirements. Co-op ones tend to be longer and more customized And because there is so much more, I don't want to say flexibility, there's just more experimentation happening. You can even run LLCs and C-Corps in a co-op fashion. There's a term for that I'm blanking on. I think people just say run as a cooperative or run to cooperative principles. Oh, that's very efficient that England and Wales do exactly that sort of thing. Yeah. Here in the States, I liked Atlas and I formed a company with Atlas even before I worked on it. So I feel safe saying that they have a nice articles of incorporation if you want. I have not seen a great articles of dpk0 when you found a company in England and Wales you get the Articles of Association that is literally written as a default in a schedule to the Companies Act 2006 unless you explicitly declare otherwise
agreement for the u.s i am going to post the recheck co-op docs publicly and talk this has kind of been a draft of what i will be so i have a recheck.dev i also have recheck.coop i don't know if i even have can i set up a redirect nope I'll do that sometime soon.

02:11:52if this stuff is interesting there is also a oh it's a ethan marcotte yeah there is a book called you deserve a tech union by ethan marcotte but it's about forming a union at your tech company and does it make sense for highly paid employees like developers and op folks and data scientists to organize as a union. Isn't that more of a... God, I hate the euphemisms, low class or working class thing. You are working class if you don't own the business. Anyways, wouldn't the world be better if everyone used my particular interpretations of incredibly contentious political terms? I think so.

02:12:47But if you are at an existing company and the employer doesn't want to sell it to you, there's a really interesting recent example. Where is it?

02:13:04I didn't say I'm not left wing. I didn't say I am left wing. I said... You know if the two hard things are off by one errors and naming things wouldn't all of politics be improved if people use my preferred definitions of things. So testable from the rails world testable is a really influential consultancy that has done a ton of rails work and. This is fairly recent. I don't see a date on top of it. I don't see a date on bottom of it. dpk0 ‘you are working class if you don’t own the business’ – and you say you’re not left wing? :D
God, I hate corporate blogs sometimes. I want to say this is a year, maybe two years ago. They have transitioned to making their business entirely employee-owned with an employee stock ownership plan. So for existing businesses, there are lots of, the general term is co-op transition. And I don't know a heck of a lot of it because I am starting de novo, but it is totally possible to transition an existing business over to being a co-op. And I am not sure that this would be called a co-op. I don't know. I don't think they're switching to a one member, one vote kind of thing. pushcx https://testdouble.com/news-and…
So they are technically, employee-owned but not a co-op, it kind of gets hair-splitty. Which, if I would like to demonstrate that I'm not left-wing, should I criticize the left for being tenditious about building circular firing squads, like whether an employee-owned company counts as a cooperative and thus is morally pure? Hmm.

02:15:12yeah yeah people's front of Judea verse today and people's front God I love that sketch that. I did see a little bit of that at the worker co op conference where. And it's really hard to tell with smaller social organizations, how much of that is. ideological commitment versus how much of it is, it's just a small community and people have personal history that is expressed with political rhetoric, but is really about like you borrowed my car 10 years ago and you gave it back with an empty tank of gas. I don't know. dpk0 we’re the people’s front of judea, etc.
I hope this doesn't come off as me dunking on them. I am more trying to get across. I think people are people, whatever the situation. Anyway, DPK, I hope that answers all your questions about co-ops. It sounds like you definitely have a running start already. Do you own or work for a co-op? Are you in the process of? I think they're just such a nice fit for capital-efficient businesses like software, SaaS. You'd like to? All right, well, me send a sell a couple hundred licenses of recheck pro and then i'll hire you ah if you founded a company it would probably be a co-op so i will mention because i believe from previous discussions that and i think i follow you on mastodon and blue sky now i know for sure mastodon dpk0 alas, no, but i’d like to!
dpk0 if i founded a company it would probably also be a co-op!
The, I have read a ton of stuff about entrepreneurship from, ah, all right. So I've read a ton of stuff about entrepreneurship from being involved with, I mean, from personal research, but also being involved with various entrepreneurship communities offline and online. nogweii I know python and javascript quite well :P
pushcx https://stackingthebricks.com/
dpk0 i’m not on Bluesky. well, i have an account, but it only tells people to go to my Mastodon account
and there are a million books that will tell you how to start a business or a million courses oh my god and some of them charge staggering amounts of money if you are a technical person so especially developers the one i would recommend is stacking the bricks the one if you had to start from nothing and you only got one i i have bought yeah i should do a full disclosure there's obviously no affiliate marketing here i am in no way involved with this business i get nothing from it i think i've bought most or all of their ebooks and one of their courses over the year ah dpk that is a long history yeah i recommend this blog and then in their products Is this just gonna be the eBooks or is this also gonna be the course? Ah, these are just the... Dang it. Well, if you get on their newsletter, they will pitch you... His name is so bad. They say so. dpk0 i’m a third generation co-operative enthusiast. my grandpa was a conservative who became a socialist after 1929, i was a conservative who became a socialist after 2008
Their occasional course is called 30 by 500, which is to say, if you are a developer and you can get 30 people paying you, 30 businesses paying you 500 bucks a month, or 500 businesses paying you 30 bucks a month, you can replace your income and go full time. And it is a repeatable, plausible process for building a bootstrapped business. I have seen it work many, many times, and it is a very particular one. It is not, I have a bright idea. How do I market it? It is, yeah, let me, pushcx https://30x500.com/academy/
this in the notes because i've said this a couple of times but i've never remembered to write it down the core idea is if you have a professional skill that someone pays you to do you can pay yourself to do it by starting a business by doing a course by xyz dpk0 huh, nice
there's a lot of great free info in the blog. This is the only entrepreneurship course I've seen and liked. dpk0 i’ll take a look, these look really interesting
And then the other part of it that I was going to say it is, this is not a general, how do I get people to buy my thing course. because that doesn't work. 30X500 is how do I use my skill and interest to understand customers well enough to develop and market a product they'll pay for? And I think Amy and Alex would be satisfied with that description so it is about taking a skill an actual honest to god skill you have and if you have a job you have a skill someone will pay you for and interest this should be something that you can stand to look at for more than a few hours the most important thing is understanding actual customers it is not enough to build it and hope they'll come Or you could sometimes get there by scratching your own itch, by building the thing you need for yourself, but you will have much better odds if you understand how to read customers and develop the thing that they are already telling you they want to pay for. And when I say pay for, If you make a wonderful product for people who don't have any money, you have made something wonderful, but you have not made a business. And if you make a wonderful product that your customer doesn't understand is wonderful, you have not made a successful business. And if they look at it and go, yeah, but it's just not worth it. You have not made a successful business. And so it's all of those components of as a technical person, how do you turn your skill into a working business? And that can be like stair step of you start with an ebook and then you make a course or you do some consulting. And then eventually you work your way into something that looks more like a product, like a one-time use software, like productized consulting where there is one fixed price for it and very little negotiation and you're not tracking hours. Or it can be a product like subscribe to my SaaS. And I really like 30 by 500. don't love the name and because it teaches or do they speak the language of developers and just nerds in general i have seen a lot of folks succeed with it if you like julia evans stuff so she's ah the cat is on the mouse now thank you sir

02:23:45So Julia Evans writes a wonderful tech blog with wonderful explanations of things and sells zines also as a former Stripe. She has talked publicly about how she went to 35. We're also long enough into the stream that I'm just going to keep tripping over my words, but She also attended at least one 30 by 500 course, and that is part of why she writes the way she does and writes some of these products. She is a successful entrepreneur because of it. And I don't know her finances, and I wouldn't share them if I did, but I believe she does pretty well for herself, even as somebody who used to work as a very highly paid developer at Stripe. And that's not any personal knowledge. I just know at the time she worked there that Stripe was very happily paying well. There's another one. Speaking of resources I mention all the time, levels.fyi is wonderful for American developers who don't know what they're worth. If... Sight's getting a little spammy. Or a little engagement-baity. Come here. That's not what I want to block. Try again. I hate these floating garbage. Oh, there's more floating garbage. Whatever. But I mentioned earlier in the stream that the... Fully loaded cost of a senior software developer at the 90th percentile in the US is something like a million dollars. And that may be a very surprising number. I mean, especially outside of the US where salaries are typically lower, but that's very much something like 200 to $250,000 in salary directly, the same or more in equity. So that can be RSUs, ISOs, there's a whole alphabet soup of options for equity. And then benefits on top of it. And in the US, health insurance is the big benefit that's very expensive for employers. If you are an employee and you are unhappy about how much gets taken out of your paycheck for healthcare, know that it is at least that expensive to your employer. And especially in high class professions, not just high paid, but high class professions, your employer may cover nearly all of it. Yes, feel free to email me if there's anything you would like to discuss, or I think you're also on IRC. You're welcome to drop me a note if there is any of that you'd like to follow up on. Anyways, you should know where salary info is. That's why I keep pointing people to this site. And when I say 90th percentile, that does mean like, yeah, only 10% of developers are getting paid like this. dpk0 it would be inadvisable of me to comment on this discussion, considering my current employment situation and that this chat is publically archived
But hey, you should be one of them if you are one of my many experienced, handsome, tall, wonderful viewers and subscribers. running out of steam all right so i will finish up with a last call if there are any other questions anybody has about recheck lobsters which is the site the community the code base i just left a comment about the difference between the code base and the community maybe that's worth talking about a second

02:28:01There was a meta thread where someone said as a feature request... Ah, well, dpk, feel free to message me anytime. And if you have to drop, have a good evening. Someone asked, why can't we submit Magnet and IPFS URLs? It was very funny reading this because this morning I got a phishing email that was using his same favorite gateway to IPFS. And I gave a fairly long answer, which was on a technical level, this is a very small change. dpk0 understandable. crap, it’s 10:30pm here. fortunately i work afternoons, not mornings, on Tuesdays …
I mean, I could do it on stream and even coding very slowly. Cause I slow down a lot on stream when I'm running my monologue. It's like a 15 minute, half hour kind of job. dpk0 oh yeah this dude
I could almost name the places to touch the code off the top of my head, but

02:29:07pushcx https://lobste.rs/s/liwv8u/why_…
It did irk me a little. I tried to be super polite, but he said, HTTP clients are needed to access lobsters, but simply because lobsters is a centralized aggregation service, which is free to pick the protocol it wants to sync readers. And I think that this is a correct but wrong way of looking at it. Because the idea that we are a centralized aggregation service is a correct description of what the code base does. Yes, it is 10,000 lines of Ruby and a MariaDB database and a couple of virtual private servers, and it is a centralized aggregation service. Yeah. I'm going to make the title recheck demo, though. That is a good title. Maybe I should make, I've never put like a slash title in it. But lobsters is a community. It is not useful to think of it in the technical term as a centralized aggregation service. dpk0 stream title: a correct but wrong description
Because you might do things like allow the database, allow the users to submit what are basically broken links that won't work for people. And when the links don't work, the conversation doesn't work. And when we have a bunch of conversations that are all broken, we get bad vibes. And I realize I am being nonspecific when I say things like bad vibes, but There is something magical that happens in communities where they reach a critical mass to get people interested to join and participate where you look at it and you think there's a lot happening there and I want to be part of it. And when half the links are broken and half the links don't have conversation or they just have re-litigating flame wars about hobby horses it's bad vibes it's the feeling that wow i don't want to deal with that so i think maybe this is a little half-assed but this difference between the code base and the community is a very, very important one. And when I talk about things, I try and mention that people can ask about the code base, the community, the site, because those are slightly different aspects of one thing. And if you only look at it through one lens, you won't understand what's happening. And maybe there is a metaphorical connection here to business is not an equity model. business is not an articles of incorporation or an articles of agreement if it's a co op, it is also the community of customers and employees and the way those overlap and treat each other, hopefully for the better synergistically if I can use my 90s business terms. I think what's really interesting about both programming and entrepreneurship is. you have to, you benefit greatly, maybe you don't have to, but you benefit greatly from being able to see problems through these many different lenses and at these multiple layers of abstraction.

02:33:17That's a fun thought to end on. So before I put my foot in my mouth any more than I already have, I am going to say thanks for tuning in, folks. I appreciate you taking a look at the demo. I have seen little email notifications. They pop up over inside of my browser because I didn't remember to close them before the stream. So I have seen a couple of folks join the recheck mailing list. I really appreciate it. And if you are willing to volunteer as a human sacrifice or as we call them in the business beta user i would really appreciate that in any case i am going to wind down take off get myself some dinner hope you all have a good rest of your day whatever your time zone and join me again on thursday where i will show off the end of the active storage avatar code which has been on a couple of streams and let me if i search for it and you see me searching for it you're gonna know the punch line clearly one of the one of the themes of this is i start telling a story and then people jump ahead and they're like oh you're going to talk about this arh did it calvin did it earlier today So next week, I really think the title is going to be Active Storage, The Rest of the Owl. dpk0 alas, i have no Ruby codebase for it
Because the Active Storage readme talks about user avatars as an example. And that's kind of step one of drawing the owl. And all of the stuff I've had to do to actually implement avatars for users on lobsters is the rest of the fucking owl. dpk0 if i’m still at current $ork when you add the Java version i might lobby for us to try it :-) that would get you in a major FOSS project
so on that meme note take care folks ah dbk thank you very much i can offer you many kudos and i don't know a discounted year of pro i wonder about java that'll be hopefully late next year i've it really depends on how quick i can build up engagement because If we get to the point that I can pay contractors or hire employees to start working on other versions, things can go very fast. We'll see. All right. I'm going to go click stop streaming. Thanks for tuning in. Take care.