NixOS on prgmr and Failing to Learn Nix
Code: , ,

This is a writeup of my notes on how to get NixOS running on a VPS at prgmr, followed by more general notes on this experiment in learning nix.


I went with the lowest tier, currently 1.25 GiB RAM, 15 GiB Disk for $5/month. I’m only running weechat for irc/twitter/fediverse/slack and some miscellaneous small things. For “pre-installed distribution” I chose “None (HVM)”.

Netboot to start install

I ssh’d into the management console, ssh [hostname]@[hostname]


Surprisingly, the included 1.25 GB of RAM was not enough to run some nix commands. I had to back up and recreate the box with some swap space. I didn’t think too hard about it, just guessed at 2 GB and it worked OK. 2018-07-09: Vaibhav Sagar suggested this is probably this known bug.

gdisk /dev/xvda
  o to create gpt
  n to create swap partition
    Command (? for help): n
    Partition number (1-128, default 1): 1
    First sector (34-31457246, default = 2048) or {+-}size{KMGTP}:
    Last sector (2048-31457246, default = 31457246) or {+-}size{KMGTP}: +32M
    Current type is 'Linux filesystem'
    Hex code or GUID (L to show codes, Enter = 8300): EF02
    Changed type of partition to 'BIOS boot partition'
    Command (? for help): n
    Partition number (2-128, default 2):
    First sector (34-31457246, default = 67584) or {+-}size{KMGTP}:
    Last sector (67584-31457246, default = 31457246) or {+-}size{KMGTP}: -2G
    Current type is 'Linux filesystem'
    Hex code or GUID (L to show codes, Enter = 8300):
    Changed type of partition to 'Linux filesystem'
    Command (? for help): n
    Partition number (3-128, default 3):
    First sector (34-31457246, default = 27262976) or {+-}size{KMGTP}:
    Last sector (27262976-31457246, default = 31457246) or {+-}size{KMGTP}:
    Current type is 'Linux filesystem'
    Hex code or GUID (L to show codes, Enter = 8300): 8200
    Changed type of partition to 'Linux swap'
  w to write and exit
mkswap -L swap /dev/xvda3
swapon /dev/xvda3
mkfs.ext4 -L root /dev/xvda2
mount /dev/xvda2 /mnt

Configuring nix

I generated the initial config and added a few prgmr-specific tweaks:

nixos-generate-config --root /mnt
cd /mnt/etc/nixos
vi configuration.nix

Here’s my tweaks:

  boot.loader.grub.device = "/dev/xvda"
  # prgmr console config:
  boot.loader.grub.extraConfig = "serial --unit=0 --speed=115200 ; terminal_input serial console ; terminal_output serial console";
  boot.kernelParams = ["console=ttyS0"];
  environment.systemPackages = with pkgs; [
  services.openssh.enable = true;
  networking.firewall.allowedTCPPorts = [ 22 ];
  sound.enable = false;
  services.xserver.enable = false;
  services.openssh.enable = true;
  users.extraUsers.pushcx = {
    name = "pushcx";
    isNormalUser = true;
    extraGroups = [ "wheel" "disk" "systemd-journal" ];
    uid = 1000;
    openssh.authorizedKeys.keys = [ "[ssh public key here]" ];

Then I ran nixos-install to install the system.


The NixOS manual says you should be able to run reboot to boot to the new system, but something in xen doesn’t reload the new boot code and I got the netboot again rather than the new system. After talking to prgmr I found it worked if I pulled up the management console and did:

After this I had a running system that I could ssh into as a regular user.

Prgmr donates hosting to Lobsters, but because Alan configured the hosting, this was my first time really using the system. It was painless and getting support in #prgmr on Freenode was comfortable for me as a longtime IRC user. I liked them before, and now I’m happy to recommend them for no-nonsense VPS hosting.


I did this setup because I’ve been meaning to learn nix (the package manager) and NixOS (the Linux distribution built on nix) for a while. As I commented on Lobsters, they look like they didn’t start from manual configuration and automate that, they started from thinking hard about what system configuration is and encoded that. (The final impetus was that I ran out of stored credit at Digital Ocean, hit several billing bugs trying to pay them, and couldn’t contact support – six tries in four mediums only got roboresponses.)

The NixOS manual is solid and I had little trouble installing the OS. It did a great job of working through a practical installation while explaining the underlying concepts.

I then turned to the Nix manual to learn more about working with and creating packages and failed, even with help from the nixos IRC channel and issue tracker. I think the fundamental cause is that it wasn’t written for newbies to learn nix from; there’s a man-page like approach where it only makes sense if you already understand it.

Ultimately I was stopped because I needed to create a package for bitlbee-mastodon and weeslack. As is normal for a small distro, it hasn’t packaged these kind of uncommon things (or complex desktop stuff like Chrome2018-07-16: I’ve learned that Nix does have a package for Chrome, but it doesn’t appear in nix-env searches or the official package list because it’s hidden by an option that is not referenced in system config files, user config files, the NixOS Manual, the Nix Manual, the man page for nix-env, the package search site, or the the documentation of any other tool it hides packages from.) but I got the impression the selection grows daily. I didn’t want to install them manually (which I doubt would really work on NixOS), I wanted an exercise to learn packaging so I could package my own software and run NixOS on servers (the recent issues/PRs/commits on lobsters-ansible tell the tale of my escalating frustration at its design limitations).

The manual’s instructions to build and package GNU’s “hello world” binary don’t actually work (gory details there). I got the strong impression that no one has ever sat down to watch a newbie work through this doc and see where they get confused; not only do fundamentals go unexplained and the samples not work, there’s no discussion of common errors. Frustratingly, it also conflates building a package with contributing to nixpkgs, the official NixOS package repository.

Either this is a fundamental confusion in nix documentation or there’s some undocumented assumption about what tools go where that I never understood. As an example, I tried to run nix-shell (which I think is the standard tool for debugging builds but it has expert-only docs) and it was described over in the Nixpkgs Manual even though it’s for all packaging issues. To use the shell I have to understand “phases”, but some of the ones listed simply don’t exist in the shell environment. I can’t guess if this a bug, out-dated docs, or incomplete docs. And that’s before I got to confusing “you just have to know it” issues like the src attribute becoming unpackPhase rather than srcPhase, or “learn from bitter experience” issues like nix-shell polluting the working directory and carrying state between build attempts. (This is where I gave up.)

I don’t know how the NixOS Manual turned out so well; the rest of the docs have this fractal issue where, at every level of detail, every part of the system is incompletely or incorrectly described somewhere other than expected. I backed up and reread the homepages and about pages to make sure I didn’t miss a tutorial or other introduction that might have helped make sense of this, but found nothing besides these manuals. If I sound bewildered and frustrated, then I’ve accurately conveyed the experience. I gave up trying to learn nix, even though it still looks like the only packaging/deployment system with the right perspective on the problems.

I’d chalk it up to nix being young, but there’s some oddities that look like legacy issues. For example, commands vary: it’s nix-env -i to install a package, but nix-channel only has long options like --add, and nix-rebuild switch uses the more modern “subcommand” style. With no coherent style, you have to memorize which commands use which syntax – again, one of those things newbies stumble on but experts don’t notice and may not even recognize as a problem.

Finally, there’s two closely-related issues in nix that look like misdesigns, or at least badly-missed opportunities. I don’t have a lot of confidence in these because, as recounted, I was unable to learn to use nix. Mostly these are based on my 20 years of administrating Linux systems, especially the provisioning and devops work I’ve done with Chef, Puppet, Ansible, Capistrano, and scripting that I’ve done in the last 10. Experience has led me to think that the hard parts of deployment and provisioning boil down to a running system being like a running program making heavy use of mutable global variables (eg. the filesystem): the pain comes from unmanaged changes and surprisingly complex moving parts.

The first issue is that Nix templatizes config files. There’s an example in my configuration.nix notes above: rather than editing the grub config file, the system lifts copies from this config file to paste into a template of of grub’s config file that must be hidden away somewhere. So now instead of just knowing grub’s config, you have to know it plus what interface the packager decided to design on top of it by reading the package source (and I had to google to find that). There’s warts like extraConfig that throw up their hands at the inevitable uncaptured complexity and offer a interface to inject arbitrary text into the config. I hope “inject” puts you in a better frame of mind than “interface”: this is stringly-typed text interpolation and a typo in the value means an error from grub rather than nix. This whole thing must be a ton of extra work for packagers, and if there’s a benefit over vi /etc/default/grub it’s not apparent (maybe in provisioning, though I never got to nixops).

This whole system is both complex and incomplete, and it would evaporate if nix configured packages by providing a default config file in a package with a command to pull it into /etc/nix or /etc/nixos for you to edit and nix to copy back into the running system when you upgrade or switch. This would lend itself very well to keeping the system config under version control, which is never suggested in the manual and doesn’t seem to be integrated at any level of the tooling – itself a puzzling omission, given the emphasis on repeatability.

Second, to support this complexity, they developed their own programming language. (My best guess – I don’t actually know which is the chicken and which is the egg.) A nix config file isn’t data, it’s a turning-complete language with conditionals, loops, closures, scoping, etc. Again, this must have been a ton of work to implement and a young, small-team programming language has all the obvious issues like no debugger, confusing un-googleable error messages that don’t list filenames and line numbers, etc.; and then there’s the learning costs to users. Weirdly for a system inspired by functional programming, it’s dynamically typed, so it feels very much like the featureset and limited tooling/community of JavaScript circa 1998. In contrast to JavaScript, the nix programming language is only used by one project, so it’s unlikely to see anything like the improvements in JS in last 20 years. And while JavaScript would be an improvement over inventing a language, using Racket or Haskell to create a DSL would be a big improvement.

These are two apparent missed opportunities, not fatal flaws. Again, I wasn’t able to learn nix to the level that I understand how and why it was designed this way, so I’m not setting forth a strongly-held opinion. They’re really strange, expensive decisions that I don’t see a compelling reason for, and they look like they’d be difficult to change. Probably they have already been beaten to death on a mailing list somewhere, but I’m too frustrated by how much time I’ve wasted to go looking.

I’ve scheduled a calendar reminder for a year from now to see if the manual improves or if Luc Perkins’s book is out.

House Rules
Games: , ,
1 comment

I really enjoy playing board games with friends, as you can probably guess from my media reviews. Over the last ~25 years of playing we’ve evolved a couple house rules that are worth formalizing and sharing.

1. Yes Take-backsies

We’re playing for the fun of learning new games and competing. If you make a mistake and we can unwind it, you can take it back and do what you meant to do or realize you should have done.

Though we don’t do take-backsies of things we can’t easily and fairly unwind. For example, in Risk, if you roll the dice to attack another player’s army and get wiped out, well, that’s not fair to unwind. Similarly, if you learn hidden information like turning over the next card on the deck, we can’t make everyone forget that (though this can bend this for someone’s first time playing a game).

Making mistakes is a normal part of learning. By helping keep them cheap and emphasizing that it’s socially rewarded to admit and correct them, everyone learns and plays better, and gets to relax and enjoy themself more. Games are fun because they’re an exercise in trust as much as exercise in formal systems.

We very often have players with uneven experience and take-backsies helps the newbies get into a game and keep the moderately-experienced competitive with the experts. We encourage suggestions and polite criticism of in-progress mistakes, too. When learning a complex game it’s hard to recognize the legal moves available and what their tradeoffs might be. So when someone looks particularly stumped, it’s normal to hear something like “Sooo… it looks like you have five or maybe six things you could do here, depending on what’s in your hand?” to offer help.

Suggestions show up even with competitive, experienced players, in part because some of us are so competitive we wrap around into helping our opponents make the best moves for their strategies so that we’re ourselves pushed into improving our own play. Suggestions aren’t a rule because some people don’t like hearing them and it’s more common in competitive play to want opponents to make mistakes, but I wanted to go into it because it’s foundational to the Yes Take-Backsies rule.

2. Public Stays Public

Information that’s revealed during gameplay stays public information and can be reviewed at any time unless doing so severely inconveniences the flow of gameplay.

What this means in practice is cards get discarded face-up and spread out for anyone to look through, or other things that have been played can be reviewed. The limit is that we might run out of table space or it would be distinctly un-fun to dig through that many cards, but when there’s that much information laying around it’s probably not particularly important what’s been played.

For example, The Great Dalmuti is a light trick-taking card game that’s a longtime favorite. Players play cards to try to empty their hands before everyone else, and between hands they shift to sit in the order they finished in to get benefits for the next hand. The last player is punished with the chore of clearing away the cards from each trick (and they have some other chores) so we task the next-to-last player. Instead of the game rule that all played cards are flipped face-down, our house rule is to arrange the best six ranks (numbered 1-6 of the deck’s 12 ranks) face-up along the side of the table. We don’t do this for ranks 7+ to balance the work vs how little strategy is at work in high cards (>85% of the time the correct play is “dump any of them at first opportunity”). This has been a big success. Beginners graduate from learning the rules to start exercising strategy in 2-3 hands instead of 5-6 hands, experts can experiment more effectively, and everyone’s happier not trying to remember “wait… did I see all of the threes, or just two of them?”

Information that’s derivable from public information is also public. In Acquire every player starts with the same amount of money and buys and sells stock in public transactions… but the rulebook suggests keeping it secret to make the game “even more challenging”. Arithmetic is not a fun game, it is a chore.

These “memory subgames” crop up all over, are stressful and uninteresting, and seem an unfair advantage for those who are better at this or spend a few days learning mnemonic techniques. When public information stays public, players make fewer uninformed and mistaken decisions.

The common downside is that sometimes players bog down the game while sifting through old cards or falling into analysis paralysis. We solve this through good-natured grumbling and some smoothly-worn old jokes that reference getting on with things before the mountains crumble into the seas, etc. In rare cases (or rare players) we’ll set a turn timer on someone’s phone. Or give suggestions! Once you’ve looked at all the things an opponent might do and thought through how you’d respond to them and gotten bored, there’s no harm in talking to the opponent about their options.

To keep things moving along (especially when playing someone with very good recall of previous game state), it’s common to ask something like, “Wait, have you played all your aces?” rather than spend twenty seconds flipping through their discards. An honesty norm has developed: you can answer honestly, you can fib and say “I’m not sure” (if you think it gives you a competitive advantage and they won’t check), in an fiercely competitive game you can say “count for yourself”, but you can’t lie. If you’ve played all your aces but say you haven’t, or vice-versa, or otherwise deliberately give false information, it’s considered very rude, unsportsmanlike conduct and is treated almost as negatively as cheating. It’s OK to want to win and normal not to want to help your opponent, but we’ve established lying about public information as something that gets you a lot of frowning friends. We’re good friends or becoming them, so social disapproval means this basically never happens. (Exception: Diplomacy is a blood sport.)

Public Stays Public is a much younger and more explicit rule than Yes Take-backsies, which grew out of long habit. We had only very small, limited experiments with it until about six months ago (December 2017), when I heard at Recurse Center that someone knew of a gaming club (maybe at MIT?) that had it as a universal house rule. I decided it was worth trying and we’ve enjoyed it in every game since.


Our process of developing house rules is a very much like developing traditions, we’re doing this slowly and sometimes only recognizing in retrospect that we have one because we learn another group doesn’t. I’m finally getting around to writing this up today because it came up in an online chat and my response kept getting longer and longer, so maybe I’ll add more in the future.

I suppose one thing worth noting is that in the last decade as games have generally gotten much better (and we’ve gotten more patient), we’ve become much more reluctant to add house rules to individual games. Usually something that seems weird and wrong is a corner of gameplay worth exploring and deliberating rather than something we feel we comfortable immediately trying to prune. We have almost no per-game house rules, really.

2018 Media Reviews
Life: , , , , ,
No comments

I’ve appreciated when people take the time to write reviews and highlight connections to other good works. This is an in-progress list that I’ll update over the course of the year. Previously: 2014 2015 2016 2017

Continue this post

2017 Media Reviews
Life: , , , ,
No comments

I’ve appreciated when people take the time to write reviews and highlight connections to other good works. This post was written in one go at the end of 2017. I didn’t keep good track of the games, so that’s spotty and may be updated in 2018 as I’m reminded of them, but is otherwise complete. Previously: 2014 2015 2016

Continue this post

Return Statement
Life: ,
No comments

I’ve finished my time at Recurse Center, and a small tradition is to write a “return statement” about what you did in your time there.

Continue this post

Life: ,
No comments

Robots and elderly relatives have finally driven me to extreme measures. My voicemail greeting is now this “you have reached a number that has been disconnected” recording repeated for sixty seconds as a multi-pronged attack on voicemail:

Continue this post

Life: , ,
1 comment

I’m now the sysop of Lobsters, a social news site focused on technology.

Continue this post

Attending Recurse Center
Life: ,

I have been accepted to the Recurse Center to spend three months on collaborative, self-directed study of programming. I’m planning to continue studying Haskell and dependent types, proof assistants, and category theory. Maybe Coq, Idris, or TLA+ if I can find someone else interested.

Continue this post

Code: , ,
1 comment

I have some random episodes of podcasts laying around waiting to get listened to from podcasts I don’t (yet) care to subscribe to. Maybe they had an interesting guest or topic, or came recommended. These downloads will lay around on my computer for months because they’re not in my podcasting app, so they’re not really in my listening queue.

Continue this post

Arithmetic Wrap in GMS2
Code: ,
No comments

I’m learning GameMaker Studio 2 because my 10-year old nephew wants to make video games (and the 10 year old inside of me wants to make video games, too). It’s a nice toolkit and IDE for games, very beginner-friendly, with a friendly community. It’s even been used in some highly polished and popular games. If you’re curious, there’s a ~90 minute tutorial playlist that’s easy to skim as a demo.

Continue this post