Queue Zero
Life: , , ,
No comments

Almost exactly a year ago, I posted about Sizing Up My Queue to count up how much video and audio I had downloaded to watch.
The final tally?

queue: 877 files, 674007 seconds = 7d:19h:13m:27s total duration

I’ve been running that script almost every day, and for the first time it said:

queue: 0 files, 0 seconds = 0d:00h:00m:00s total duration

I did track values over time, and after a lot of frustration LibreOffice permitted this hideous graph – the Y axis is how many days of media remain:

media-queue

  • Most of the early drop was me shrugging and saying “yeah, OK, I’m really not interested enough in that podcast to actually listen to it”
  • The half-day jump in December is when I fixed the script to include .mov files
  • Big gap and accumulation in March/April is when I was working on my talk and book
  • About half of the drop at the end was archiving a video site I finished scraping
  • I almost exclusively listen to podcasts when doing chores or playing video games, so I’d have hit zero a month earlier if I didn’t play ~65 hours of Crypt of the Necrodancer
  • The last file was the recording of my 2015 RailsConf talk – watching my own presentations really makes me squirm, though it’s invaluable for improving as a speaker

This was a fun little project.
There’s still a few thousand files in ~/queue.
It’s a bit of a junk drawer (games waiting for me to have Windows again, photos to file away, archived web pages), but the majority of it is books and papers.
I suppose next I could write a script to take the word count of epub/mobi/pdf/html files… it’d be a bit of fiddling running different commands to dump word counts from the various formats, but it could work.

Well, it could work in that, yes, I could technically write that script.
I’ve known for decades that my to-read list has been growing faster than I read.

Deleting Spam From sup Maildirs
Code: , , ,
No comments

A quirk of the sup email client is that it doesn’t sync back changes like deletes to mail sources. “Deleted” messages are only flagged and hidden from the user.

After a few hundred thousand spam messages, sup slows down a bit and I have to actually delete it. I forget how every time, so this is mostly a note to myself in the hopes that someone else needs to delete mail from maildirs managed by sup.

First, type L to search for spam and delete the spam. Use T to tag all messages and =d to delete all tagged messages, perhaps after using \ to filter your view to obvious spam (and use !! to load everything that matches the query). Then quit sup and run:

# update the email filenames in the maildir so deleted ("trashed") messages get the "T" flag
$ sup-sync-back-maildir -nu
# get rid of them
$ mkdir /tmp/deleted-spam
$ find ~/path/to/mail -type f -regex ".*,.*T" -exec mv {} /tmp/deleted-spam
# sync sup to know they're gone
$ sup-sync --all-sources -o

After checking that everything’s fine, delete /tmp/deleted-spam (or ignore it; the contents of /tmp are wiped every reboot).

Sizing Up My Queue
Code: , , ,
No comments

I have a folder named “queue” that I download podcasts, videos, and books to. It occurred to me that it hasn’t been empty for years. That’s OK, the world is a very interesting place, and I care that I produce things, not just consume them.

But I wondered how big the queue is. Not in terms of disk space, that doesn’t tell me much because the different encoding rates and formats result in very different file sizes. (The disk space command is df -h, if you need it.)

So I wrote a script. I hacked it all out for a first pass, then tidied it into functions. It depends on having ffmpeg installed, and here’s the source:

#!/bin/ruby
 
MEDIA_EXTENSIONS = ['*.mp3', '*.ogg', '*.ogv', '*.avi', '*.mp4', '*.webm', '*.flv', '*.mov', '*.mpeg', '*.mkv', '*.mov']
 
MINUTE = 60
HOUR = MINUTE * 60
DAY = HOUR * 24
 
def seconds_to_duration seconds
  days = seconds / DAY
  seconds -= days * DAY
 
  hours = seconds / HOUR
  seconds -= hours * HOUR
 
  minutes = seconds / MINUTE
  seconds -= minutes * MINUTE
 
  "%dd:%02dh:%02dm:%02ds" % [days, hours, minutes, seconds]
end
 
def duration_of_file filename
  raw_duration = `ffprobe "#{filename}" 2>&1 | grep Duration | cut -d' ' -f 4 | cut -d',' -f 1`
  h, m, s, x = raw_duration.split(/[:\.]/)
  h, m, s = h.to_i, m.to_i, s.to_i
  seconds = s + m * 60 + h * 60 * 60
  seconds
end
 
def duration_of_files path, spec
  spec.map! { |s| "**/**/#{s}" }
  previous = Dir.getwd
  Dir.chdir path
  count = 0
  total = 0
  Dir[*spec].each do |filename|
    count += 1
    seconds = duration_of_file filename
    total += seconds
  end
  Dir.chdir previous
  return count, total
end
 
def report_on path, spec
  count, total = duration_of_files path, spec
  puts "#{path}: #{count} files, #{total} seconds = #{seconds_to_duration total} total duration"
end
 
dirs = ARGV.empty? ? [Dir.pwd] : ARGV
dirs.each do |dir|
  report_on dir, MEDIA_EXTENSIONS
end

Running it on my queue directory took a minute or so, then reported:

queue: 877 files, 674007 seconds = 7d:19h:13m:27s total duration

Looks like I have a bit of material ahead of me…

Installing You Need a Budget 4 (YNAB) on Arch Linux
Code: , , ,
No comments

We only want to advertise YNAB for platforms that we can enthusiastically proclaim, “This will work great for you! We promise! If it doesn’t, we’ll work until it does.” The fact is, we can’t do that for Linux anymore.
YNAB & Linux, 2011

Life events have me experimenting with You Need a Budget, software for planning and tracking budgets. I like that it has an opinionated plan that balances prudent buffers and real-world complexity. Unfortunately YNAB 4 was built on Adobe AIR, which no longer runs on Linux.

It took a bit of tinkering to get it running in Wine, the program for running Windows apps on Linux, so I wanted to write up my notes to save anyone else some hassle.

YNAB on Linux

First, I installed wine and some needed libraries:

$ pacman -S wine 
$ pacman -S lib32-lcms lib32-lcms2
$ pacman -S samba wine-mono wine_gecko lib32-gnutls lib32-mpg123 lib32-ncurses

Then, I downloaded YNAB and installed it by running wine with a WINEARCH because I heard YNAB might be flaky in 64 bits (I didn’t try to check this) and WINEPREFIX to isolate it from any other wine installs:

$ WINEARCH=win32 WINEPREFIX=~/.wine-YNAB wine ~/Downloads/YNAB\ 4_4.3.656_Setup.exe

This installed fine, and I made a little shell script named ynab to run the app:

#!/bin/bash
 
WINEARCH=win32 WINEPREFIX=~/.wine-YNAB wine C:\\Program\ Files\\YNAB\ 4\\YNAB\ 4.exe

After I had YNAB running I noticed that tooltips were displayed as a series of boxes because of a missing font, which some searching said winetricks would install. I installed winetricks-git from the AUR and embarked on some yak-shaving.

At first whenever winetricks called wget to download the corefonts it failed with an error about not finding libgnutls.so.26. After installing libgnutls26 from AUR it was fine.

After that was fixed it happily downloaded files but failed to verify the downloads:

$ WINEPREFIX=/home/pushcx/.wine-YNAB WINEARCH=win32 winetricks corefonts
Executing w_do_call corefonts
Executing load_corefonts
Executing mkdir -p /home/pushcx/.cache/winetricks/corefonts
Downloading http://downloads.sourceforge.net/project/corefonts/the%20fonts/final/arial32.exe to /home/pushcx/.cache/winetricks/corefonts
--2015-04-30 20:32:59--  http://downloads.sourceforge.net/project/corefonts/the%20fonts/final/arial32.exe
Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 216.34.181.59
Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|216.34.181.59|:80... connected.
HTTP request sent, awaiting response... 307 Temporary Redirect
    # Try download twice
    checksum_ok=""
    tries=0 
Location: http://sourceforge.net/projects/corefonts/files/the%20fonts/final/arial32.exe/download?use_mirror=superb-dca2 [following]
--2015-04-30 20:32:59--  http://sourceforge.net/projects/corefonts/files/the%20fonts/final/arial32.exe/download?use_mirror=superb-dca2
Resolving sourceforge.net (sourceforge.net)... 216.34.181.60
Connecting to sourceforge.net (sourceforge.net)|216.34.181.60|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 14566 (14K) [text/html]
Saving to: ‘arial32.exe’
 
100%[==========================================================================================================================================================================>] 14,566      --.-K/s   in 0.02s   
 
2015-04-30 20:33:00 (591 KB/s) - ‘arial32.exe’ saved [14566/14566]
 
------------------------------------------------------
Checksum for /home/pushcx/.cache/winetricks/corefonts//home/pushcx/.cache/winetricks/corefonts/arial32.exe did not match, retrying download
------------------------------------------------------
mv: cannot stat/home/pushcx/.cache/winetricks/corefonts//home/pushcx/.cache/winetricks/corefonts/arial32.exe’: No such file or directory
Downloading http://downloads.sourceforge.net/project/corefonts/the%20fonts/final/arial32.exe to /home/pushcx/.cache/winetricks/corefonts
--2015-04-30 20:33:00--  http://downloads.sourceforge.net/project/corefonts/the%20fonts/final/arial32.exe
Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 216.34.181.59
Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|216.34.181.59|:80... connected.
HTTP request sent, awaiting response... 307 Temporary Redirect
Location: http://sourceforge.net/projects/corefonts/files/the%20fonts/final/arial32.exe/download?use_mirror=colocrossing [following]
--2015-04-30 20:33:00--  http://sourceforge.net/projects/corefonts/files/the%20fonts/final/arial32.exe/download?use_mirror=colocrossing
Resolving sourceforge.net (sourceforge.net)... 216.34.181.60
Connecting to sourceforge.net (sourceforge.net)|216.34.181.60|:80... connected.
HTTP request sent, awaiting response... 200 OK
 
    The file is already fully retrieved; nothing to do.
 
/sbin/winetricks: line 503: /home/pushcx/.cache/winetricks/corefonts//home/pushcx/.cache/winetricks/corefonts/arial32.exe: No such file or directory
------------------------------------------------------
sha1sum mismatch!  Rename /home/pushcx/.cache/winetricks/corefonts//home/pushcx/.cache/winetricks/corefonts/arial32.exe and try again.
------------------------------------------------------

I spent a while wondering why the filename at the end was doubled up, but that never actually mattered. When I looked at what had been downloaded, it was not a cab .exe (though it had that filename) but an HTML download page from Sourceforge. Sourceforge tweaked something that prevented the script from grabbing the correct file, presumably to show more ads on download pages.

Then I went to the corefonts download page and grabbed all the .exes. I tossed them into ~/.cache/winetricks/corefonts and running winetricks again installed them all. A quick rm -r ~/.cache/winetricks cleaned up the cache. To save future hassle, I put up a mirror on GitHub.

After that hassle, the ynab command brings up YNAB for hours of fun budgeting and reconciliation. Yay, adulthood!

Create localhost Aliases for Different Projects
Code: , , , ,
No comments

As a consultant I’m getting set up to develop on a new Rails project every few weeks or months. And I’ll jump back to an earlier project to answer questions, fix bugs, fix typos, etc. Eventually, something overlaps between them. I got bit by this again, so I wanted to write it up.

As is always the case, a bug came up on a project I’d finished a few weeks before. I did a git pull to make sure I was up-to-date, started it up again and… it blew up. I couldn’t load any pages at all, and it failed deep in ApplicationController of all places.

It didn’t take too long to run down the problem: the old project and my current project happened to set a cookie with the first name. The current project had discarded the cookie it didn’t recognize, but the old project wasn’t as robust and failed when it saw the cookie it considered invalid.

The problem is that everything is on the domain name localhost and the browser considers that to be one site. I had one really frustrating debugging session when the browser used its cache for a different version of jQuery from a different site I’d been working on earlier that day. I’ve also seen security software that refuses to run on localhost, sites with “service-oriented architecture” that need multiple web servers running, and I’m sure there’s more reasons I’m not thinking of.

So let’s use a different domain name for each project. It’s as easy as editing the /etc/hosts file. I like to make entries like this:

127.0.0.1 foo-dev.com # 2015-03-27 foo project for foo llc
127.0.0.1 blah-dev.org # 2015-02-01 blah project for blah

I don’t use fake domain top-level domains like “local.dev” because some security software (including FIDO U2F don’t permit unknown top-level domains. (And, worse, .dev is now Google’s private turf.)

I also don’t use subdomains like “local.foo.com” for two reasons. First, nothing stops the site itself from using that subdomain, so visiting the production site will now try to pull in assets or iframes from your local development copy. Second, cookies can behave in surprising ways when set or accessed on a subdomain vs. hostname.

In writing this up I’m reminded how many of the tiny development habits I have are about preventing some weird corner case that have cost me hours and hours of time, so I have to close with a famous quote:

…and of course you must be careful not to overwrite the bounds of memory blocks, free a memory block twice, forget to free a memory block, use a memory block after it’s been freed, use memory that you haven’t explicitly allocated, etc. We C++ programmers have developed tricks to help us deal with this sort of thing, in much the same way that people who suffer severe childhood trauma develop psychological mechanisms to insulate themselves from those experiences.

Joseph A. Knapka

Case Sensitivity in In-Page Anchors
Code: , , ,
No comments

As I’ve been working on Chibrary, I ran into a small cross-browser compatability issue: only some browsers treat anchors as case-sensitive. The call numbers that uniquely identify messages would be perfect for linking to in the middle of a long discussion, but some of them only differ by case.

So I wondered: is that acceptable in linking to an anchor inside the page? A quick experiment with Firefox worked fine, but I wanted to be thorough.

When the HTML5 spec talks about scrolling to a “fragment identifier” (aka “fragid” or “named anchor”) there’s mention that a special value of “top” is case-insensitive and otherwise:

  1. Let decoded fragid be the result of applying the UTF-8 decoder algorithm to fragid bytes. If the UTF-8 decoder emits a decoder error, abort the decoder and instead jump to the step labeled no decoded fragid.
  2. If there is an element in the DOM that has an ID exactly equal to decoded fragid, then the first such element in tree order is the indicated part of the document; stop the algorithm here.
  3. No decoded fragid: If there is an a element in the DOM that has a name attribute whose value is exactly equal to fragid (not decoded fragid), then the first such element in tree order is the indicated part of the document; stop the algorithm here.

So it’s not case-insensitive according to HTML5 (though it used to be in HTML 4.01).

The HTML5 spec goes on to reference RFC 3986 for what a URL is, and section 3.5 begins its definition:

The fragment identifier component of a URI allows indirect
identification of a secondary resource by reference to a primary
resource and additional identifying information. The identified
secondary resource may be some portion or subset of the primary
resource, some view on representations of the primary resource, or
some other resource defined or described by those representations. A
fragment identifier component is indicated by the presence of a
number sign (“#”) character and terminated by the end of the URI.

So the relevent specs have no mention of case-sensitivity. Still, being a paranoid developer, I created a test case. The page had enough lorem ipsum to fill a few screens, and two anchors named #ANCHOR and #anchor. They’re color-coded so I could asses at a glance, which I did by linking to the lower-case version and seeing if I ended-up at the earlier upper-case version.

Want to guess which one browser got it wrong?

Yep, Internet Explorer, versions 6 through 11. Here’s a screenshot of a current IE 11 on Windows 8.1 when linked to #anchor:

IE 11 on Win 8.1

I briefly considered base64 encoding the call numbers, but then I remembered my longstanding commitment to myself: never fix an IE-only bug unless required by a job. Internet Explorer has been broken in many ways for a long time, and often fails to correct bugs for backwards compatibility. As long as it’s going to keep wasting so much of my time working around it (…and for a good while after, should it ever stop), I’m not going to donate more time to it.

Dual-Booting Arch Linux on Lenovo X1 Carbon 3rd gen
Code: , , , , , , , ,
8 comments

I decided to replace my mid-2011 Macbook Air 3,2 with a non-Apple machine, but every laptop I looked at was unsuitable. Most were overpriced, with a big and clunky design. The Lenovo X1 Carbon was promising, but the 2nd generation had a keyboard that was just too weird (and the function keys changing modes means you can’t touch-type them anymore). Standard qwerty is bad, but it’s the devil I know.

That was a year ago. In January 2015 Lenovo released a third generation of the X1 Carbon with some slightly improved specs and a normal keyboard, so I picked one up. I decided to switch from Ubuntu because I’ve felt out-of-touch with how it works under the covers and I’ve been curious about Arch Linux. This post is my notes on setting that up.

2016-04-07: A year and three weeks after I started using this laptop (I know precisely because that put the laptop juuuust out of warranty), without warning or accident, the display stopped working. Lenovo’s expensive repair service estimated 6 businesses days for repair but the repair took over 5 weeks. If you might purchase a Lenovo laptop (perhaps the recent X1 Carbon 4th gen), I suggest you consider whether randomly losing hundreds of dollars and access to your laptop for 5 weeks would inconvenience you.

2016-05-16: …and by “5 weeks” I mean “9 weeks”. When Lenovo sent my laptop back with a new motherboard, it was one model down with slower processor, half the RAM, etc. It took another several weeks of dealing with their incompetent bureaucracy before they replaced it.

2016-12-18: Well, shit, the display on the replacement has started flickering.

BIOS Settings

The out-of-the box settings were decent, but I made a few tweaks.

  • Config
    • Wake on LAN: Disabled. I don’t use it.
    • USB/Charge in Battery Mode: Enabled. I want to be able to charge my phone in my bag.
    • Power/Intel Rapid Start: Disabled. Insecure hibernation. See partitioning below.
    • Intel(R) AMT/Intel (R) AMT Control: Disabled. I do not have an enterprise IT department accessing my laptop.
  • Security
    • Security Chip/Security Chip: Disabled. A chip to prove to copyright owners that your computer is secure to them against you.
    • Anti-Theft/Computrace: Disabled. A kill switch owned by someone else.
    • Secure boot/Secure boot: Disabled while installing Linux, re-enabled after.
  • Startup
    • Option key Display: Disabled. Just looks nicer.

Partitioning

The laptop shipped with Windows 8.1 Standard installed. I wanted to keep it around for a few video games that don’t run on Linux, which meant resizing down its partition, removing the recovery partition, and removing a mysterious undeletable 7 GB “OEM partition”. The next paragraph is a rant and can be safely ignored.

Windows 8.1 is flippers-on-dry-land unusably bad. I have no idea how this shipped. I counted four GUI toolkit styles, inconsistent navigation, missing help files, privacy-invasive defaults, and settings broken up into several different screens in different programs that all look and work differently. Programs appeared and disappeared with little or no reason. Random DLL crashes on every boot. And even after tidying the many places it wastes disk space it needs 25 GB before any programs are installed. I can’t understand how people put up with this for daily use and spent an evening overwhelming frustrated after a couple hours of trial use. That the most common operating system looks and acts this way is a failure for my profession.

Removing the Windows recovery partition was easy. One of Windows’ several disk management programs slowly copied it to a 16 GB and removed the partition.

None of Windows disk management programs could explain or delete the “OEM partition”. This partition exists for the Intel Rapid Start feature, which (despite the name) exists to save battery life. IRS waits until the computer has been asleep for a while, copies RAM into this partition, and halts the computer. When the computer is started it takes a few seconds to restore RAM and the computer is usable normally. If this sounds an awful lot like the Windows/Linux “hibernate” feature, that’s because it’s the exact same thing implemented in the BIOS instead of the OS. And it saves your RAM unencrypted, so a thief can extract passwords, encryption keys, etc.

Even after turning off this misfeature in the BIOS, the Windows tools refuse to delete the partition, but if you can figure out how to start an admin console diskpart can remove the partition. I also jumped through some hoops to turn off swapping (the pagefile.sys) and hibernation (hiber.sys) and reclaim some disk space, but it’s too tedious to recount.

The Windows disk tools can enlarge an NTFS storage partition but could barely shrink it… it claimed the smallest the 230 GB partition could be reduced to was 111 GB.

But I had my Arch Linux bootable USB key in hand and was off to Linux to get this computer set up. A quick setfont sun12x22 made the terminal a lot more readable on the HiDPI 2560×1440 screen.

The ntfsresize program quickly prepared the the NTFS partition for resizing. (It would be better named “ntfscompact”, because it compacts the drive content in preparation for resizing; it does not change parition size on its own.) I used ntfsresize -i /dev/sda4 to examine it, ntfsresize -nv -b -s 40G /dev/sda4 to test, and the same command without the -n to do the job in a few seconds.

After shrinking it, I used gdisk to “resize” it by deleting the partition and creating a new partition that starts at the same sector, has the same 0700 “Microsoft basic data” partition type, but is only 40 GB large. I used the “i” command to print info about the partition, snapped a photo with my camera, and used that to replace the “Partition unique GUID”. I don’t know if Windows would care if it changed, but it was so easy to carry over I didn’t want to risk it.

Finally I added a partition filling the rest of the disk space with the type 8300 for Linux, wrote the table, and rebooted to see Windows still working (phew).

Linux Partitioning

I see a lot of client data and security keys as a consultant. Last year I worked on an app that did essential financial planning for a multi-billion dollar multinational company. If my laptop is lost or stolen, I don’t want that data (or my own personal data) easily readable.

Using the dm-crypt guides and Pavel Kogan’s notes on encrypting /boot, I set up full-disk encryption. GRUB is the only part of the Linux system unencrypted, which would be much more of a pain in the ass to compromise than an unencrypted /boot partition in the vanishingly-unlikely case that someone wanted to break the encryption. This is a nice little improvement for a little extra work, but I’m more concerned with protecting against opportunistic thieves than evil maids.

One major difference is that I didn’t mess around with LVM. I only want a single root partition filling my whole drive with /boot encrypted inside. I won’t be resizing or rearranging partitions, so there’s no reason to take on extra complexity for that unused optoin. With that in mind, and the Arch guides to device encryption, system configuration, and encrypting an entire system explaining all the details, here’s my command log:

# encrypt and format the partion
cryptsetup -c aes-xts-plain64 -s 512 luksFormat /dev/sda5
cryptsetup luksDump /dev/sda5
cryptsetup luksOpen /dev/sda5 root
mkfs.ext4 /dev/mapper/root
mount /dev/mapper/root /mnt

My hard drive setup is done until it’s time to finish configuring grub for the encrypted volume, so it’s on to the installation guide:

Bootstrapping Arch

Following the guide was easy:

wifi-menu # took a few tries to see my network
vi /etc/pacman.d/mirrorlist # commented everything out, uncommented a few US mirrors
pacstrap /mnt base
genfstab -L -p /mnt >> /mnt/etc/fstab
arch-chroot /mnt
echo [hostname] > /etc/hostname
ln -sf /usr/share/zoneinfo/US/Central /etc/localtime
vi /etc/locale.gen # search for _US, uncomment
locale-gen
echo en_US.UTF-8 > /etc/locale.conf
echo FONT=sun12x22 > /etc/vconsole.conf
cp /usr/lib/systemd/system/systemd-vconsole-setup.service /etc/systemd/system/systemd-vconsole-setup.service

I did have to edit /etc/fstab to change relatime to noatime on root and add the EFI and Windows partitions:

/dev/mapper/root / ext4 rw,noatime,data=ordered 0 1
efivars /sys/firmware/efi/efivars efivars defaults 0 0
/dev/sda2 /boot/efi vfat rw
/dev/disk/by-label/Windows8_OS /opt/Windows8_OS ntfs rw,noauto

I also had to edit that copied system-dvconsole-setup.service file to add After=systemd-udev-settle.service and Wants=systemd-udev-settle.service under the [Unit] heading to set the font after the graphics driver was loaded, as described here.

As prompted by Kogan’s encryption guide, I created the encryption keyfile and added it to LUKS:

dd bs=512 count=4 if=/dev/urandom of=/crypto_keyfile.bin
cryptsetup luksAddKey /dev/sda5 /crypto_keyfile.bin
mkinitcpio -p linux
passwd
pacman -S grub efibootmgr
vi /etc/mkinitcpio.conf
vi /etc/default/grub

With that first vi I added “encrypt” to HOOKS and “/crypto_keyfile.bin” to FILES in mkinitcpio.conf. With the second I added grub to add:

GRUB_ENABLE_CRYPTODISK=y
GRUB_CMDLINE_LINUX="cryptdevice=/dev/sda5:root"

before I run this (which needs to run every time grub’s config is tweaked):

grub-mkconfig -o /boot/grub/grub.cfg
grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=grub_uefi --recheck

Next, time to install some basic networking tools so I can install the rest of the system from inside itself rather than the bootable usb:

pacman -Syyu
pacman -S iw dialog net-tools pkgfile wpa_supplicant
pkgfile --update

Figuring out which packages I needed involved a lot of booting from the usb and running the following commands.

wifi-menu
cryptsetup luksOpen /dev/sda5 root
mount /dev/mapper/root /mnt
arch-chroot /mnt

Then I exited the arch-chroot and ran umount -R /mnt to see everything cleanly unmount.

Installing Arch

It’s finally time to configure wifi (replacing the network ssid and password in brackets) and connect:

# I had to bring the network up manually, first:
wpa_supplicant -B -D nl80211,wext -i wlp4s0 <(wpa_passphrase <essid> <passphrase>)
dhcpcd wlp4s0
 
# then save it
wifi-menu # saved as wlp4s0-<essid>
netctl enable wlp4s0-<essid>

I lost a lot of time trying to get netctl to work reliably with wpa_supplicant. Sometimes it just failed to work after a reboot. Disabling netctl in favor of NetworkManager (whether through nm-applet or nmcli) was a big win.

Next I need to shave a different herd of yaks. I want to use etckeeper to track /etc (this turned out to be absolutely vital as I struggled with netctl). Which means I need to build and install it from the Arch User Repository, which I’d like to do with aura, which means I need to install enough tools to build and install that, which means I need to create a user account (for makepkg to run)…

# add a user:
useradd pushcx
passwd pushcx
visudo # to append "pushcx ALL=(ALL) ALL"
# log out, log in as pushcx
 
# install aura:
curl https://aur.archlinux.org/packages/au/aura-bin/aura-bin.tar.gz -O
tar -xvf aura-bin.tar.gz
makepkg -s
sudo pacman -U aura-bin-1.3.0.2-1-x86_75.pkg.tar.xz
 
# install etckeeper:
cd
sudo aura -Ai etckeeper
sudo aura -Ap etckeeper
sudo aura -Ax etckeeper
 
# configure etckeeper:
sudo etckeeper init
sudo git config --global user.email "root@hostname"
sudo git config --global user.name "root"
sudo git commit -m "initial commit"
sudo git gc # saved 5m

Phew, not too bad a shave, and I’m starting to see the Arch philosophy at work. This feels a lot like Slackware with some build automation. I used Slackware for about six years until I realized I wasn’t learning anything new. Lately I’ve felt out of touch with how my Linux install works; I hope Arch is the right level of automation and configuration.

(Speaking of philosophy, even after reading the justification I don’t understand how Arch settled on systemd. It seems the antithesis of the Unix Way and the Arch Way. I’m glad to be learning it as the majority of distros are moving to it, but it’s a beast.)

Digging into the general recommendations page showcases what first caught my attention about Arch: comprehensive, decently-written documentation about all aspects of the system. I’ve already been using Arch’s wiki for tips on understanding and maintaining my Ubuntu system for a year or so.

Meanwhile, I’ll add some basic packages, get into configuring the GUI, and test all the hardware drivers.

GUI Configuration

timedatectl set-ntp true
# console essentials:
pacman -S openssh tree tmux vim
# x essentials:
pacman -S xorg-server xorg-xinit lightdm \
lightdm-gtk3-greeter awesome vicious xterm \
urxvt-unicode notification-daemon ttf-freefont \
ttf-dejavu xorg-xmodmap xorg-xrdb gnome-keyring
# drivers:
pacman -S libva-intel-driver xf86-video-intel xf86-input-evdev xf86-input-synaptics

To configure lightdm, I created a ~/.xprofile:

#!/bin/sh
 
eval $(/usr/bin/gnome-keyring-daemon --start --components=pkcs11,secrets,ssh)
export SSH_AUTH_SOCK
 
xrdb -merge $HOME/.Xdefaults &
gnome-settings-daemon &
nm-applet &

And I had startx‘s startup script ~/.xinitrc call that:

#!/bin/sh
 
[ -f ~/.xprofile ] && source ~/.xprofile
 
exec awesome

And added a ~/.Xresources for the HiDPI display:

xterm*utf8: 1
Xft.dpi: 180
Xft.autohint: 0
Xft.lcdfilter: lcddefault
Xft.hintstyle: hintfull
Xft.hinting: 1
Xft.antialias: 1
Xft.rgba: rgb

I configured /etc/lightdm/lightdm-gtk-greeter.conf similarly:

[greeter]
xft-antialias=true
xft-dpi=180
xft-hintstyle=hintfull
xft-rgba=rgb
hide-user-image=true

Starting X and poking around my virtual terminals, Xorg was on F1, F2-F6 were more consoles, and F7 displayed a horizontally flickering cursor. After seeing that flicker, switching to Xorg or the text terminals displayed the same horizontal flickering; it looked like it was trying to repeatedly draw the screen at several different horizontal offsets that moved in and out of phase. This seems to be a bug in the intel video driver I’ll need to report; it appears fairly regularly as I switch between console and video modes.

2015-03-15: This was fixed in the latest version of the xf86-video-intel driver. Thanks to Jeremy Fleischman for the heads up.

Powersaving

I installed the laptop mode tools and configured powersaving. After running a laptop with a 3.5 year old battery, I’m really delighted and impressed to see 6-8 hour battery times when I nudge the LCD brightness down a little (in X or by editing /sys/class/backlight/intel_backlight/brightness) and even more if I turn off the wifi.

pacman -S laptop-mode-tools acpi acpid ethtool wireless_tools
systemctl enable laptop-mode

/etc/laptop-mode/laptop-mode.conf only needed one or two tweaks, like LM_BATT_MAX_LOST_WORK_SECONDS=60. The acpi command line tool is handy for watching battery from the console (I was buddy breathing my AC adaptor during the install).

Bluetooth

Following the wiki instructions:

pacman -S bluez bluez-utils gnome-bluetooth
hciconfig hci0 up
bluetoothctl
[bluetooth]# devices
[bluetooth]# scan on
# ... wait a minute ...
[bluetooth]# scan off
[bluetooth]# agent on
[bluetooth]# pair F0<tab>
[bluetooth]# trust F0<tab>
[bluetooth]# connect F0<tab>

And to bring bluetooth online automatically after restarts, add /etc/udev/rules.d/10-local.rules:

# Set bluetooth power up
ACTION=="add", KERNEL=="hci0", RUN+="/usr/bin/hciconfig hci0 up"

This works fine, but I also did aura -A blueman for a nice X applet.

In Conclusion

Hope this helps if you’re setting up Linux on an X1 Carbon or just curious about what the steps of setting up Arch Linux look like. As expected, Arch is a whole lot of fiddly little configuration steps. They gave me the excuse I wanted to see current Linux software config and tinker, but it’s quite time-consuming.

2015-02-12: I’ve started a page on the Arch wiki about the X1 Carbon (3g). It contains more information than this about setting it up, including some known bug.

Personal Workflow
Life: , , , ,
5 comments

For about a year I’ve been using Trello, a free web app for organizing notes, to track my personal to-do lists across various projects. I’ve used it to create the Well-Sorted Version (which included repeatedly proofing 600 pages of gibberish) and update NearbyGamers from Rails 2.1 to 3.2.13 (while moving it from a VPS to Heroku and from MySQL to Postgres — a yak-shaving marathon) while staying on top of daily chores and other life maintenance. For the first time I feel reliably productive and in control of the overwhelming procrastination that’s kept me from from finishing these and many other projects for years.

Continue this post

Recreating My Firefox Profile
Code: , , ,
No comments

With the release of Firefox 5 a few days ago, I thought it was time to recreate my Firefox profile. You may not know what it is because you only have one: it’s the set of your add-ons, bookmarks, history, and every other kind of customization you can do to Firefox.

As a web developer, I have a half-dozen for testing reasons, but one that I use for all of my personal browsing and most of my work. Very infrequently, I’ve noticed that it sometimes would have errors on web pages that other Firefox profiles didn’t. After some thinking, I realized that I’ve been copying the same Firefox profile nearly ten years, copying it between computers as I upgrade. And all the while I’ve been tinkering with it, poking around in its config files, testing extensions, changing options in about:config.

I wanted a fairly mindless activity today, so I recreated my profile from scratch, re-installing all my extensions and configuring everything just the way I like it. It took a few hours, and my notes ended up surprisingly long.

The few times I’ve thought about switching to Chrome I’ve looked for one or two of these add-ons and seen that there’s no equivalent. Now that I have these notes I can check comprehensively the next time I get the urge.

The notes will mostly be of interest to web developers. My browser is pretty heavily customized with tools, privacy protections, and productivity tweaks (mostly to make it feel like vim and allow me to pull things offline easily). It’s an intimidating list to look at, but it’s ten years of small changes every few weeks all at once.

Add-Ons

  • AdBlock Plus http://adblockplus.org/en/
    • add EasyList (button in tab on startup)
    • ‘Options’ menu, uncheck ‘Show tabs on Flash and Java’
  • Better Privacy
    • select any LSOs you want to keep
  • Cookie Monster
    • uncheck “Automatically Reload…”
  • Download Statusbar
    • General
      • check ‘Download speed’
      • uncheck ‘Keep a download history
      • check ‘Clear finished downloads when the browser closes’
      • ‘Automatically clear these filetypes’: *
      • after 90 seconds
    • Appearance – choose ‘Custom Style’
      • File Name Size: 9
      • Height: 13
      • check ‘Main Downloads Button’
      • check ‘Clear Finished Button’
      • check ‘Enable Speed Colors’
  • FacebookBlocker
  • Flashblock
    • check ‘Block Silverlight as well’
    • add any whitelist sites
  • Greasemonkey
  • LeechBlock
    • block tvtropes.com
    • General – check all
  • Pentadactyl

    ~/.pentadactylrc

    "1.0b4.3 (created: 2011/01/05 17:55:12)
    
    loadplugins '\.(js|penta)$'
    set guioptions=BrsC
    set runtimepath=/home/harkins/.pentadactyl
    set urlseparator=,,s
    
    " vim: set ft=pentadactyl:
    

  • NoSquint
    • Zooming tab, uncheck ‘Show current zoom levels’
    • Set ‘Default full page zoom level’ to 100%
    • Exceptions tab, add .github.com, .tumblr.com, .blogspot.com, .posterous.com
  • Pixlr Grabber
    • uncheck ‘To edit images’ and ‘To edit backgrounds’
    • select ‘Always save to desktop’
  • Readability https://www.readability.com/addons
  • RefControl
    • click ‘Edit’ button, choose ‘Block’ and check ‘3rd Party requests only’
    • outside of settings, right click button at bottom of window, remove icon
  • Sauce Launcher https://addons.mozilla.org/en-us/firefox/addon/sauce-launcher/
  • ScrapBook
    • Show up to: 10
  • SQLite Manager
    • select ‘in a new tab’
    • uncheck ‘Always confirm’
    • uncheck everything in Prompts tab
  • Tamper Data
  • Tree Style Tab
    • Appearance
      • Skin ‘Sidebar’
      • Tree Twisties: choose ‘None’
    • Menu
      • uncheck all but ‘Reload this Tree’ and ‘Close this Tree’
    • New Tabs
      • uncheck ‘Always ask’
      • select ‘Open in new tabs’
    • Tree
      • uncheck ‘When a tab gets focus’
      • check ‘Double-click on a tab’
    • Advanced
      • uncheck ‘Enable Animation Effects’
      • uncheck ‘Show tree contents’
  • UnPlug
    • Integration – uncheck all but tools menu
    • Downloads – ‘Preferred download method’: Save as
  • User Agent Switcher
  • Web Developer
    • General
      • Check to hide the context menu, confirmation, and informational
    • Validation
      • select CSS 3

Incompatible

These aren’t yet compatible with Firefox 5, so they didn’t get into my notes.

GreaseMonkey Scripts

Sorry for the lack of titles, but this was annoying enough to compile already.

51281
62706
7705
23555
53547
60314
38516
31804
56690
100746
104489
38985
45032
75263
77801
94112
30646
23175
28374
29279
29682
29681
29829
29828
95298
29666
49366
74340
76300
45032

Toolbars

  • right click on the toolbar, uncheck Web Development toolbar
  • right click again, choose ‘Customize’, choose ‘Use Small Icons’ and remove:
    • read it later
    • search box
    • zoom buttons
    • home button
    • bookmarks button
    • FireBug button
    • drag GreaseMonkey button to status bar

Cookies

click on CookieMonster at bottom of window, click View Cookies, show exceptions.
‘Allow’ sites you regularly visit
‘Block’ google properties

Bookmarks menu -> Show all Bookmarks

‘Import and Backup’ -> Back up to json in old profile, Import ‘Choose File’ in new

Same with ScrapBook – open sidebar, Tools, Import / Export, select all…

Edit -> Preferences

  • General

    • ‘When Firefox starts’: Show my windows and tabs
    • ‘Home Page’: about:blank
  • Tabs

    • check everything but ‘When I open a link in a new tab’
  • Applications

    • search for VLC, Windows, and QuickTime, change items to ‘Always Ask’
  • Privacy

    • check ‘Tell web sites I do not want to be tracked’
    • select ‘Firefox will’: Use custom settings for history
    • select ‘Keep until’: I close Firefox
    • select ‘When using the location bar, suggest’: History
  • Security

    • uncheck ‘Remember passwords for sites’
  • Advanced -> Update

    • uncheck ‘Search Engines’

about:config

  • browser.autofocus false
  • browser.dom.window.dom.enabled true (new)
  • browser.history_expire_days 30
  • browser.tabs.animate false
  • browser.tabs.closeButtons 2
  • browser.tabs.loadBookmarksInBackground true
  • browser.tabs.loadDivertedInBackground true
  • browser.tabs.selectOwnerOnClose false
  • browser.zoom.full false
  • browser.jsannoyances.disabled true (new)
  • downbar.display.clearButton false
  • downbar.display.mainButton false
  • downbar.display.percent true
  • downbar.function.useAnimation false
  • network.http.pipelining true
  • network.http.pipelining.maxrequests 8
  • network.prefetch-next false

~/.mozilla/firefox/[profile]/chrome

userChrome.css

    @namespace url("http://www.mozilla.org/keymaster/gatekeeper/there.is.only.xul");
 
    /* don't bold active tab title */
    /* no longer works, argh */
    tab[selected="true"] { font-weight: normal !important; }
 
    /* stop TreeStyleTab's Sidebar theme from changing its color when in/active */
    .tabbrowser-strip[treestyletab-style="sidebar"][treestyletab-mode="vertical"] {
      background-color: #ccc !important;
    }

userContent.css

    /* indicate nofollow links with a dotted blue underline */
    a[rel~=nofollow] { border-bottom: 1px dotted blue !important; }

Got any neat suggestions I’ve missed?

Deploying Crontab With Your Rails App
Code: , ,
1 comment

This is a short one. If cron is an old friend, don’t futz around with weird Ruby. You know the pitfalls of cron (environment variables, long jobs without lock files). So write your crontab and check it into config/crontab.

0 0 * * * /usr/bin/some_important_command

Add this to your config/deploy.rb:

namespace :deploy do
  desc "Install crontab"
  task :crontab, :roles => :app do
    run "crontab #{current_path}/config/crontab"
  end
  after "deploy:symlink", "deploy:crontab"
end

Check in your code. Now every time you cap deploy, your crontab will be updated on your app servers. So simple it’s not even a gem on github.

Note that if you have multiple app servers and a job that should only run once (as opposed to once per app server), you’ll need to create a role and alter the task accordingly. I guess if someone were so inclined they could make a fancy gem (and host it on github, OF COURSE) that supports multiple crontabs named after roles, but it’s late and I’m not quite so inclined.


Fatal error: Call to undefined function twentyseventeen_get_svg() in /home/malaprop/push.cx/wp-content/themes/pushcx/archive.php on line 45