Not A Good Day for Ruby

Peter Harkins — Sun, 22 Jun 2008 23:44:11 +0000


*** glibc detected *** /usr/bin/ruby: double free or corruption (out): 0x299b2b08 ***
======= Backtrace: =========
/lib/tls/i686/cmov/libc.so.6[0xb7d40d65]
/lib/tls/i686/cmov/libc.so.6(cfree+0x90)[0xb7d44800]
/usr/lib/libruby1.8.so.1.8[0xb7ed5d79]
/usr/lib/libruby1.8.so.1.8(ruby_xmalloc+0x85)[0xb7ed60c5]
/usr/lib/libruby1.8.so.1.8(rb_str_buf_new+0x50)[0xb7f25b80]
/usr/lib/libruby1.8.so.1.8[0xb7ef157e]
/usr/lib/libruby1.8.so.1.8[0xb7eb733a]
/usr/lib/libruby1.8.so.1.8[0xb7ebee8e]
/usr/lib/libruby1.8.so.1.8[0xb7ebfbc8]
/usr/lib/libruby1.8.so.1.8[0xb7ec797e]
/usr/lib/libruby1.8.so.1.8[0xb7ec78fc]
/usr/lib/libruby1.8.so.1.8[0xb7ebf766]
/usr/lib/libruby1.8.so.1.8[0xb7ebfbc8]
/usr/lib/libruby1.8.so.1.8[0xb7ec797e]
/usr/lib/libruby1.8.so.1.8[0xb7ebf766]
/usr/lib/libruby1.8.so.1.8[0xb7ebfbc8]
/usr/lib/libruby1.8.so.1.8[0xb7ec797e]
/usr/lib/libruby1.8.so.1.8[0xb7ebf766]
/usr/lib/libruby1.8.so.1.8[0xb7ebfbc8]
/usr/lib/libruby1.8.so.1.8[0xb7ec797e]
/usr/lib/libruby1.8.so.1.8[0xb7eca68a]
/usr/lib/libruby1.8.so.1.8(rb_yield+0x21)[0xb7ecbad1]
/usr/lib/libruby1.8.so.1.8(rb_ary_each+0x31)[0xb7ea41a1]
/usr/lib/libruby1.8.so.1.8[0xb7eb7345]
/usr/lib/libruby1.8.so.1.8[0xb7ebee8e]
/usr/lib/libruby1.8.so.1.8[0xb7ebfbc8]
/usr/lib/libruby1.8.so.1.8[0xb7ec797e]
/usr/lib/libruby1.8.so.1.8[0xb7ec953d]
/usr/lib/libruby1.8.so.1.8[0xb7ec6d64]
/usr/lib/libruby1.8.so.1.8[0xb7ebf766]
/usr/lib/libruby1.8.so.1.8[0xb7ebfbc8]
/usr/lib/libruby1.8.so.1.8[0xb7ec797e]
/usr/lib/libruby1.8.so.1.8[0xb7ec8d9f]
/usr/lib/libruby1.8.so.1.8[0xb7eca68a]
/usr/lib/libruby1.8.so.1.8[0xb7ec771d]
/usr/lib/libruby1.8.so.1.8[0xb7ec87d3]
/usr/lib/libruby1.8.so.1.8[0xb7ebf766]
/usr/lib/libruby1.8.so.1.8[0xb7ebfbc8]
/usr/lib/libruby1.8.so.1.8[0xb7ec797e]
/usr/lib/libruby1.8.so.1.8[0xb7ec8d9f]
/usr/lib/libruby1.8.so.1.8[0xb7eca68a]
/usr/lib/libruby1.8.so.1.8[0xb7ec771d]
/usr/lib/libruby1.8.so.1.8[0xb7ec7d91]
/usr/lib/libruby1.8.so.1.8[0xb7ebf766]
/usr/lib/libruby1.8.so.1.8[0xb7ebfbc8]
/usr/lib/libruby1.8.so.1.8[0xb7ec7aa4]
/usr/lib/libruby1.8.so.1.8[0xb7ec8d9f]
/usr/lib/libruby1.8.so.1.8[0xb7ebf766]
/usr/lib/libruby1.8.so.1.8[0xb7ebfbc8]
/usr/lib/libruby1.8.so.1.8[0xb7ec797e]
/usr/lib/libruby1.8.so.1.8[0xb7ebf766]
/usr/lib/libruby1.8.so.1.8[0xb7ebfbc8]
/usr/lib/libruby1.8.so.1.8[0xb7ec766a]
/usr/lib/libruby1.8.so.1.8[0xb7ec76ce]
/usr/lib/libruby1.8.so.1.8[0xb7ebf766]
/usr/lib/libruby1.8.so.1.8[0xb7ebfbc8]
/usr/lib/libruby1.8.so.1.8[0xb7ec766a]
/usr/lib/libruby1.8.so.1.8[0xb7ec76ce]
/usr/lib/libruby1.8.so.1.8[0xb7ebf766]
/usr/lib/libruby1.8.so.1.8[0xb7ebfbc8]
/usr/lib/libruby1.8.so.1.8[0xb7ec766a]
/usr/lib/libruby1.8.so.1.8[0xb7ec76ce]
/usr/lib/libruby1.8.so.1.8[0xb7ebf766]
======= Memory map: ========
08048000-08049000 r-xp 00000000 08:03 1704787    /usr/bin/ruby1.8
08049000-0804a000 rw-p 00000000 08:03 1704787    /usr/bin/ruby1.8
0804a000-2af12000 rw-p 0804a000 00:00 0          [heap]
b5f00000-b5f21000 rw-p b5f00000 00:00 0
b5f21000-b6000000 ---p b5f21000 00:00 0
b6055000-b605f000 r-xp 00000000 08:03 1802307    /lib/libgcc_s.so.1
b605f000-b6060000 rw-p 0000a000 08:03 1802307    /lib/libgcc_s.so.1
b6060000-b7626000 rw-p b6060000 00:00 0
b7626000-b765e000 r-xp 00000000 08:03 737293     /usr/lib/ruby/1.8/i486-linux/nkf.so
b765e000-b7661000 rw-p 00037000 08:03 737293     /usr/lib/ruby/1.8/i486-linux/nkf.so
b7661000-b7662000 rw-p b7661000 00:00 0
b7662000-b7675000 r-xp 00000000 08:03 737288     /usr/lib/ruby/1.8/i486-linux/dl.so
b7675000-b7676000 rw-p 00012000 08:03 737288     /usr/lib/ruby/1.8/i486-linux/dl.so
b7676000-b77a1000 r-xp 00000000 08:03 1671348    /usr/lib/i686/cmov/libcrypto.so.0.9.8
b77a1000-b77b6000 rw-p 0012a000 08:03 1671348    /usr/lib/i686/cmov/libcrypto.so.0.9.8
b77b6000-b77b9000 rw-p b77b6000 00:00 0
b77b9000-b77f6000 r-xp 00000000 08:03 1671350    /usr/lib/i686/cmov/libssl.so.0.9.8
b77f6000-b77fa000 rw-p 0003c000 08:03 1671350    /usr/lib/i686/cmov/libssl.so.0.9.8
b7804000-b7807000 r-xp 00000000 08:03 819234     /usr/lib/ruby/1.8/i486-linux/digest/sha1.so
b7807000-b7808000 rw-p 00002000 08:03 819234     /usr/lib/ruby/1.8/i486-linux/digest/sha1.so
b7808000-b780b000 r-xp 00000000 08:03 737287     /usr/lib/ruby/1.8/i486-linux/digest.so
b780b000-b780c000 rw-p 00002000 08:03 737287     /usr/lib/ruby/1.8/i486-linux/digest.so
b780c000-b7842000 r-xp 00000000 08:03 2297548    /usr/lib/ruby/1.8/i486-linux/openssl.so
b7842000-b7844000 rw-p 00035000 08:03 2297548    /usr/lib/ruby/1.8/i486-linux/openssl.so
b7844000-b7a45000 rw-p b7844000 00:00 0
b7a45000-b7a59000 r-xp 00000000 08:03 1640498    /usr/lib/libz.so.1.2.3.3
b7a59000-b7a5a000 rw-p 00013000 08:03 1640498    /usr/lib/libz.so.1.2.3.3
b7a5b000-b7a5c000 rw-p b7a5b000 00:00 0
b7a5c000-b7a60000 r-xp 00000000 08:03 737298     /usr/lib/ruby/1.8/i486-linux/strscan.so
b7a60000-b7a61000 rw-p 00003000 08:03 737298     /usr/lib/ruby/1.8/i486-linux/strscan.so
b7a61000-b7a6b000 r-xp 00000000 08:03 737296     /usr/lib/ruby/1.8/i486-linux/socket.so
b7a6b000-b7a6c000 rw-p 00009000 08:03 737296     /usr/lib/ruby/1.8/i486-linux/socket.so
b7a6c000-b7a74000 r-xp 00000000 08:03 737323     /usr/lib/ruby/1.8/i486-linux/zlib.so
b7a74000-b7a75000 rw-p 00007000 08:03 737323     /usr/lib/ruby/1.8/i486-linux/zlib.so
b7a75000-b7a8f000 r-xp 00000000 08:03 737299     /usr/lib/ruby/1.8/i486-linux/syck.so
b7a8f000-b7a90000 rw-p 0001a000 08:03 737299     /usr/lib/ruby/1.8/i486-linux/syck.so
b7a90000-b7cd7000 rw-p b7a90000 00:00 0
b7cd7000-b7e1b000 r-xp 00000000 08:03 1838871    /lib/tls/i686/cmov/libc-2.6.1.so
b7e1b000-b7e1c000 r--p 00143000 08:03 1838871    /lib/tls/i686/cmov/libc-2.6.1.so
b7e1c000-b7e1e000 rw-p 00144000 08:03 1838871    /lib/tls/i686/cmov/libc-2.6.1.so
b7e1e000-b7e21000 rw-p b7e1e000 00:00 0
b7e21000-b7e44000 r-xp 00000000 08:03 1838875    /lib/tls/i686/cmov/libm-2.6.1.so
b7e44000-b7e46000 rw-p 00023000 08:03 1838875    /lib/tls/i686/cmov/libm-2.6.1.so
b7e46000-b7e4b000 r-xp 00000000 08:03 1838873    /lib/tls/i686/cmov/libcrypt-2.6.1.so
b7e4b000-b7e4d000 rw-p 00004000 08:03 1838873    /lib/tls/i686/cmov/libcrypt-2.6.1.so
b7e4d000-b7e74000 rw-p b7e4d000 00:00 0
b7e74000-b7e76000 r-xp 00000000 08:03 1838874    /lib/tls/i686/cmov/libdl-2.6.1.so
b7e76000-b7e78000 rw-p 00001000 08:03 1838874    /lib/tls/i686/cmov/libdl-2.6.1.so
b7e78000-b7e79000 rw-p b7e78000 00:00 0
b7e79000-b7e8d000 r-xp 00000000 08:03 1838885    /lib/tls/i686/cmov/libpthread-2.6.1.so
b7e8d000-b7e8f000 rw-p 00013000 08:03 1838885    /lib/tls/i686/cmov/libpthread-2.6.1.so
b7e8f000-b7e91000 rw-p b7e8f000 00:00 0
b7e91000-b7f4f000 r-xp 00000000 08:03 1755193    /usr/lib/libruby1.8.so.1.8.6
b7f4f000-b7f52000 rw-p 000bd000 08:03 1755193    /usr/lib/libruby1.8.so.1.8.6
b7f52000-b7f62000 rw-p b7f52000 00:00 0
b7f62000-b7f64000 r-xp 00000000 08:03 819232     /usr/lib/ruby/1.8/i486-linux/digest/md5.so
b7f64000-b7f65000 rw-p 00001000 08:03 819232     /usr/lib/ruby/1.8/i486-linux/digest/md5.so
b7f65000-b7f66000 r-xp 00000000 08:03 737291     /usr/lib/ruby/1.8/i486-linux/fcntl.so
b7f66000-b7f67000 rw-p 00000000 08:03 737291     /usr/lib/ruby/1.8/i486-linux/fcntl.so
b7f67000-b7f6b000 r-xp 00000000 08:03 737297     /usr/lib/ruby/1.8/i486-linux/stringio.so
b7f6b000-b7f6c000 rw-p 00003000 08:03 737297     /usr/lib/ruby/1.8/i486-linux/stringio.so
b7f6c000-b7f6e000 r-xp 00000000 08:03 737290     /usr/lib/ruby/1.8/i486-linux/etc.so
b7f6e000-b7f6f000 rw-p 00001000 08:03 737290     /usr/lib/ruby/1.8/i486-linux/etc.so
b7f6f000-b7f72000 r-xp 00000000 08:03 737301     /usr/lib/ruby/1.8/i486-linux/thread.so
b7f72000-b7f73000 rw-p 00002000 08:03 737301     /usr/lib/ruby/1.8/i486-linux/thread.so
b7f73000-b7f75000 rw-p b7f73000 00:00 0
b7f75000-b7f8f000 r-xp 00000000 08:03 1808831    /lib/ld-2.6.1.so
b7f8f000-b7f91000 rw-p 00019000 08:03 1808831    /lib/ld-2.6.1.so
bfef7000-bff1c000 rw-p bfef7000 00:00 0          [stack]
ffffe000-fffff000 r-xp 00000000 00:00 0          [vdso]
Aborted (core dumped)

The output of one of the four crashes I earned from the Ruby interpreter today. It’s likely the rubyzip library being flaky, but ouch. I’ve got some long jobs pushing around ~12G (uncompressed) for a project I’ll show off later this week and it’s hard to get finished when I have to babysit jobs.

Clean Up Your Mess

Peter Harkins — Thu, 05 Oct 2006 13:17:12 +0000

Too many sysadmins is a bad thing, especially if one of them doesn’t care about keeping the servers up.

The development box at work wasn’t letting me check anything into subversion — commits were just sitting there, not even timing out. In fact, so were updates. Something was seriously wrong.

I talked about it with a coworker and went to look at the box. Afew a few minutes of poking around, the problem became clear: someone installed a backup program that was trying to do some kind of fake-filesystem and wedged the box. Any process that tried to read from disk froze and couldn’t even be kill -9‘d.

And thanks to this odd little behavior, I could see three reboot processes frozen, presumably trying to read the shutdown scripts. So the person that wedged the box knew they wedged it but they just left it that way.

I got the coworker in the office to pull the plug on the box and it came up OK, but I edited /etc/init.d/arkeia to spit out the following note instead of try to start the backup program:

Dear whoever the hell decided to install arkeia:

You left the dev box wedged overnight, wasting at least an hour of two coders time to figure out what you did and fix it. And we know that you know you broke it, we could see that you tried to reboot and then LEFT IT FOR SOMEONE ELSE TO DEAL WITH rather than actually fix it.

Don’t be a jerk! Clean up after yourself!

Please talk to Jim and Harkins and explain why you left the box broken before you try playing with arkeia and wedge the box again.

Push cx » crash

Not A Good Day for Ruby

Clean Up Your Mess