Skip to content

Upgrading from Ubuntu 9.04 to 9.10 – memcached problems, postgresql upgrades, and svn

I recently upgraded a couple of servers over at slicehost from Ubuntu 9.04 to 9.10 via the command line upgrade. There were only a couple problems.

During the install, I kept my php and svn config files, since the diff showed no major changes to the ones in the package (other than my configurations, of course).

The first problem was that the Postgresql 8.3 databases needed to be updated to 8.4, and the installer doesn’t do that. I made sure to run these before the installer removed the 8.3 package (it asks before doing so at the end of the install script), though I don’t know if that is strictly necessary. The upgrade was relatively simple:

#see what clusters you currently have
pg_lscluster

#drop the empty new 8.4 cluster the installer created
sudo pg_dropcluster 8.4 main

#upgrade your existing cluster
sudo pg_upgradecluster 8.3 main /var/lib/postgresql/8.4/main

The upgrade command SHOULD take care of copying the data, upgrading it to the new format, stopping the old cluster, and starting the new cluster on the same port. On one of my machines the process of stopping the old cluster failed, but doing it manually with pg_ctlcluster worked.

Once I upgraded the Postgres databases, I allowed the script to remove the old packages and reboot the machine. Then I selected the kernel upgrade in the slicehost control panel (which rebooted the machine again) and let it come up. Almost everything worked perfectly.

The other issue I encountered was that after the upgrade, memcached no longer started on reboot. After a bit of poking around, I discovered that the upgraded package that comes with 9.10 comes with a new file in /etc/default/memcached which defaults to having memcached disabled. To enable it, set ENABLE_MEMCACHED=yes and memcached starts when your server boots. This does have a launchpad bug, but nobody seems to be taking responsibility for fixing it.

Another gotcha turned out to be the fact that memcached must already be running when Django starts. I only resolved the issues with memcached after my Django processes had started on the other server, and it took me a few minutes to remember that I needed to restart them to fix caching.

The only other minor issue was that (as expected) the upgrade overwrote my script that gets invoked whenever svnserve is called. I have this in place so that I can svn+ssh into the server and check files out without specifying their absolute path in the file system. My script looks like this:

#!/bin/sh
umask 002
exec /usr/bin/svnserve.bin -t -r [path to svn repo] "$@"

I keep the actual svnserve binary (default installed to /usr/bin/svnserve) in svnserve.bin and the script in svnserve.ssh. I then link /usr/bin/svnserve to svnserve.ssh, so when svn gets upgraded, I just recopy the binary and re-link.

Posted 24 February 2010 § Code ‡ Comments (0)

Creativity in Education

My friend Volty recently started blogging with a focus towards intellectual development. His post on Sir Ken Robinson’s claim that creativity is as important as literacy in education encouraged me to consider my own responses.

The Oxford English Dictionary defines literate as “Acquainted with letters or literature; educated, instructed, learned.”

I do not take well to being told how to behave. I never have. I take even less well to being told how to think. How did this affect my schooling?

I will repeat what others have said, and assert that I’ve managed to prevent my schooling from getting in the way of my education. I’ve been supported in this attitude by friends, family, and even teachers for as long as I can remember.

Did I go to a “struggle to pass the state-mandated standardized test” public school? No. Did my schooling kill my creativity, or discourage my natural curiosity? Most profoundly, no. The dichotomy is certainly revelant — private schools can be assumed to have more resources devoted to tasks other than passing standardized tests in core subjects. Does my outcome represent my innate personality, or a difference in educational method from the one that Robinson is criticizing? I tend to lean towards the former.

I can only speak to the schools I have attended. In these, literacy was clearly held as a higher priority than creativity. There were creative classes, and creative activities within academically focused classes, but the primary goal was academic, not creative, proficiency.

In retrospect, I feel that this was appropriate. I disagree with Robinson’s (most likely intentionally) inflammatory claim that creativity must be the equal of literacy in education.

On the contrary, literacy is absolutely fundamental to true creativity. It provides the foundation for meaningful contributions. Without a firm grasp of the existing subject matter, how can one hope to make a new contribution to a creative field? Academic knowledge should not constrain a creative individual to follow an existing path. Instead, it shows possible paths which may optionally be followed for a while, before diverging into uncharted territory. Without the palette, there can be no masterpiece.

Literacy allows creativity without the painful steps of rediscovering that which could be taught by instruction. There is nothing shameful or innately wrong about organized instruction in a topic which allows creativity. In fact, the example Robinson himself cites — the dance school — is precisely this.

Perhaps it does boil down to talking about different kinds of education after all. In my experience, school has always been about learning how to ask questions, thinking of things that are “wrong” or unconventional, and generally exploring the topic at hand through whatever means available. I have yet to experience a topic which truly consisted of little but rote memorization, and if confronted with such, would most likely argue that it had no place in “education.” Whether this is a matter of personal character, or just going to the right schools, I cannot say. I can say that the education system he seems to presuppose is not one in which I would willingly participate.

I should stop before I again begin to diverge into unrestrained metaphor. Next time, I hope to consider intellectualism. Should everyone be explicitly trained in this discipline? Is everyone capable of it? How does “natural temperament” play a role?

As a footnote, I do really appreciate the written transcripts which accompany each TED Talk Video. They make the content far more accessible for review.

Posted 10 November 2009 § Conversations ‡ Comments (0)

Wordpress and music on android

Posting this via wptogo on my android. The app seems to work pretty well even if typing on a touchscreen does suck a bit.

I think I’ve been being overprecise in my typing. I can actually go much faster than I have been.

Since I got it, I’ve loved my ipod shuffle for being small, light, and just what I need without too many extras. I’ll listen on it literally all day. However, it IS an extra thing to carry around, keep charged, and lose.

Therefore I’m going to experimentally try using my phone for this for the next while. In the past I have been dissatisfied with this setup for several reasons, the largest of which is battery life. It is not ok to run my phone out of battery because I was listening to music. I need at least a day and a half of battery life (I get a bit more than two right now, depending on usage). I’m not happy about the need for a headphone adaptor. HTC needs to fix this serious flaw in otherwise nice hardware. I’m also a bit worried about sound quality – the amps in the last phone I had sucked horribly. In any case, it should be interesting.

Composing posts on this keyboard sucks. Coffee makes it worse.

Posted 28 August 2009 § General ‡ Comments (0)

IBM x346 Power Connector

For the record, IBM P/N: 90P4632 (FRU P/N: 25R5163) is made up of 2 end-to-end molex part number 43025-1800 connectors with female terminals.

Molex connector info:

http://www.marvin3m.com/connect/

Posted 27 August 2009 § Uncategorized ‡ Comments (0)

Thoughts on Android, fixing os slowdown

I recently started using an android-powered Google Ion (also known as the HTC Magic or T-Mobile myTouch) courtesy of the Google I/O conference.

I really like it, though I do miss the hard keyboard of the G1 I borrowed for several weeks. I feel that the smaller size is a fair trade, since it now fits comfortably in my pocket, where the G1 felt quite bulky. I had assumed that I would generally use the horizontally oriented keyboard since the keys are larger in that format, and typing should be faster. In fact, I find that I usually use the vertical orientation and type with my thumbs. The reason for this is that I can see more of the content I’m replying to, and don’t have to toggle the keyboard on and off to switch between reading a conversation and replying to it. Overall, the soft keyboard works pretty well, though I’ve found it works better with my thumbs than my fingers (I recently broke both my wrists, and my thumbs are immobilized, so I’m forced to type with the fingers).

When I first got the phone, the interface was snappy, and everything responded quickly. Over the past few weeks of use, things seemed to slow down to the point that menus would take several seconds to load, and pulling down the notification tray was sometimes not possible. Needless to say, I was extremely unhappy about this turn of affairs. I tried rebooting the phone, but that only caused a minor improvement. I checked to make sure there weren’t spurious background processes, and that the phone’s memory wasn’t overflowing. Finally today, I discovered the fix quite by accident. I am notoriously bad about checking my voicemail, and have had messages in my inbox for nearly the entire time I’ve had the phone. Amazingly, when I checked my voicemail and cleared out the inbox, the phone instantly returned to it’s old snappy self. My best guess is that clearing the notification tray entirely was what fixed the problem.

This may be related to issue 3453 in the android source, but my resolution seems to be different enough that it might be a separate bug.

Posted 15 August 2009 § General ‡ Comments (0)

5DmkII video – Playback on PS3, Editing in Premiere

The Canon 5D Mark II shoots beautiful video, but it is unfortunately encapsulated in the .mov container, which the Playstation 3 will not play natively. After much trial and error, I’ve found that H.264 encoded video and Dolby AC3 audio muxed into a .m2ts file works well.

If you simply want to watch your unedited video on the PS3, follow the following steps:

  • Install YAMB and select “Editing” and “Click to extract streams…”
  • Select your input .MOV and select “Extract to Raw Format” and then click “Extract”
  • Use BeSweet to convert the audio file (it’s a PCM wave) to AC3
  • Install tsMuxeR, and add the extracted video file and converted audio file to the input file list
  • Select “M2TS muxing” and specify your output filename
  • Click “Start muxing”, and when it’s finished, copy the file to the PS3 and play it

The above steps work well, but don’t allow for much editing to take place. I use Adobe Premiere Pro CS4 to edit video, but none of the native H.264 output formats that include both audio and video seem to play back correctly on the PS3.

This is the workflow I use for editing and rendering these videos:

  • Create a new project using the correct settings. Here’s my Adobe Premiere New Sequence Preset preset for the Canon 5D Mark II (It goes in the “Documents and Settings\<username>\Application Data\Adobe\Premiere Pro\4.0\Settings\Custom” folder on Windows XP)
  • Import your .MOV files and edit them as usual
  • Export your project using my Canon 5D Mark II Video Export Preset for PS3 or use these settings:
    • Format: H.264
    • Export Video and Audio checked
    • Multiplexer set to None
    • Video set to NTSC, 1920×1080 resolution, 30fps, None(Progressive) Field Order, Widescreen pixel aspect ratio, High profile. 20-30 Mbps Bitrate
    • Audio Format set to Dolby Digital
  • Open tsMuxeR and import your separate rendered video and audio files
  • Select “M2TS muxing” and specify your output filename
  • Click “Start muxing”, and when it’s finished, copy the file to the PS3 and play it

This produces full-resolution video output that does not drop frames, stutter, or display excessive transcoding artifacts.

This is what has worked for me. If you’ve got suggestions, or have problems with these methods, let me know and I’ll see if I can help you.

Posted 08 June 2009 § Photography ‡ Comments (1)

Canon 5D Mark II’s “Auto Lighting Optimizer” example

I captured a pair of images yesterday that demonstrate the Canon 5D Mark II’s “Auto Lighting Optimizer” setting pretty effectively. The setting is supposed to “analyze image brightness and automatically adjust dark areas in images so that they appear brighter.” It seems to work pretty well for some high contrast images.

It does improve this picture significantly, nicely bringing up the detail in the shadowed area on the robe. However, I’m leaving it off in most situations on my own camera, simply because I like having greater control over the exposure. The left image is with the Auto Lighting Optimizer off, the right with it set to “Strong”, the exposures are otherwise identical.

Auto Lighting Optimizer Comparison - Off on left, Strong on right

Auto Lighting Optimizer Comparison - Off on left, Strong on right

Posted 08 June 2009 § Photography ‡ Comments (1)

Ubuntu Upgrade, Server Thoughts

I upgraded my laptop to Ubuntu 9.04 lately, and since I’m using an x61 with the intel graphics card, compiz (and all my associated settings) stopped working. I de-blacklisted the drivers, and everything seems to be working properly. Hopefully I’m not one of the 20-30% of users who see crashes with the drivers…

The trick was the following command:

mkdir -p ~/.config/compiz/ && echo SKIP_CHECKS=yes >> ~/.config/compiz/compiz-manager

Other than that, I’m pretty pleased with the upgrade, and the improved boot time is wonderful.

I’ve been thinking about the appropriate way to build a file server for a while now. My goal is a box that will provide reliable redundant storage for several terabytes of personal files now, expandable over the next 5-10 years.

This needs to be a separate machine from my desktop machine. I’m not interested in using windows software raid for this box, and since my desktop machine tends to be in a state of flux much of the time, I have no interest in having to re-initialize a software raid every time I reinstall windows. Likewise, the “hardware” raid that comes on consumer motherboards is universally agreed to be worse than useless. Between poor driver support, poor performance, and (often) the inability to rebuild a damaged array, it’s not an option.

A separate machine brings up the issue of connectivity. I’ve looked at some of the fancier interconnect options, and think that Gigabit ethernet is probably appropriate for my needs. It’s the fastest of the cheap interconnect options, and should be fast enough for my needs (this is primarily an archive box, after all). This is nice because it allows the box to be on the network and accessible from my laptop and via wireless. I may eventually make it accessible from the outside world as well.

Reliability is important to me. In my previous experience, one of the best ways to kill a drive (or set of drives) is to install and use them in an improperly ventilated environment. This means that installing a bunch of drives in a typical desktop tower cage is not going to be acceptable, given the lack of ventilation and potential for extreme heat buildup.

After surveying a large number of options, I’m pretty convinced that rackmount server hardware is the best option for achieving this goal. It’s generally got quick-swap harddrives, good ventilation, and is designed for 24×7 operation.

There are a number of very nice rackmount SAS boxes (designed to just house drives, and connect to another computer), but the cost of the entire setup for even a cheap one would be pushing beyond my budget for this project. There are also some decently cheap esata + port multiplier solutions that would work, but the port multiplier hardware does not seem to be supported by the OS I want to use (Solaris).

In the end, I think my best option is going to be to simply purchase an old(er) server. Looking at ebay and craigslist, there seem to be surplus used servers in the $400 price range which are quite nicely suited to my needs: front-accessible quick-swap hard drive caddies, dual Xeons, 2GB+ ram, server-grade motherboards, hardware supported by Solaris. Many of these won’t have SATA onboard, but I should be able to add it with a pci express card. I’ll need to make sure that the quick-swap drive bays aren’t configured with a hard SCSI connection in the back of the bay.

After considering various types of raid, I think my best option is to use software for data redundancy. The various modes tend to require identical pairs of drives, and make adding further storage to the pool difficult. Hardware raid is expensive and less flexible than I want. I plan to use Sun’s ZFS filing system. It provides copy-on-write for data integrity, good resilvering options (to detect and correct random errors in data), various options for the amount of redundancy, and flexible tools for expanding and upgrading the storage pool over time. Since I plan to replace failed (and older) drives with larger ones over time, this level of flexibility is important.

Posted 01 May 2009 § General ‡ Comments (0)

BSG

Just finished watching the last episode of the last season of Battlestar Galactica. Really fucking good. Really worth watching, the all of it. Damn fine show, damn fine season and show finale.

Posted 20 April 2009 § Uncategorized ‡ Comments (0)

web.py, Dreamhost, and the django templating framework

Getting web.py working properly on Dreamhost was a real pain, especially since the wiki article on the subject was flat out wrong in several places. I ended up choosing to use fcgi for talking to apache, which is good for performance but not optimal for development work (you sometimes have to kill the process and restart it for changes to take effect). This is generally ok, since the built in web.py dev server works for local code debugging.

To use web.py 0.3 with fcgi on Dreamhost:

  • Enable FCGI in the Dreamhost panel. If you don’t do this, nothing will work! It can take a while (half an hour) to become enabled.
  • Download and unpack the web.py package to your working directory (or put it in your python path)
  • Download fcgi.py and put it in your path
  • Modify web/wsgi.py to replace the runfcgi function with this: 
    def runfcgi(func, addr=('localhost', 8000)):
    """Runs a WSGI function as a FastCGI server."""
    import fcgi as flups
    return flups.WSGIServer(func,
                            multiplexed=False,
                            bindAddress=addr).run()
  • Create your .htaccess:
    RewriteEngine On
    RewriteCond %{REQUEST_FILENAME} !-f
    RewriteRule ^/?(.*)$ dispatch.fcgi/$1 [L]
  • Create your dispatch.fcgi(see below) and chmod it 755

You have several options for your dispatch.cgi file. Mine looks like this:

#!/usr/bin/python2.5
import code
code.app.run()

Your dispatch.fcgi WILL NOT WORK if you use dos newlines, and it must not be group or world writeable. My web.py entry point (called code.py) contains (near the bottom): 

if __name__ == "__main__":
    app.run() #this is normally only called from dispatch.cgi
else:
    web.wsgi.runwsgi = lambda func, addr=None: web.wsgi.runfcgi(func, addr)

This allows me to use the built in web.py server for local testing while the same file works flawlessly on the server. There’s another (less good) option for this at the end of the post.

I had trouble getting web.py set up to work with the django templating system. It was raising a RuntimeError: Settings already configured. It turns out (obviously in retrospect) that web.py loads your main code more than once, and running the settings.configure command as recommended in the django documentation fails the second time. The solution is to put it within a try (or your dispatch.fcgi if it is set up like mine):


from django.conf import settings
try:
    settings.configure(DEBUG=True,
                       TEMPLATE_DEBUG=True,
                       TEMPLATE_DIRS=('/templates', ))
except:
    pass
from django.template.loader import render_to_string

Below is an alternative and less good approach for getting web.py working. Some may find it slightly easier. I dislike it.

  • Rename your entry point (the tutorial calls it code.py) to dispatch.fcgi
  • Insert the line:!#/usr/bin/python2.5 at the top.
  • Add the line: 
    web.wsgi.runwsgi = lambda func, addr=None: web.wsgi.runfcgi(func, addr)

    right above the if __name__ == "main": line.

  • Copy your new dispatch.fcgi to dispatch.py. From now on, every time you change one, you must update the other to match.
  • Make SURE your copy of dispatch.fcgi on the server is executable, but is not group or world writable (chmod 755).

If you’re having trouble, try running dispatch directly from the command line:
$ ./dispatch.fcgi
Often this will tell you more than a cryptic 500 error. Also be sure to check your apache logs.

Posted 09 April 2009 § Code ‡ Comments (2)