Skip to content

Canon 5D Mark II’s “Auto Lighting Optimizer” example

I captured a pair of images yesterday that demonstrate the Canon 5D Mark II’s “Auto Lighting Optimizer” setting pretty effectively. The setting is supposed to “analyze image brightness and automatically adjust dark areas in images so that they appear brighter.” It seems to work pretty well for some high contrast images.

It does improve this picture significantly, nicely bringing up the detail in the shadowed area on the robe. However, I’m leaving it off in most situations on my own camera, simply because I like having greater control over the exposure. The left image is with the Auto Lighting Optimizer off, the right with it set to “Strong”, the exposures are otherwise identical.

Auto Lighting Optimizer Comparison - Off on left, Strong on right

Auto Lighting Optimizer Comparison - Off on left, Strong on right

Posted 08 June 2009 § Photography ‡ Comments (3)

Ubuntu Upgrade, Server Thoughts

I upgraded my laptop to Ubuntu 9.04 lately, and since I’m using an x61 with the intel graphics card, compiz (and all my associated settings) stopped working. I de-blacklisted the drivers, and everything seems to be working properly. Hopefully I’m not one of the 20-30% of users who see crashes with the drivers…

The trick was the following command:

mkdir -p ~/.config/compiz/ && echo SKIP_CHECKS=yes >> ~/.config/compiz/compiz-manager

Other than that, I’m pretty pleased with the upgrade, and the improved boot time is wonderful.

I’ve been thinking about the appropriate way to build a file server for a while now. My goal is a box that will provide reliable redundant storage for several terabytes of personal files now, expandable over the next 5-10 years.

This needs to be a separate machine from my desktop machine. I’m not interested in using windows software raid for this box, and since my desktop machine tends to be in a state of flux much of the time, I have no interest in having to re-initialize a software raid every time I reinstall windows. Likewise, the “hardware” raid that comes on consumer motherboards is universally agreed to be worse than useless. Between poor driver support, poor performance, and (often) the inability to rebuild a damaged array, it’s not an option.

A separate machine brings up the issue of connectivity. I’ve looked at some of the fancier interconnect options, and think that Gigabit ethernet is probably appropriate for my needs. It’s the fastest of the cheap interconnect options, and should be fast enough for my needs (this is primarily an archive box, after all). This is nice because it allows the box to be on the network and accessible from my laptop and via wireless. I may eventually make it accessible from the outside world as well.

Reliability is important to me. In my previous experience, one of the best ways to kill a drive (or set of drives) is to install and use them in an improperly ventilated environment. This means that installing a bunch of drives in a typical desktop tower cage is not going to be acceptable, given the lack of ventilation and potential for extreme heat buildup.

After surveying a large number of options, I’m pretty convinced that rackmount server hardware is the best option for achieving this goal. It’s generally got quick-swap harddrives, good ventilation, and is designed for 24×7 operation.

There are a number of very nice rackmount SAS boxes (designed to just house drives, and connect to another computer), but the cost of the entire setup for even a cheap one would be pushing beyond my budget for this project. There are also some decently cheap esata + port multiplier solutions that would work, but the port multiplier hardware does not seem to be supported by the OS I want to use (Solaris).

In the end, I think my best option is going to be to simply purchase an old(er) server. Looking at ebay and craigslist, there seem to be surplus used servers in the $400 price range which are quite nicely suited to my needs: front-accessible quick-swap hard drive caddies, dual Xeons, 2GB+ ram, server-grade motherboards, hardware supported by Solaris. Many of these won’t have SATA onboard, but I should be able to add it with a pci express card. I’ll need to make sure that the quick-swap drive bays aren’t configured with a hard SCSI connection in the back of the bay.

After considering various types of raid, I think my best option is to use software for data redundancy. The various modes tend to require identical pairs of drives, and make adding further storage to the pool difficult. Hardware raid is expensive and less flexible than I want. I plan to use Sun’s ZFS filing system. It provides copy-on-write for data integrity, good resilvering options (to detect and correct random errors in data), various options for the amount of redundancy, and flexible tools for expanding and upgrading the storage pool over time. Since I plan to replace failed (and older) drives with larger ones over time, this level of flexibility is important.

Posted 01 May 2009 § General ‡ Comments (0)

BSG

Just finished watching the last episode of the last season of Battlestar Galactica. Really fucking good. Really worth watching, the all of it. Damn fine show, damn fine season and show finale.

Posted 20 April 2009 § Uncategorized ‡ Comments (0)

web.py, Dreamhost, and the django templating framework

Getting web.py working properly on Dreamhost was a real pain, especially since the wiki article on the subject was flat out wrong in several places. I ended up choosing to use fcgi for talking to apache, which is good for performance but not optimal for development work (you sometimes have to kill the process and restart it for changes to take effect). This is generally ok, since the built in web.py dev server works for local code debugging.

To use web.py 0.3 with fcgi on Dreamhost:

  • Enable FCGI in the Dreamhost panel. If you don’t do this, nothing will work! It can take a while (half an hour) to become enabled.
  • Download and unpack the web.py package to your working directory (or put it in your python path)
  • Download fcgi.py and put it in your path
  • Modify web/wsgi.py to replace the runfcgi function with this: 
    def runfcgi(func, addr=('localhost', 8000)):
    """Runs a WSGI function as a FastCGI server."""
    import fcgi as flups
    return flups.WSGIServer(func,
                            multiplexed=False,
                            bindAddress=addr).run()
  • Create your .htaccess:
    RewriteEngine On
    RewriteCond %{REQUEST_FILENAME} !-f
    RewriteRule ^/?(.*)$ dispatch.fcgi/$1 [L]
  • Create your dispatch.fcgi(see below) and chmod it 755

You have several options for your dispatch.cgi file. Mine looks like this:

#!/usr/bin/python2.5
import code
code.app.run()

Your dispatch.fcgi WILL NOT WORK if you use dos newlines, and it must not be group or world writeable. My web.py entry point (called code.py) contains (near the bottom): 

if __name__ == "__main__":
    app.run() #this is normally only called from dispatch.cgi
else:
    web.wsgi.runwsgi = lambda func, addr=None: web.wsgi.runfcgi(func, addr)

This allows me to use the built in web.py server for local testing while the same file works flawlessly on the server. There’s another (less good) option for this at the end of the post.

I had trouble getting web.py set up to work with the django templating system. It was raising a RuntimeError: Settings already configured. It turns out (obviously in retrospect) that web.py loads your main code more than once, and running the settings.configure command as recommended in the django documentation fails the second time. The solution is to put it within a try (or your dispatch.fcgi if it is set up like mine):


from django.conf import settings
try:
    settings.configure(DEBUG=True,
                       TEMPLATE_DEBUG=True,
                       TEMPLATE_DIRS=('/templates', ))
except:
    pass
from django.template.loader import render_to_string

Below is an alternative and less good approach for getting web.py working. Some may find it slightly easier. I dislike it.

  • Rename your entry point (the tutorial calls it code.py) to dispatch.fcgi
  • Insert the line:!#/usr/bin/python2.5 at the top.
  • Add the line: 
    web.wsgi.runwsgi = lambda func, addr=None: web.wsgi.runfcgi(func, addr)

    right above the if __name__ == "main": line.

  • Copy your new dispatch.fcgi to dispatch.py. From now on, every time you change one, you must update the other to match.
  • Make SURE your copy of dispatch.fcgi on the server is executable, but is not group or world writable (chmod 755).

If you’re having trouble, try running dispatch directly from the command line:
$ ./dispatch.fcgi
Often this will tell you more than a cryptic 500 error. Also be sure to check your apache logs.

Posted 09 April 2009 § Code ‡ Comments (3)

Airbears login script

Logging into airbears through the browser each time my laptop connects is a pain. This script made it less painful:


#!/usr/bin/perl -w
use strict;
$|++;

use File::Basename;
use WWW::Mechanize 0.72;

my $mech = WWW::Mechanize->new(  );

$mech->get( "https://wireless-gw1.berkeley.edu/logon" );
$mech->success() or die "Couldn't connect to wireless login page",
    $mech->response->status_line;

# Select the form, fill the fields, and submit

$mech->form_name( "logonForm" );

$mech->set_visible([text=>"username"],[password=>"password"]);

$mech->click("logon_action");
$mech->success or die "Wireless login didn't seem to work.:",
    $mech->response->status_line;

print "Logged onto Airbears.\n";

I needed to install the libwww-mechanize-formfiller perl package before it ran:
sudo apt-get install libwww-mechanize-formfiller-perl

Yoinked and (slightly) corrected from Yossi Farjoun.

Posted 04 March 2009 § Code ‡ Comments (2)

messaging

I’ve been thinking about a peer-to-peer message passing network for short-range radio communications. The system should be useful for various scales of devices, from longer-range mobile installations (in a vehicle using commercial bands) to short-range person-to-person systems implemented in the ISM bands. Hardware like TI’s eZ430-RF2500 eval kit would be an ideal development platform.

One problem inherently present in such a system is that of identification of other nodes. In an open network, any available node may accept and transmit messages (though only the intended recipient may read them). However, in order for efficient message passing, there must be a system to identify nodes.

Not all users will be comfortable with the idea of broadcasting their identity to the world at large. They may only be comfortable participating as an anonymous node. Others may prefer only to reveal their identity to other nodes they trust. This presents a problem: How do two nodes identify themselves securely to each other, without revealing their identity to untrusted parties?

I propose an exchange system as follows:
Alice sees a nearby unidentified node (Bob).
Alice sends a message asking if Bob wishes to know her identity.
Bob responds with a randomly generated one-use key to encrypt the next message with.
Alice responds with a message encrypted with the public keys of all her trusted nodes, and further encrypts this message using Bob’s one-use key.
Bob decrypts the outer message using the one-time key, then attempts to decrypt the enclosed message using his private key.
If Alice trusts Bob, he can do this, and finds within her message a temporary key to use for further communications with her.
Bob continues communication with Alice using this new key.

At this point, Alice knows that she trusts Bob, but does not actually know his identity, unless he chooses to reciprocate the authentication steps.

The one-time encryption key wrapper used by Bob is a measure to prevent Carol, who is also trusted by Alice, from decrypting the key exchange message and spoofing communications from Bob.

Because no node broadcasts a canonical name, an attempt to decrypt each packet with each available key must be made. This quickly becomes impractical in any multi-node situation. To solve this problem, a node prefixes each message sequence with a randomly generated “conversation” id. This way, if Alice and Bob are talking, Carol only has to attempt to decrypt one packet from each, and can then safely ignore any messages using those conversation ids. In the event of a conflict (two nodes happen to use the same conversation id), both nodes should immediately select a new, not-recently-seen key. Key negotiation conversations are prefixed with a temporary identifier, in addition to the conversation key. Temporary conversation keys are dropped from the ignore list much faster (perhaps 30 seconds).

Messages must contain conversation sequence numbers inside the encrypted payload. These should be initialised randomly. Packet loss may occur, which means that some messages may arrive with a packet ID several higher than the last, but key renegotiation should take place if the difference is too great, or a packet with a sequence number lower than the highest known arrives (an indication of an attempted replay attack). This number should roll over to 0 when it reaches the maximum value.

There are a number of areas of potential information leakage. If Mallory can determine that Carol is ignoring messages with the conversation ids used earlier by Alice and Bob, Mallory can conclude that Carol is not Alice or Bob. To counteract this and other scenarios, the default behavior MUST be to ignore all invalid or unexpected messages. Error messages for these circumstances, even encrypted, may leak information.

Posted 02 March 2009 § General ‡ Comments (0)

OOK Update/Wrapup

It’s been a while since I posted about this. Since I last posted, I received the pair of custom lenses and have been wearing them on a regular basis until now.

People often ask whether I can see with the lenses in. Yes, they correct the vision even during the acclimatization period, so if you absolutely have to be able to see, you can. I drove to LA wearing them before my vision was good enough to drive without them in. This is not a good idea though, because the overnight prescription is designed to be centered on the eye with your eyelids closed, which is not the same position (in my eyes at least) as when they’re opened. This can result in an off-center treatment area, and subsequent (temporary) vision quality problems when the lenses are removed.

It seems that I have unusually dry eyes, and as such, have had problems with the overnight wear. When I wake up, the contacts have been stuck to my eyes, requiring extensive wetting and re-wetting before they can be safely removed. If I don’t spend the requisite time (often as much as 15 minutes) working them to naturally unstick (and sometimes even if I do) taking them out results in the pealing off of a layer of corneal cells. The eye heals quickly, the pain goes away in a few hours, and this damage is usually mostly repaired by the time I put them back in in the evening, but it is far from optimal. Most patients do not experience this, and the lenses are still floating freely on a layer of tears when they wake up. Unfortunately, the dryness is causing enough problems that my doctor feels that I should discontinue wear of the lenses. This is unusual, given that OOK is often suggested as a treatment alternative for people who have dry eyes which make daytime wear of lenses uncomfortable.

I’m sad to be stopping.

Overall, I really like the lenses. It’s fantastic to be out and about without anything in your eyes, to be able to see while showering, swimming, or any other activity. After nearly a month of wear, the end-of-day degradation in vision quality has reached a point where I still find the vision acceptable even late into the night. It’s convenient to wear the lenses at night and have them out during the day – no more worrying about carrying around a case so that I can take them out if it gets too late while I’m still out, and no more painful eyes from wearing lenses too long during the day. The flexibility is nice. Of course, the opposite is also true: if I skip a night, the vision the next day is noticeably bad.

The quality of vision achieved with the lenses has been variable for me. Because of the dryness, particularly in the left eye, the vision is sometimes degraded while the damaged areas of the cornea heal. This manifests itself as general blurriness, and difficulty resolving fine detail which is generally gone a few hours after lens removal. Additionally, because I have unusually large pupils, I can see the edges of the treatment zone at night, resulting in halos. This has gotten better as I’ve worn the lenses long term, but is still present. Furthermore, low-light intensifies the natural daily degradation. I can still function just fine, and driving is no problem, but things aren’t crystal clear.

However, when there are not dryness problems, or I’m outside during the day, I’m able to see better with the treatment than I ever have before. Soft contact lenses do not compare. The world is fantastically crisp, so much sharper, more detailed. I highly recommend them, since for most (almost all) users, the dryness I experienced will be a non-issue.

Moving forward, I plan to wait a month or so for my eyes to fully return to their natural shape, and get a pair of hard contact lenses. Even though the treatment lenses were uncomfortable at first, the vision quality provided by the hard lenses far surpasses that of soft contacts. I think I’m willing to trade the slight comfort margin for the improved vision.

You might also want to look at my other posts on Overnight Orthokeratalogy.

Posted 18 February 2009 § Personal ‡ Comments (0)

Disabling WordPress 404 handling using .htaccess

I run WordPress in the root of my site, but dislike the fact that whenever anyone mistypes a url anywhere else in my site, they get sent to the WordPress error page. The fix is fairly simple if WordPress is installed in its own directory…

Edit your .htaccess in the root of your site to look like this:

# BEGIN WordPress

RewriteEngine On
RewriteBase /
RewriteRule ^archives/.*$ /index.php [L]
RewriteRule ^feed.*$ /index.php [L]
RewriteRule ^page/.*$ /index.php [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^wordpress/.*$ /index.php [L]

# END WordPress

Now, when a user mistypes a url in the non-wordpress portions of your site, they will see a normal apache error. I use the archives/ formatting for my old posts, you may have to customize yours differently. Additionally, you may need to modify the last line if your blog is not installed in wordpress/. The last few lines DO need to remain there, because WP seems to do some fancy re-writing for the more advanced bits of the admin interface, such as adding open-ids to user profiles.

You may need to add other lines depending on what other modules you may have installed that add further items to the root namespace besides the page views.

I also have the lines:

Redirect 301 /login http://thefire.us/wordpress/wp-login.php
Redirect 301 /logout http://thefire.us/wordpress/wp-login.php?action=logout


which allow me to log in and out via http://site/login rather than retyping or bookmarking the admin url.

Posted 13 February 2009 § Code ‡ Comments (6)

Recent Photographic Happenings

I’m shooting on new equipment – the Canon EOS 5D Mark II and the Canon EF 70-200mm f/2.8L IS USM Lens. Both are superb. I’ll write up my thoughts on the 5D after I’ve played with it a bit more, but initially at least, I’m very happy.

Below are a few photos taken with the 5DII and the 50mm f/1.8 II. I really like this lens on the full-frame body. These are resized, but otherwise directly off the camera – 1/50th at f/1.8 and auto-ISO ranging from 800 to 2000.
Continue Reading »

Posted 17 January 2009 § General ‡ Comments (0)

reading books

I don’t really believe in new years resolutions, but I’ve made a decision this morning that somewhat resembles one. Since graduating from highschool, I seem to have almost completely stopped reading for pleasure. I miss it. I’m going to make a point of finishing at least one (non-school) book each month from now on. I may read it in print, or listen to an unabridged audio rendition, but I really want that part of my life back.

I’m going to start by finishing “Dark Star Safari: Overland from Cairo to Capetown” by Paul Theroux.

Posted 06 January 2009 § Personal ‡ Comments (0)