Software development is not yellow

Best blog post I've read in a long time (and I read a lot!): Chris Dillow's Experts and the Demand for Certainty.

It's a sharp, spot-on, well-reasoned, well-written analysis, and -- it finally explains to me why I've always gotten so much push-back when, as an expert in my field (software development, and the management thereof), I offer "customers" my forecasts in the right form.

By "customers", here, I mean stakeholders of any stripe, including most product managers, top managers, etc -- even other developers, when they look up to me as a manager, mentor, or "senior advisor" figure rather than sensibly interacting with me as a peer!

And, the right form of forecast is always going to be expressed something like "X plus or minus Y" (or equivalently "between Z and T", where Z=X-Y and T=X+Y), accurately and explicitly expressing the margin of error that always accompanies any prediction (you can take the "with 95% probability" proviso as implied in any such quantitative expression).

Here's why I invariably get vehement push-back against expressing forecasts in the only sensible ways: the "customers" don't want from me, the expert, my knowledge; rather, what they want is certainty -- fake, bogus, fraudulent certainty is so much better than the messy, error-margin-involving business that's inevitable when you reach towards knowledge of reality:-(.

Now, this important realization doesn't teach me how to force stakeholders to face reality, but it does help me frame and understand their mindset, and thus helps me predict (with some margin of error, to be sure;-) how well or badly various communication strategies will work...

Strong types, weakly bound

Reading Strong opinions, somewhat weakly held on Jason Cohen's excellent A Smart Bear blog this morning, I was quoting this excellent soundbite (you really need to simplify it down to "strong opinions, weakly held" to make it work as a soundbite) out loud to Anna and musing about it -- and exactly at the same time she and I chorused about the analogy this has with Python's approach to typing... objects with strong, precise types, but "weakly held" (i.e., weakly bound) via names or slots in containers that might just as well hold objects of other (also strong) types.

This analogy may actually be the least "obvious" thing she and I found ourselves "chorusing" about, though such "choruses" happen more and more often and so it's becoming hard to tell;-). Anyway, I'll be sure to use this next time I need to present Python's typing approach -- now if I can only find a way to work ducks into it, too...

Get your blog typealyzed!

Fun URL to try: Typealizer (HT Greg Mankiw).

What Typealizer has to say about Aleaxity (quoted verbatim, including punctuation/grammar errors and typoes)...:

"""

The analysis indicates that the author of http://aleaxit.blogspot.com is of the type:
ISTP - The Mechanics
[ISTP]
The independent and problem-solving type. They are especially attuned to the demands of the moment are masters of responding to challenges that arise spontaneously. They generelly prefer to think things out for themselves and often avoid inter-personal conflicts.

The Mechanics enjoy working together with other independent and highly skilled people and often like seek fun and action both in their work and personal life. They enjoy adventure and risk such as in driving race cars or working as policemen and firefighters.
"""

The "enjoy adventure and risk" is as off-base as it could possibly be (if anything, I enjoy reducing and controlling risks and adventures by carefully and prudently balancing solid architectures, sound methodologies, best practices, ...), but other bits do appear rather spot-on;-).

Python tips: pickling it right

Here's an often-seen Python snippet...:

pickle.dump(stuff, open('foo.pik', 'w'))

What's wrong with this? Well, several things, as it turns out...!

1. Use cPickle, not pickle: that will speed things up by 5 or 6 times, effortlessly.


2. The common, sloppy use of open without a corresponding close is theoretically OK in today's cPython, but there's really no good reason to support it. Be neat instead, and write (after a from __future__ import with_statement if you're still using Python 2.5):
   

   
   with open('foo.pik', 'w') as f:
    cPickle.dump(stuff, f)
   



3. Unless there's a very special reason to make you want the pickle dump to be in ASCII (and I've hardly ever seen a good one), don't just use pickle's default, legacy protocol! Rather, explicitly request protocol 2, or better still, unless you need pickle files loadable by older releases of Python, request "the best protocol available".
   

So, the best equivalent of that little sloppy but alas-too-common idiom is:

with open('foo.pik', 'wb') as f:
 cPickle.dump(stuff, f, cPickle.HIGHEST_PROTOCOL)

Don't forget the little b in 'wb', by the way — it won't matter under Linux, OSX, or Solaris, but it will matter in Windows... and, anyway, as we all know, explicit is better than implicit!-)

Python: introspecting for generator vs function

At Baypiggies yesterday evening, Fernando Perez asked how to tell by introspection whether you're dealing with a "plain old Pythjon-coded function", like, say,

def f(): return 1

or rather a generator function, like, say,

def y(): yield 1

Apparently, he needs that info to perfect some decorator they're using as part of a nose-based test framework for scipy.

I don't think there's any other way except by looking at the bytecode for Yield vs Return opcodes; so, I jotted down on the spot the quick-and-dirty approach based on that idea:

import dis, sys
def is_generator(f):
   save_stdout = sys.stdout
   fake_stdout = cStringIO.StringIO()
   try:
       sys.stdout = fake_stdout
       dis.dis(f)
   finally:
       sys.stdout = save_stdout
   return ' YIELD_VALUE ' in fake_stdout.getvalue()

Of course, this does much more work than necessary (AND can be fooled by a function containing a peculiar string literal... like itself!-) . So, early this morning, I prepped a somewhat better performing and more solid version:

import opcode
YIELD_OP = opcode.opmap['YIELD_VALUE']
RETURN_OP = opcode.opmap['RETURN_VALUE']
HAVE_ARG = opcode.HAVE_ARGUMENT

def isgen(f):
    code = f.func_code.co_code
    n = len(code)
    i = 0
    while i < n:
        op = ord(code[i])
        i += 1
        if op >= HAVE_ARG:
            i += 2
        elif op == YIELD_OP:
            return True
        elif op == RETURN_OP:
            return False
    return False

Hope this can prove useful to someone else!-)


Alex

Meme spotted on Steve Holden's blog:

• Grab the nearest book.
• Open it to page 56.
• Find the fifth sentence.
• Post the text of the sentence in your journal along with these instructions.
• Don’t dig for your favorite book, the cool book, or the intellectual one: pick the CLOSEST.

Closest to me on the couch is Barry Cunliffe's Europe Between the Oceans: 9000 BC to AD 1000, which I just bought and haven't gotten around to reading yet. 5th sentence on p. 56 is: "The fame of early Troy owed much to its command of this favoured location.". (I got lucky: other sentences around this one, explaining in detail about the bay that is the location in question, and its importance, are way longer;-).

Carmen: great music, lousy geography

My Pedant self can't believe nobody else appears to have noticed this somewhere on the Web, but... I'm just back from an excellent Carmen at West Bay Opera, but, throughout it (except when Bizet's wondrous music, so well played and sung, just swept my pedantry away for a while!), I couldn't help being troubled by its plot's obvious imprecision regarding the geography of Spain.

Consider: Carmen takes place in Sevilla -- Micaela is coming to meet Don José there (presumably on foot? but, even if she was riding, or on a carriage, that wouldn't change much...) from their native village in Navarra, and claim it's a day trip ("demain je verrai votre mère", "I'll see your mother tomorrow", she tells José -- and said mother is back in said village). Sevilla is in Southern Spain, Navarra in the far North thereof... 912 km by road between Sevilla and Pamplona (the latter being the largest town in Navarra), according to Google Maps.

I also suspect that the mountains on which the smugglers are in the 3rd act are meant to be the Pyrenees, not the Sierra Morena which is reasonably close to Sevilla, simply because it's not clear where you'd be smuggling to and from over the Sierra Morena -- Andalucía to Extremadura? I doubt there were customs officers patroling that administrative border, given that both regions were and are part of the same country, Spain; however, I am not certain about that... it would just go well with the misconception that Sevilla and Navarra are very close to each other (Navarra, of course, is in fact on the Pyrenees).

So I'm disappointed with Meilhac and Halévy, the librettistas -- even though I absolutely adore them for their authorship of so many of Offenbach's best librettos. I revere Halévy and Meilhac when they play fast and loose with geography for good reason -- as in Offenbach's "Les Brigands", ``at the border between Spain and Italy'' (;-)... but in Carmen, they seem to have simply been sloppy and careless -- really unforgivable for two stalwart of the Académie Française!!!

Pedantry: assure, ensure, insure

My (beloved) local (print!) newspaper publishes today (10/13) an article (also available online at http://www.mercurynews.com/ci_10711901?IADID=Search-www.mercurynews.com-www.mercurynews.com) claiming that (my emphasis) "To assure continued fiscal security, Kane thinks local officials need to build stronger relationships with state representatives.". Isn't the added value of print newspapers vs blogs supposed to be good copy-editing?! And yet, here they are at least as bad as any unedited blog could possibly be. I'm crying.

A concise explanation can be found e.g. at http://www.wsu.edu/~brians/errors/assure.html : "To “assure” a person of something is to make him or her confident of it. According to Associated Press style, to “ensure” that something happens is to make certain that it does, and to “insure” is to issue an insurance policy.". I can understand hesitation between "ensure" and "insure" (AP has it right, but not everybody agrees on that), but to misuse "assure" as if it meant "ensure", as the quoted newspaper article just did, is simply unforgivable.

Scientists and Engineers

I've been described, on occasion, as a computer scientist. This is incorrect: I'm not a scientist, I'm an engineer.

Not only is my degree in Engineering, not Science: I'm an engineer to the core, because I am thrilled by making things that are useful far more than I am about discovering truths about Nature -- the latter is fine, don't get me wrong!, a wonderful pursuit, and indispensable to keep pushing our civilization forwards in the long run, but -- as my own personal pursuit, it doesn't get my heart pulsing faster, make me sit upright, make my eyes shine, anywhere as much as the making of useful things does.

I would say I'm kind of a typical engineer: just like, say, Leonardo da Vinci, Ludwig Wittgenstein, Antoine de Saint-Exupery (he couldn't actually get his engineering degree, but, he tried!), Rowan Atkinson (of "Mr Bean"'s fame), ...

Python tips: the right way to show errors

I've lost count of how many times I've seen Python code like this...:

logging.error("Unknown user '%s' for '%s'"
             % (username, resourcename))

This error-showing code has two problems:

1. logging.error and other such functions do the formatting for you -- so don't duplicate their work! So, the first level of fixing is to change the call into:
   

   
   logging.error("Unknown user '%s' for '%s'",
                 username, resourcename)
   

   
   


2. any time you're displaying a string that's somehow "in error" (not just in calls to logging functions, but also when instantiating an exception, etc), use %r, not %s! This way, if the problem is that the string contains "invisible" characters, you'll see them clearly displayed as escape sequences, while %s might hide those characters, making debugging much harder. So, the second (and final;-) level of fixing is to further change the call into:
   

   
   logging.error("Unknown user %r for %r",
                 username, resourcename)
   

   
   

Upper-class vs Lower/Middle-class English

I'm watching Branagh's "Henry V" with my wife, and (after I expressed my absolute awe at Emma Thomson's acting skill... all actors in that movie are good, Thomson upstages them all - she's AWESOME!) we stopped midway through (getting too late) and briefly discussed the play's famous structure. After the obvious notice of the alternation between "upper class" scenes (featuring kings, princes and dukes) and the "lower/middle class" ones (with corporals and, at best, lieutenants), I idly remarked how much easier it was for me (a foreigner who studied English as a second or third language -- it's debatable whether I studied more English, or French!) to follow the dialogue in the "upper-class" parts, than the others...

Well, guess what: Anna's exactly the other way 'round -- SHE follows the "lower or middle class" dialogue much better than the higher-class one! She, of course, is Minnesota (heart of the Midwest) born and bred... MY English comes mostly through Milton, Swift, Sterne, Smith, Franklin, Poe, Melville, Hardy, Shaw, Conrad, &c... HERS, though she IS uncommonly well-read for an American, comes mostly through the organic, word-of-mouth, generation-to-generation "normal" process of language transmission...

...and, it appears, the subset of English best transmitted by highbrow "culture" (e.g. to foreign-born students like me) is QUITE different from that which best survives through "natural" means (to native-born speakers) -- and the "class distinction" is just what one would expect! Reminds me of the way Latin came into Italian through two similarly separate channels: the spoken-word natural one (which e.g. made gold into "oro", laurel into "alloro", &c) AND the "high culture" mostly-written one (which gives us such words as "aureo", golden, and "laurea", the university degree traditionally celebrated with laurel crowns).

The change of classic high-class diphthong "AU" into common-speech "O" is well attested even in late-Republic Rome -- e.g. the member of the ancient Claudius family who went for unstinting populism signaled that by changing his name to Clodius!-) -- so that's a particularly good example;-). However, this general kind of distinction (between high-culture, mostly-written transmission of language, and normal-people, mostly-spoken one) is VERY common in all languages, and English, this most wonderful and most mongrel of languages, is no exception;-).

The Pedant Reigns

Robert Reich is always worth reading, content-wise... but I'd have to unsubscribe from his blog if I couldn't lower my blood pressure by venting when he slaughters the beautiful English language! In his latest post, he (rightly, content-wise) reminds the reader how the GSEs "blocked any attempt to reign in the risks". Robert, it's REIN IN, not REIGN IN.

Reins (no G in it) are what you control a horse with, and by analogy they're used to indicate such control (as in the idiom you mis-used); it comes from Vulgar Latin "RETINA", constraint, from "RETINERE", to constrain (from "re-tinere", literally "to hold again"); close cognates in English include "retain" and "retinue".

A reign (with a G) is the period during which a certain monarch is on the throne, that monarch's authority and dominion, his or her rule; it comes from Latin "REGNUM", kingdom or reign, related to "REGULARE", ``to rule''; deep down, it coms from Proto-Indo-European root *reg-, just like your own surname, and a wide variety of words such as "right", "rich", "Raja", "rector", "erect", "royal", "realm", "rule", ... which got into English through many different routes.

It IS unfortunate that these unrelated words, rein and reign, ended up in English with the same pronunciation, very similar spelling, and even vaguely cognate meanings...!-). But PLEASE think of their differences, and let your readers focus on the very important and relevant messages you're sending (in this latest blog, about the horrors of the GSEs, Fanny and Freddie, based on private profits but socialized losses), rather than risk apoplexy over your usage of English... THANKS!

PIL on Mac OS X 10.5 with Google App Engine

I have recently opened a couple of issues on the Google App Engine tracker: 685 about installing PIL on Mac OS X 10.5, which is harder than GAE's docs make it out to be, and 695 about actually getting it to work with the wonderful GoogleAppEngineLauncher (so you can run locally GAE apps that use the images API), which requires some trick. So I just thought it might be useful to somebody else who uses Mac OS 10.5 and GAE if I summarized the necessary steps and workarounds.

GAE docs claim that "Installing PIL on Mac OS X 10.4 and 10.5 can be completed in a few simple steps", but if you just follow the steps they suggest, on 10.5, the PIL installer will refuse to install -- the installer was written for 10.4 and it does not recognize the "stock" Python that comes with the OS for 10.5. As Matt Kangas has posted, before you run the PIL installer, you need to make a symlink:

cd /Library/Frameworks
sudo ln -s /System/Library/Frameworks/Python.framework/ Python.framework

He also suggests placing in your .bashrc

export PYTHONPATH=/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/

The latter environment variable (I placed it in .profile) is indeed sufficient to let you use PIL from a Terminal session, including one in which you run dev_appserver.py from the symlink that GAE installation places (if you allow it to) in /usr/local/bin. However...

Once you've done all of this, the GoogleAppEngineLauncher still won't let you use PIL (and, therefore, it will disable the images API in the GAE apps it runs locally), because it doesn't know about your .bashrc or .profile! You need to find another way to extend Python sys.path's appropriately. The simplest way is to make a file named PIL.pth with a single-line content:

/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages

and place that PIL.pth in any of the directories that all Python processes see; in my case, I used /Library/Python/2.5/site-packages/ -- but that directory is not available in a pristine installation of Mac OS X 10.5 so you may have to use some other directory.

This post is (I hope!) only of temporary usefulness, as it will be superseded when the PIL installer is rebuilt to properly support Mac OS X 10.5 (or, if that installer is never rebuilt, this post can still be made obsolete by the tiniest fix to GAE and its docs to close the issues I've mentioned above!), but, in the meantime, HTH!

The Pedant Awakes

Some are born pedants, some achieve pedantry, and some have pedantry thrust upon 'em.

I'm in the first category. I can't help it: malapropisms LEAP to my eyes, raise my blood pressure, make me feel PAIN. I've long mused that I should start a blog to vent about these painful attacks against my sanity -- the it's/its'/its horrors, the there/they're ones, "reign in" to mean "rein in", "baited breath" for "bated breath".... and so on, and so forth... venting would lessen the pain a little bit, maybe lower the ol' blood pressure a tad, too. But, round tuits tend to be in short supply.

Today's the day the Pedant awakes from his slumbers and SHOUTS -- "I'M MAD AS HELL, AND I'M NOT GOING TO TAKE IT ANY MORE". Watch out -- the Pedant awakes, the Pedant is coming!

Google Reader "share with friends" is a delightful little touch... and today my wife "shared" a generally excellent post by Garr at Presentation Zen, so of course I read that... I enjoy Garr's blog and his excellent book, and anything my wife thinks is worth sharing deserves attention.

Midway through... "Fast/slow, loud/quite" -- OUCH. I take a deep breath or three -- come on, that's no malapropism, it's a perfectly normal typo, swapping the final E and T... they can happen to anybody, they happen to me, too. Calm down. Resume reading. But, towards the end... "his son Michael Reagan eluded to today". AAAARGH!!! I stormed upstairs to my wife, shouting, DEMANDING to know WHY, OH WHY, she would ever share a post using "eluded" to mean "alluded" -- it's not as if my life insurance was juicy enough to make her WANT to cause me to have a heart attack, after all... so what WAS her motivation?!

Turns out... she didn't even notice. When the general disrespect and spite for proper usage has become SO bad to numb even Anna -- whom I love, among innumerable other reasons, for being quite a match for me as a pedant and copy-writer supreme -- when SHE fails to notice something as horrible as THAT, well... then it's HIGH TIME for the Pedant to Awake, and Stride the Blogosphere ranting and venting and whining.

So *WATCH OUT* from now on, especially if you write generally interesting and worthwhile prose, whether you actually MEAN "insure" or "ensure" (or maybe "assure"...?!), whether you're REALLY trying to speak about the "File Transfer Protocol protocol" ("FTP protocol" occurs hundreds of thousands of times on the web...) or the "North-Atlantic Treaty Organization organization" ("NATO organization" has over ten thousands hits), whether you mean "except" or "expect", and so forth... if you slip, the horrible price to pay just MIGHT be... *a mention on this blog*!!!