Code Vs Data Vs Getting Something Done

I work a lot on software automation, which means writing a program to do a task that otherwise would have to be done by someone by hand. The best tasks are those that are repetitive and boring. If it's repetitive, it means you will get a lot of bang out of the automation. If it's boring, it means there are no complex decisions involved and that there's a good chance of successfully automating the task. There are some tasks that fit this criteria that I don't tackle, like driving to work. This can be done and in fact there are probably prototypes of solutions -- self-driving vehicles. But that's not the problem I'm trying to solve, perhaps because no one has offered to pay me to work on it. (Why would anyone do that when you can get a bunch of grad students with no personal life to work on it at a much lower cost?)

The problems I do work on are related to systems configuration, data entry, data translation and of course, quality assurance.

One problem that crops up a lot in this arena is the problem of name/value parameters. For example, I have an application that will let me configure and schedule programs for automated, distributed execution. But to configure a single program to run within this application (really a collection of applications) may take over one hundred unique settings: everything from the display name of the job to a list of days that I don't want the job to run. The application provides a GUI for entering these things, but believe me, you only want to use this approach once or twice. It's really tedious clicking and typing your way through ten or twenty dialogs to set or verify a hundred parameters.

So fine, I can write a program that can just push into the application all the settings I want through a handy-dandy interface. In most cases, the programs are similar to each other, so for over a hundred parameters, I may only really care about five or ten. I can reuse all the other settings each time, but I'd still like to be able to override those defaults when the need arises.

But how to do this? Perhaps this sounds like a job for OOP. I could start with a base class and specialize that class for the variations. But OOP is about variations in behavior, not variations in state. So really this is the problem of the name/value pairs. In other words how to associate various placeholders for state (the names) with actual, unique states (the values). If the number of name/value pairs is small (less than five), then the answer is easy: pass them to the program as command line parameters.

If the number is large, then there are two alternatives: put the name/value pairs in your code, or put them in data. But even among these two alternatives, there are many interesting choices to consider.

If you do it in code, you could write something like the following:

n_v_table: ARRAYED_LIST[TUPLE[STRING, STRING]] is
once
create Result.make_from_array(<< [ "Name 1", "Value 1"],
[ "Name 2", "Value 2"],
...

Obviously this has several drawbacks, such as being hard to type and forcing you to recompile when anything changes. Recompiling becomes something you want to avoid doing, especially with Eiffel. In the same vein, I could replace entries with constants, or even code:

n_v_fancy_table: ARRAYED_LIST[TUPLE[STRING, STRING]]
do
create Result.make_from_array(<< [ "Name 1", some_constant],
[ "Name 2", get_some_value(with_some_parameter)],
...

but that's really just pushing the problem around in the code. It does however leave open the next alternative, assuming that we can rewrite get_some_value to do whatever we want. That alternative is to put the name/value pairs into a data file. People often end up using the old standby .INI format:

Name1=Value1
Name2=Value2

This too has drawbacks. Forget about doing anything "smart", like calling code to get a value. It also means you have to write a parser to read in and validate the input. Bleagh. You can potentially overcome some of these limitations by making your parser smart enough, and adding special constructs for what gets parsed. One horrible yet entirely possible solution is to denote a reference to a constant or variable with a dollar sign, and a function call with two dolllar signs:

$some_constant="constant_value"
Name1=$some_constant
Name2=$$get_some_value($with_some_parameter)

Before long, one begins to fully believe in Greenspun's tenth rule: Any sufficiently complicated C or Fortran program contains an ad hoc informally-specified bug-ridden slow implementation of half of Common Lisp.

Though I would say this holds true for any statically compiled program, not just for C and Fortran.

If you're working with an dynamic, interpreted language, another alternative becomes possible. Use that language to write out the name/value pairs, and then implement your program so that it loads the data and evaluates it as part of itself. I was introduced to the full power this technique when working as a contractor for Bivio. Prior to that I'd skirted around the edges of this approach on my own, but the Bivions took this much further. Bivio's language of choice is Perl. I don't consider Perl to be an optimal language to work with because of the syntax, but it's powerful, compact, and has an enormous library/support base.

Going back to the example above, we could write the name/value pairs into a file like so:

my %table = (
Name1 => 'string constant',
Name2 => $variable,
Name3 => function($parameter),
...


Then we write a Perl command that understands what %table is all about. It loads the file and (maybe) wraps it in some other Perl code, and then does an eval on the whole thing.

Now we have something really interesting. The Perl code can load nearly arbitrary data and act on it as if it was compiled into the program itself. All we have to do, to do something new, is create a new fil0e of name/value declarations and run the command across it. Usually these declarations will be much simpler to deal with than it would be to write a normal Perl script. In a way, this falls into the notion of a mini-language, or application-specific language. The syntax for this language is the same as the interpreter's, but the semantics are specific to the given problem domain.

I've recently begun working with Python, and of course it has its own version of eval(), and it also lends itself to this style of programming. For certain focused utilities, this is an approach that is pretty hard to beat.

Restaurants and Groceries

Wine Country

Mustard's
On Highway 29 just north of Yountville, it's been around for 20 years.

Pizzeria Tra Vilne
St Helena
Allegedly has a way of serving a Ceasar Salad inside a pizza crust. Okay, I gotta see that.


Indian Hot Springs
Callistoga

Berkeley / East Bay

Viks Chaat Corner
726 Allston Way
Berkeley Ca
510-644-4412
Funky but tasty and authentic Indian food

The Left Bank in Pleasant Hills, near the Borders Bookstore.

A Cup of Tea in Berkeley, on Alcatraz near College

San Francisco Restaurants

Amber India Offers a Sunday brunch?

25 Yerba Buena Lane

San Francisco CA

415-777-0500

Dong Baek Korean Restaurant
631 OFarrell St
San Francisco Ca
415-776-1898
Near Union Square

Chutney Restaurant
511 Jones St, San Francisco, CA 94102
An Indian-style restaurant that we may have gone to a lot in the past (or maybe it's the one next door).

Treo == 666

My company gave me a Treo that's hooked up to my office email. I literally get 4,000 or so emails a day, and after heavy filtering with the incredibly pinheaded Outlook filtering rules,  I can still be getting two email alerts a minute, on average.

It took me two weeks to find the magic handshake that makes it not tell me about alerts. So imagine this. It can't be turned off (it can be put to sleep, but that's not the same). When the was sound is on, it kept chiming all the time. If I turned the sound off, it did this death-rattle vibrating buzz. I could take it offline so it wouldn't get the alerts, but then putting it back online (say to dial 911?) it would lock up doing chimes or death-rattles for 20 minutes while it received a huge backlog of email alerts. So during those first couple of weeks, I gave my new Treo a nickname:

ALF, the Angry Little F***er.

This caught on with friends and family pretty quickly.

What I Would Like in a Technical Documentation Editor

Motivation: I've begun writing an introduction to the Eiffel programming language. Being very constrained in how much time I can dedicate to this, I'm frustrated by the tedium of developing the text and the examples in parallel, and trying to keep the two in sync. It would be great if I could develop the code examples outside of the document and then integrate them into it seamlessly, or if I could develop the code in the document and then have the examples automatically become test programs, or even better, go both ways.

It should support the evolution of the code as I write the book, and the evolution of the examples as the code progresses through the book.

In some ways, this like literate programming, but goes deeper, since I want to iterate over the examples and refine them.

1. Ease of use -- of course
2. The ability to extract entire programs from the document and automatically compile and test them.
3. Cohesion of code fragments. When a fragment is repeated in different sections, every place that fragment is referenced is properly updated. In other words, rather than the code itself, what's in the document is a reference to the code that gets expanded appropriate for display purposes?
4. Appropriate styling of code. Reserved words should be one format, regular code another, comments italicized, etc.
5. Wiki-like reference support. In addition to styling the text, I'd like the reserved words to become links to documentation that explains them, or explains the programming construct(s) that use them, or both.
6. The Wiki-like references can go deeper. When the text makes a reference to a concept like "Boolean expression" then the text should automatically become a reference to the explanation or sidebar about boolean expressions.

In other words, I would like an editing tool that can read my mind, is extremely efficient for generating intertwined code and documents, that makes optimum use of the hypertext capabilities of electronic publishing, and can then coerce documents and code into a format that's completely acceptable for printing.

The tool (or suite of tools) should automatically verify the integrity of the document in parts and as a whole.

Welcome to China!

We arrived at Dalian (a large harbor city close to Beijing) on a cruise ship in late one evening in October, 2005. Berthed next to us is a barge unloading scrap metal. A huge claw descends into its hold, pulls out a load and the drops it on a large pile with a horrendous clatter.

Across the dock from us stands a squalid, three story building in desperate need of repair. Weeds sprout from cracks on the trash-strewn roof. Raw brick and mortar stand exposed where the plaster has fallen away. Rust stains weep down the walls.

Below us handful of Chinese soldiers patrol the dock the gangway. They stand alertly despite the late hour, the cold, and the dim lighting.

Somone familiar with mainland China says that our captain must not have properly bribed the inspectors with food, wine and red bags. But who can blame him? To get into Japan or Korea, only three customs inspectors came aboard. Since China has many more people, they apparently also have many more inspectors; they sent thirty. It was a little humorous watching them tour the ship. They looked more like tourists than the passengers. So they had to eat at the buffet line, and didn't get any free hootch, and to show their displeasure we had to dock at this garden spot instead of the much nicer berth at another pier. Maybe the captain would have hospitably received a small delegation; hosting a busload would be off-putting.

As I watched the guards mill about and the barge being unloaded, a shiny white car raced down the dock and close to the gangway. "Ooh, someone important has arrived."

No one got out for a while, then the driver emerged. He was here to pick up someone important, not drop them off. He paced the length of the ship. He didn't see the passengers watching him from above (cruise ships are pretty tall). As he drew closer to the ship, he boosted up on tippy toe and tried to peek in through the lower portholes. "What's the pervert looking at?" I asked, sotto voce. My fellow passengers laughed. The driver noticed us and wandered away from our ship.

Another passenger asked why they needed soldiers on the dock. "They are concerned that we may try to sneak off the ship and illegally migrate to this Worker's Paradise."

Before disembarking I had to fill out a health questionnare (have you ever been treated for psychosis? Have you ever been diagnosed with a venereal disease?), and when I handed the form over to those kind and caring Chinese officials, they demonstrated their concern for my health by taking my temperature.

To take our temperatures, they used a little thermal scanner waved over the right thumb. I'm left handed, and was corrected when I offered the wrong thumb. Maybe these scanners are more accurate with right thumbs than left? I wondered what would happen if I clutched a bit of ice in my hand before getting scanned. "This fellow can't go ashore. He's dead!"

Of course, once you get past the officials and the ugly dockside architecture, the city is quite nice. And I have to say, all the passengers I saw and talked to took the whole thing very much in stride. It's almost as if they felt sorry for these poor benighted Chinese officials, stooping to such silly behavior.

What I Did Today

Today I have developed a new branch of mathematics related to set theory. I call it infinite transgressions.

The California Real Estate Market

I originally wrote this post in 2006. We've hit the major correction I mentioned in the article, but I think many of these issues are still relevant, given a market that is defined by high costs and low liquidity.

When we first moved to the Bay Area, we searched for a new home for several years. The real estate market had a single, unwritten rule:

Rule #1: Screw the Buyer

I've seen what must be hundreds of houses and I've talked with dozens of real estate agents. I finally found one broker who was willing to work with us as an agent's buyer for a flat fee, on the premise that my wife and I would do all the legwork. He'd only be involved in writing offer letters and any final contract negotiation. This was perhaps one of the smartest moves we could have made.

Here's how buyers get screwed in this market.

1) Your agent doesn't really represent your best interests. Buyers' agents are paid a percentage of the gross. They have a strong incentive to get the deal done as quickly as possible, not to get you the lowest possible price on a house. We've had agents pressure us to make offers way above asking "to be sure we got in while we could". In one case we backed off from a deal where the agent tried to strong-arm us into increasing a good offer by another $50,000. Angry, we walked away. The house eventually sold for $50,000 less than what we'd originally offered.

2) Agents on both sides can withhold and distort price information. A free market is based on the free exchange of information, or at least I thought so. If real estate is a free market, then as a potential buyer shouldn't I be allowed to know what other bids have been placed on a house, and the amount of those bids? Apparently not. In fact, I was told by one seller's agent that it's unethical to provide this information. Less profitable perhaps, but unethical?

3) The press pumps up buyer's frenzy. Reading the papers here (before 2008), you'd think every house you walk into is going to be a multiple bid death-match, with the biggest wallet winning. Horror stories of twenty offers and final prices 30% above asking abound. Yet on the ground, this doesn't happen all that often, and there are certain segments of the market that are soft.

4) The construction cost here runs $100 to $200 per square foot. This is greater than the retail cost per-square-foot of many other regions. Labor costs may be a bit higher, as well as insurance and permits, but that much more expensive?

4) The state and communities of California use zoning laws and restrictive ordinances to help keep prices high. Land is relatively scarce, but zoning forces houses to be single occupancy. So if a decent-sized lot costs $600K, the builder has an incentive to put up the largest single family home that zoning will allow, and sell it for $1.3 million to maximize his profit. Changing the zoning laws would reduce sprawl and increase available housing, but that would negatively impact real estate values. Can't have that.

Here, the tax rate on a home is based on the last purchase price. The higher the price, the higher the tax revenues. In fact, unless I get very lucky when I finally do buy a home, I'll be paying more in property taxes than I will be paying in state income taxes.