PragDave

Another piece of music…

In my ongoing attempt to keep myself honest, here's another piece I wrote, again played by Mike Springer. I'm not sure how to categorize this one—I wanted to play around with 7/8, and ended up with what I think are some interesting patterns. Anyway...

And here's the music

The Passionate Programmer

I've known Chad Fowler for almost ten years. He's a seriously good guy: musician, developer, leader, and friend. So when he came to me some years ago with an idea for a book, I jumped at it. I was hoping for something good, but what we got was something truly great.

Andy and I wrote The Pragmatic Programmer over 10 years ago. It was a book full of advice on the job of programming, from low-level coding to teams and projects.

Well, Chad had written the companion book. Where our book looked outward at the job you were doing, The Passionate Programmer looked inward at the person doing the work. It's a book about finding fulfillment in what you do. It's a book about rekindling the fires that you felt when you first entered the profession. It's a book about creating a career, rather than turning up for a job.

I was blown away.

And then I did something really stupid. The book came out about 5 years ago, and outsourcing was big on everyone's mind. So I thought "let's make a jokey title that'll grab people's attention." I called the book My Job Went to India (And All I Got Was This Lousy book).  I've already blogged about how stupid that was—it was probably the single dumbest thing I've done since I got into publishing. The title put people off; it gave them the wrong idea. The book was really nothing to do with outsourcing, but you wouldn't know it from the cover. This book deserved so much better

So, four years later, we revisited the book. Chad's given it a spring cleaning, revising and updating it. And we gave it a new title and a new cover, both of which do a better job of conveying the energy and enthusiasm that lie within.

I'm not into selling: my philosophy with our books is “if they're good, people will buy them.” But this book is different. I really think it can make a difference. So I'd love for you to read the extracts and maybe take this book home with you. I honestly think you'll thank me (and, most importantly, Chad) if you do.

Amazon Tighten Their Grip on Kindle Distribution

I've had an on-again, off-again relationship with my Kindles. The original was not much more than a toy, but the Kindle 2 was a more usable device. It still needs work in the formatting department (hopefully the Lexcycle acquisition will help with this—I very much hope to see epub support rolled in to the Kindle at some point courtesy of the folks who brought us Stanza). But the distribution model is a good one. If you buy a book from the Amazon store, it gets delivered over the air to your Kindle.

Now, the Pragmatic Bookshelf doesn't participate in the Amazon eBook store: it is frankly too expensive for us, and we value having a direct relationship with our readers. So when we announced our support for the Kindle, we did distribution from our own site: you download your .mobi file from us and drag-and-drop it onto your Kindle via USB.

But… Amazon also let you send files to your Kindle via e-mail. In the old days, this used to cost $0.10 per file. So we modified our store. If you have a Kindle, you can register it with us and we'll send your eBooks to it automatically. If we release a new beta, it'll get sent to you, and we'll send you an e-mail to let you know it's there. It was a nice compromise, and at 10 cents a pop, not a bad proposition for our readers.

But then, yesterday, Amazon changed the rules. They now charge for transfers on a metered basis. Send a file to your Kindle over the air, and it will cost 15 cents/megabyte. Suddenly what once looked attractive now starts to become expensive. If we update a beta book 10 times, and the .mobi file is 5.5Mb (about our average), then you're out of pocket $9. Because we don't want our readers to get hit by surprise Amazon bills, we'll probably remove the feature from our store over the next few days (let us know if you think we should keep it).

Color me paranoid, but I suspect, that's exactly what Amazon wants publishers to do. Of course, they could argue that it's fair for readers to pay for bandwidth usage, but in reality I suspect that these transfers are a vanishingly small percentage of their overall network usage.  Instead, Amazon probably realizes that this facility allows publishers to bypass their store and have direct relationships with clients. And Amazon d0esn't want that. This charge is just enough to discourage most readers to sign up with publishers such as ourselves to receive automatic updates to the eBooks they own.

I think that's a shame. We were looking at an entirely new model. Imagine how cool it would be if your books updated themselves while you slept. Now we'll have to wait for another device to become popular before that becomes a reality (Apple tablet, I'm looking at you…).

In the meantime, if you own a Kindle, and you'd like to receive automatic updates on it without having to mortgage your house, you might want to consider asking Amazon to reconsider this charge. I don't hold out much hope, but you never know.

Displaying Code on the Kindle

So, I'm in a quandary.

Having used one now for a few months, I'm slowly warming to Amazon's Kindle 2. Sure, it still feels a bit cheap compared to the Sony 505, but it's fast, and the ability to download over their wireless network is a bit plus. (Stay tuned for an announcement from us about that...)

But my biggest issue with the Kindle is the markup it supports. Think 1995 browser, minus the <blink> tag. It is very, very limited.

I pride myself on the look of our books, so when I added support for eBooks on the Kindle, I worked hard to get something that looks half-way decent. We're still tuning it, but by and large I think our books look good. But we still have an issue. It is really hard to display code listings on the device. Let me explain why, and show you some examples. Then, at the end, I'll ask for your help.

First, the problem. When the Kindle was first launched, it lacked a fixed-width font. They recently added one. But the choice of sizes is limited, and the code tends to end up too big, even at <font size="-2"> This leads directly to the main problem. The Kindle display is narrow. You can get roughly 45 fixed-width characters across it at the smallest font size. That means code lines will either be truncated, or they'll wrap.

But, we went ahead and did our best. And we didn't much like the result. In particular, I didn't like the way long lines in the code wrapped onto the next line. So I went back to the drawing board and took a new approach.

In the books as currently published, code listings on the Kindle are actually images. That means I get to choose the font, and I get to choose the size. And I can do some cleverer things than I could do with straight HTML formatting. For example, if a listing is wide, I know before hand that it won't fit the horizontal size of the screen, so I scale the font down until it does. And, because the font I'm using is narrow, I can fit more code per line anyway. But there are some major downsides of this approach.

• The code doesn't scale up and down when you change the font size on the device.

• The font we're currently using in the images doesn't have bold and italic versions, so we're not syntax highlighting it. This is clearly fixable, but it would be something of a pita to do.

• The .mobi files we generate are about three times bigger than they'd otherwise be because of the embedded images.

• If an image is too big to fit on a page vertically, we currently chop it off.

So, there's no clear winner. Just to show you what I'm talking about, here are some comparison images taken from Daniel's Cocoa book. The ones on the left are pages from the books as we currently distribute them. The ones on the right use text, rather than images, do display code. (You can click on an image to see it at full size.)

Code As Image	Code as Text

So, given the pros and cons of each approach, what should we do? Stick with the current scheme? Move to text-based code? Or...?

Twitter Should Move Away from Ruby

Oh dear. The chattering classes are at it, talking about how the Twitter folks are dissing Ruby by announcing the replacement of some Ruby code with Scala code.

Please stop.

At the kinds of volumes that Twitter handles (and with what I assume is a somewhat scary growth curve), Twitter needs  to improve concurrency—it needs an environment/language with low memory overhead, incredible performance, and super-efficient threading. I don't know if Scala fits that particular bill, but I know that current Ruby implementations don't. It isn't what Ruby's intended to be. So the move away is just sound thinking. (I suspect it also took some courage.) I applaud Alex and the team for this.

Instead of defending Ruby when it's clearly not an appropriate solution, let's think about things the other way around.

The good folks at Twitter started off with Ruby because they wanted to get something running quickly, and they wanted to experiment. And Ruby gave them that. And, what's more, Ruby saw them through at least two rounds of phenomenal  growth. Could they have done it in another language? Sure. But I suspect Ruby, despite the occasional headache, helped them get where they are now. 

And now they've reached the status of world-wide wunderkind, it's time to move on. 

I for one wish them luck. I look forward to the day when our online store reaches the kind of size where we have to move away from Rails. I'll tweet the fact with a tear in my eye, while my yacht sails me off to the sunset.

Using comments as rake task descriptions

Over lunch at Scotland on Rails, Jim, Chad, and I got talking about the way you use desc to document tasks in rake.

desc "Remove intermediate files and other work products"
task :cleanup do
  ...
end

This has always been an interesting feature of rake—the idea that you have a kind of temporal coupling between one method call and the next. It works well, but somehow it's always struck me as a little obtrusive.

So, during the afternoon, I hacked up a quick change to rake that looks for a single-line comment on the line prior to all task, file, etc directives, and uses that as the task description. With this version of rake,  you can write

# Remove intermediate files and other work products
task :cleanup do
  ...
end

The comment becomes the description of the task, because it occurs before the task directive with no intervening blank line. desc still works, and if both desc and a comment are present, the desc takes priority. It's purely a matter of taste, but I personally find the comment form more natural.

You can get this patched version from http://github.com/pragdavespc/rake/tree/master. I've passed the patch on to Jim, so if you like the idea, let him know and maybe he'll roll it into the next rake release.

Shipping the Rails Book

Last week we completed our largest direct shippment, as we sent out the preorders for Agille Web Development with Rails, Third Edition. We had tons of books delivered, and bought tens of thousands of dollars of shipping supplies and postage, but despite the volume everything went well (apart from the inevitable paper cuts). I even learned where the big boys go to the Post Office (which turns out to be a pretty impressive facility right next to DFW airport).

So I thought I'd share just a couple of statistics. Domestically, we shipped books to every state except North Dakota. Internationally we shipped to three US bases, and a total of 55 different countries (from Angola to Vietnam). Sometimes I close my eyes and think of the trajectories of all those books spreading out from Raleigh and Dallas—it's a cool picture.

So, if you bought a Rails book from us: thank you! It's been an interesting year (way longer than would would have liked, but that's the joy of working with a rapidly changing technology). We hope you enjoy the result. And, remember, we now have eBook formats  available for your Kindle, iPhone, and so on.

So I need a symetric coroutine example

I'm getting close to wrapping up the new PickAxe. One of the last things I need is an example of symetric coroutines for the standard library section on the Fiber class.

I have lots of asymetric examples, but I'm struggling to come up with something decent for symetric coroutines (which use transfer to pass control between themselves.) I've coded up Conway's line squeezer, but it works better using asymetric coroutines. I've tried to come up with puzzles that are best solved with symetric coroutines, but without luck.  I've coded up a simple blackjack game, but again it works better with asymetric coroutines. Although, for fun, here's the dealer, which shows some of the new Ruby 1.9 array methods:


But, I'm still stuck. So... can any of you clever folk come up with something compelling that uses less that 35 lines of code?

Rails Studio Early Registration ends soon

View Full Size on Viddler

I love doing the Pragmatic Studios. Nicole, Mike, and Chad are all good friends, and that creates a really relaxed and fun atmosphere for what could otherwise be a pretty intense three days.

The next studio is in Denver at the end of January, and I'm really looking forward to it—living in Texas, I relish being able to see mountains and snow. I just wish I was a skier so I could have an excuse to take an extra day or two to head on up to a resort.

If you are planning on going, though, don't put it off—early registration (a $300 savings) ends in a week.

There's one bad side to the studios, though. Mike put together a video showing what studios are like. Notice anything? Food. Lots of it. Scarily good food. Must resist. Must…

Little bit of musical fluff

I decided a while back to publish the music I'm writing, if nothing else as a way of keeping myself honest. So, here's a little piece of fluff I wrote just to play with time signatures—alternating 5/8 and 6/8 has a nice drive to it.

But, I have a problem: I can't actually perform any of the pieces I write, so I'm somewhat at the mercy of the good will of others (notably my teacher) to record them for me. And Christmas is a busy season for my teacher, so I don't yet have a recording of this piece.

Because this piece is so simple, Sibelius' built-in playback sounds OK, so that's what I'm posting. But if anyone fancies recording this, or taking it and turning it into something cool (as Chris Morris did with my previous piece) please send me a link and I'll add it here.

In the meantime, here's

• the score,
• the Sibelius mp3, and
• a midi version (don't listen to this—it sounds horrible. It's just here for folks who want to import to a sequencer)

Update: here's my teacher's rendition—thanks, Mike!

Clearly I've missed an opportunity with our book covers

Like the rest of our publishing business, I like to keep our covers simply and easy. But now I realize that these covers fail to capture the true epic nature of our books. Joey deVilla posted some reworkings of book covers last year, but I only came across them today. I'm inspired...

The strange thing is: I'm not sure where he got hold of the image. After we posed for the picture, DHH and I thought better of it. We thought we had destroyed all the copies...

Ruby 1.9 can check your indentation

All Ruby programmers regularly encounter the mystical error “syntax error, unexpected $end, expecting keyword_end.” We know what it means: we left off an end somewhere in the code. As Ruby compiled our source, it keeps track of nesting, and when it reached the end of file ($end), it was expecting to see one more end keyword, and none was there.

So, we trundle back through the source, and after a while discover we'd deleted just one too many lines during that last edit.

Ruby 1.9 makes that easier. For example, here's a source file:

class Example
  def meth1
    if Time.now.hours > 12
      puts "Afternoon"
  end
  def meth2
    # ...
  end
end

Run it through Ruby 1.9, and you'll get the same old error message:

dave[RUBY3/Book 8:26:48*] ruby t.rb  
t.rb:10: syntax error, unexpected $end, expecting keyword_end

But add the -w flag, and things get more interesting.

dave[RUBY3/Book 8:26:51*] ruby -w t.rb
t.rb:5: warning: mismatched indentations at 'end' with 'if' at 3
t.rb:9: warning: mismatched indentations at 'end' with 'def' at 2
t.rb:10: syntax error, unexpected $end, expecting keyword_end

It's the small things in life...

Names of encodings built in to Ruby 1.9

I've heard of about 10% of these...

What an amazing piece of work.

What's on my mind...

Forking Ruby—my RubyConf Keynote is now up

There's a sound that no presenter wants to hear, and that's dead silence. And that's what greeted me when I made the suggestion in my RubyConf keynote that the community should fork the Ruby language. I think by the end of the talk, though, most people were convinced.

Am I anti Ruby? no.

Am I suggesting Matz is doing a bad job? Not in the least.

But I do think the complexity of the current language inhibits experimentation with language features. Want to implement parallel Ruby? If you do it with the current syntax and semantics, you'll be struggling with the integration of global variables, the non-threadsafe nature of require, continuations, and so on, and so on. So before starting these kinds of experimental projects, I'm saying we should fork the language. Produce variants that let us focus in on the aspects under test. Make the new language a rich-enough subset that you can do useful work with it. Then let people play with it. Maybe the ideas are stellar, in which case we can all talk about integrating the changes back into the mainline language. And maybe the idea didn't pan out, in which case we can quietly forget it, and we've done no damage to the language itself.

A mutation is an altered fork of the original. And mutations are essential to the diversity and strength of a genetic line over time. Let's not be afraid to create Ruby mutants and let them compete.

The keynote is available from Confreaks.

See Rails request paths in 'top'

During our sale, we had one particular request that came in and wedged the application: every time it hit, the mongrel process size zoomed steadily up to 500Mb, so we had to kill it. But finding out which request was doing this was tricky. The log files didn't help—with the amount of traffic we were getting, it was a small needle and a large haystack.

Eventually, we found the culprit. But it would have been a lot easier if I'd thought of this hack on Friday, and not after the sale ended.

If you put this into your application controller:

before_filter :set_process_name_from_request

def set_process_name_from_request

  $0 = request.path[0,16] 

end   

after_filter :unset_process_name_from_request

def unset_process_name_from_request

  $0 = request.path[0,15] + "*"

end  

then Ruby will set the cmd field in your process control block to the first 16 characters of the request path. You can then use top to see what request is being handled by each mongrel.

Once the request has been handled, an asterisk sign is appended, so you can see the last URL when a mongrel becomes idle. 

If your version of top doesn't show the short command by default, use the c keyboard command to see it.

This is probably common knowledge, but I thought it was cool.

Just in case you missed it...

We're having a sale on Friday...

Ruby 1.9.1 Preview 1 is out

The path to a stable Ruby 1.9 has been long. Almost a year ago, 1.9.0 was released. This turned out to have been less than stable. The core library has grown by many hundreds of methods since then. And, probably more significantly, people have started using 1.9 for real, and they've discovered rough edges that needed to be fixed. In particular, the multinationalization support, which is incredibly ambitious, turned out to be hard to use for library writers. James Edward Gray II was probably the first to bump into this as he updated his FasterCSV library (now part of core Ruby) to handle any encoding supported by Ruby. His problems lead to a lot of discussion, and eventually to an entirely new concept inside the interpreter. And as a result, the core team have decided to slip the release of 1.9.1 by at least a month while they investigate other encoding-related issues lurking in the libraries.

In the meantime, a 1.9.1 preview has been released. Details are here.

If you are the maintainer for any publicly available Ruby code (be it a Gem, an application, or whatever) I strongly urge you to download this preview. You'll be doing the community a great service in two ways. First, the various incompatibilities between 1.9 and 1.8 mean that there's a chance that your code may not work without some tweaks. Making those changes now will help others using your code. As importantly, by using 1.9.1 for real, with real code, you'll potentially discover other rough edges. Reporting these back through ruby-core or RedMine will help the Ruby developers further hone the interpreter.

Now, all this delay leaves me with a problem. The new 1.9-specific PickAxe is now content complete. Some people have been waiting for it for 10 months. And I have to decide: should I hold on for the official 1.9.1 release (which will be, at the earliest, at the end of January 2009) or should I send it to the printers? Let me know what you think.

Fun with Ruby 1.9 Regular Expressions

I've reorganized the regular expression content in the new Programming Ruby, and added some cool new advanced examples. This one's fairly straightforward, but I love the fact that I can now start refactoring my more complex patterns, removing duplication.

The stuff below is an extract from the unedited update. It'll appear in the next beta. It follows a discussion of named groups, \k and related stuff.

---

There’s a trick which allows us to write subroutines inside regular expressions. Recall that we can invoke a named group using \g<name>, and we define the group using (?<name>...). Normally, the definition of the group is itself matched as part of executing the pattern. However, if you add the suffix {0} to the group, it means “zero matches of this group,” so the group is not executed when first encountered.

sentence = %r{ 
    (?<subject>   cat   | dog   | gerbil    ){0} 
    (?<verb>      eats  | drinks| generates ){0} 
    (?<object>    water | bones | PDFs      ){0} 
    (?<adjective> big   | small | smelly    ){0} 

    (?<opt_adj>   (\g<adjective>\s)?     ){0} 

    The\s\g<opt_adj>\g<subject>\s\g<verb>\s\g<opt_adj>\g<object> 
}x

md = sentence.match("The cat drinks water") 
puts "The subject is #{md[:subject]} and the verb is #{md[:verb]}"
 
md = sentence.match("The big dog eats smelly bones") 
puts "The adjective in the second sentence is #{md[:adjective]}" 

sentence =~ "The gerbil generates big PDFs" 
puts "And the object in the last is #{$~[:object]}" 

produces:

The subject is cat and the verb is drinks 
The adjective in the second sentence is smelly 
And the object in the last is PDFs 

---

Cool, eh?

The iPhone Book is in Beta!

We've heard back from Apple. We've double checked with the authors. And the iPhone book is in beta.

After a rocky start, I have to say we've had nothing but help and support from folks in Apple. And eventually the senior management listened to the community and did the right thing. Thanks to our friends in Apple for their help, and thanks to everyone for your support and patience.

Enjoy the book. And write some killer iPhone apps.

Dave