I found that if I created the image with caption in Visual mode as usual, but then went into HTML mode, I could enter an external link successfully.
The reason it was breaking before is that the caption treats everything inside it as a link:
Code:
[caption id="attachment_3029" align="alignleft" width="200" caption="Anant Nambiar"]<img class="size-full wp-image-3029" title="anants" src="http://www.larchmontgazette.com/wp27/wp-content/uploads/2009/04/anants.jpg" alt="anants" width="200" height="221" />[/caption]
Code:
[caption id="attachment_3029" align="alignleft" width="200" caption="Anant Nambiar"]<img class="size-full wp-image-3029" title="anants" src="http://www.larchmontgazette.com/wp27/wp-content/uploads/2009/04/anants.jpg" alt="anants" width="200" height="221" /><a href="http://www.cnn.com">[/caption]
I found a thread at wordpress.org involving a change to the footer.php code, but that seems like overkill.