Tsaiberspace

Migrating calendar data was supposed to be easy compared to migrating Notes and Contacts data. The unanticipated problem was that Palm Desktop v6 stores its data in some proprietary format that appears to be a Microsoft Access Database file, and it “exports” that data into another proprietary format called a “Datebook Archive”.

After much blood, sweat, and tears, this is how I got 6000 calendar events out of a Palm and into Google Calendar (and thus into an iPod Touch):

• Export Palm Datebook data as a vCalendar file:
  • Download and install Palm Desktop v4 in a clean environment – one not polluted by Palm Desktop v6. In my case, I installed Palm Desktop for Mac.
  • Hotsync the Palm into this Palm Desktop v4.
  • Launch Palm Desktop and start the Datebook.
  • Export as vCal.
  • Close and uninstall Palm Desktop; you’re done with it.
• Convert the vCalendar file into an iCalendar file. I used Apple iCal.app to import a vCal file.
• Export the data as an .ics file. I used Apple iCal.app to export its data, which will create an .ics file.

At this point, you should theoretically be able to upload this .ics file into Google Calendar and be done. If you have a small calendar, then this will probably work. Stuff that didn’t work:

• dba2ics promised to convert my Palm Datebook archive into an iCal file. It got a lot of stuff, but it also missed a lot of stuff.
• Apple iCal can read the vCal export, but has some bugs where it transforms the vCal data into something else:
  • All-day single-day events in the vCal file get transformed into two-day events in iCal.
  • There is some kind of leap-year-ish bug where a Mar 1 event in the vCal file got transformed into a Feb 28 event in iCal.
• I tried many different ways to get the iCal.app data into Google Calendar:
  • Create a CalDAV calendar (pointing to Google Calendar) in iCal. Copy-paste all the iCal events into the CalDAV calendar. This just froze iCal.app.
  • Use the Google Calendar ‘import’ dialog to receive the .ics file:
    • There is an undocumented maximum-file-size limitation of 1MB or so.
    • After splitting my .ics file into smaller pieces, Google Calendar barfed on something and presented me with an empty error dialog box.
    • Also, when uploading these .ics files, Google Calendar also reported API-call quota limits.
• Google Calendar itself has its own quirk where recurring events appear to only get about 720 recurrences after which they stop (for DAILY events, this is about two years).

I wrote some python code to insert calendar events using the Google Calendar API. This unsurprisingly quickly encountered API-call quota limits, but now I could see that the limits were on the order of 5000 API calls per day “burst rate”, or 1 API call every 10 seconds sustained.

Fellow Palm refugees: you can find icalutil at github.com; I hope you find something useful.

The Palm “Memo” app is much better than the built-in iPhone “Notes” app, except for the small matter of PalmOS being discontinued. My requirements for the iPhone “Memo” app replacement:

• Categories – or “folders”, or “tags”, or whatever. Single-tags is sufficient, multiple tags per memo is not required.
• Per-note password protection. The Palm “Memo” app allows for individual notes to be password-protected so that one doesn’t have to password-lock the entire iPhone.
• No cloud-service requirement. The existence of personal data in the Palm Memos makes synchronizing with a cloud service a deal-breaker. Notably, this excludes Google Docs (popular with many apps) and the apparently-popular EverNote, Notespark, and Appigo/Toodledo.com apps.
• Data Portability. I need to import 400+ memos from my Palm.

The app I settled on is called Notes+, which at $1.99 on iTunes is very reasonably priced compared to other apps. Aside from meeting all of the above requirements, it features a freely-downloadable companion desktop application (a super-stripped-down Palm Desktop “Memo” application) which can synchronize with the iPhone, and, most importantly, stores its data in a very developer-friendly SQLite3 database.

Palm Memo notes can be exported by Palm Desktop 6.2 as a CSV file.

I wrote memo2notes.py to read this CSV file and put its contents into the Notes+ SQLite3 database. Finally, I synchronized the desktop app contents into the iPhone.

The Notes+ app itself is fine; I only have a few small complaints:

• Category navigation is not as good as in the Palm app. It has the same “category” button at the top of the screen. On the Palm, tapping this button yielded a dropdown for category selection (either to change the browsed category, or for changing the category of the viewed memo). However, in Notes+, clicking this button brings up a category chooser at the bottom of the screen, requiring that the finger move from the top of the screen to the bottom of the screen.
• The index of notes doesn’t use screen real estate as efficiently as in the Palm. In the Palm, the index view uses a single row of text per note. In Notes+, each note in the index takes up two rows: the subject (just like in Palm), and a second row for the note’s category. This is somewhat understandable because having only a single row would make things difficult to hit with a finger (since iPhone users don’t typically uses styluses). But it would have been nice to have a single-row option.
• Starting up the app is delayed with a splash screen. It looks nice, but it gets in the way, and is seems gratuitous. The app is already paid for, it’s not like this is a free app trying to upsell you to a paid app, or showing some advertising.
• On Windows, the desktop app doesn’t minimize into the system tray; it takes up Taskbar space. It is nice that you can leave this app running so that the iPhone can back up its data without having to be plugged into iTunes. Unfortunately for Linux geeks like me, there is no “headless” non-GUI version of this app.

Other Palm refugees will hopefully find this tool helpful.

(I am not affiliated in any way with SEPV Corporation, just a satisfied customer.)

I was recently tasked with migrating a long-time Palm user’s data into a iPod Touch, using a Windows PC. The important Palm apps/data:

• Address
• Date
• To Do
• Memo

These would be rather simple to do, except that Palm has done a great job integrating the “Categories” feature into all their apps. The use case is simple:

• Each app’s data can be tagged with a “category”. For example, each Address entry can be categorized as “Personal”, “Work”, etc. (just one category, this is not web-2.0 “tagging”).
• To look up an entry on the Palm, simply click on the desired category, and browse/scroll to find the desired entry. On a PDA, click/browse is much faster than search/type, because data entry is just pitiful, whether it is a touch-screen keyboard, graffiti, or anything other than a full-sized keyboard.

Unfortunately, it is not trivially simple to preserve this “category” information when exporting this data out of Palm Desktop and into something else.

The first step is to export all of the Palm Desktop data:

• Go to “Address”
• Select a Category
• Select All
• File… Export vCard…
• Save to a file named something like “address-personal.vcf”
• Repeat for each category, with a different filename like “address-work.vcf”, etc.

The next trick is to import all these vCard files into the iPod while preserving the category information (known on an iPod as “Groups”).

The things I tried that didn’t work:

• Import into Yahoo! Address Book, then use iTunes to “Sync Contacts with Yahoo! Addres Book”. The import was mostly painless. The “find duplicate entries” feature mostly worked, but it missed some, which is forgivable, and then does not offer a way to manually merge contacts, which is not forgivable. Furthermore, when using iTunes to sync this information for the first time, the groups information is lost; all you get is a big “All Contacts” list.
• Import into Google Contacts, then use iTunes to “Sync Contacts with Google Contacts”. Importing all the data into Google Contacts was breeze. Duplicate removal works better, and there is a manual fallback for things that are missed. However, again, the groups information is not preserved during an iTunes sync.
• Uploading VCF files into Yahoo! Address Book and Google Contacts in the hopes of seeding them with groups information also didn’t help.
• Use Palm Desktop to export to CSV … this was just a disaster.

What worked was the final option in iTunes, “Sync Contacts with Windows Address Book”. The humble Windows Address Book came to the rescue!

• Open Windows Explorer and navigate to the folder containing all the exported Palm Address VCF files.
• Start up Windows Address Book. It lives in the Start Menu via Programs … Accessories.
• Select “New Group…” to create a new group. Repeat for each category.
• Expand “Main Identity’s Contacts” to reveal all the groups.
• Click on a group.
• Drag the appropriate VCF file from Windows Explorer into the Address Book window.
• Click the next group, and repeat, for each VCF file.

Finally, in iTunes, go to the Info tab and check “Sync Contacts with Windows Address Book”, and click “Sync”. Your iPod “Contacts” app should have full groups information.

Unfortunately, there is no good way to truly synchronize group information with Yahoo! or Google. Syncing iTunes to either service loses the group information; everything appears in Yahoo! or Google as “Unfiled”. And the real problem is that there is no interface on the iPod for editing group information for a contact. So over time, all contact organization is doomed to eventually flatten to “Unfiled”.

Without buying an app from the iTunes app store, the only way to really maintain group information is to use a desktop program (Windows Address Book, or some equivalent on the Mac) as the primary means of adding new contacts.

Date, To Do, and Memo promise to be equally fun and exciting …

In Dynamic DNS with 1&1, I wrote about how I host a site on a server with a dynamic IP address (such as a server on a typical home connection).

The shortcomings of the 1&1-based service:

• No IMAP access to e-mail. 1&1 only provides POP access (no SSL, either). POP is no good for multiple points of presence (checking mail from a phone would interfere on the “headless” downloading of mail at home).
• Very limited number of subdomains. Ridiculously, 1&1 only provides for 5 subdomains per 1&1 account (not per domain, per 1&1 account).

The above was written before Google Apps For Your Domain became widely available. 1&1′s $8.99 is still the cheapest registration out there – it gets you one year of registration with private WHOIS registration included. Other closely-priced offerings charge extra for the private WHOIS registration. With Google Apps, e-mail and DNS services can be separated from the low-cost 1&1 domain registration:

• MX records can be pointed to Google Apps
• DNS can then be delegated elsewhere to any of the numerous FreeDNS services (I will likely select http://freedns.afraid.org/, Namecheap, or maybe EveryDNS – recommendations are welcome).

Today we talk about configuring a home Linux system for use with Google Apps for e-mail in a way that still provides for web-based access to the same e-mail account (home download of e-mail shouldn’t cause mail to appear as “read” by the webmail client).

Getmail

I like getmail for mail download. It is an easy-to-configure alternative to fetchmail. Google provides the bare-bones “Configuring other mail clients”, but leaves it to you to figure out the exact configuration. Create a $HOME/.getmail/getmailrc:

[retriever]
type = SimpleIMAPSSLRetriever
server = imap.gmail.com
port = 993
username = username@gmail.com
password = #####
mailboxes = ('I',)

[destination]
...

[options]
read_all = False
delivered_to = False
delete = True

The interesting bits:

• I set up the Gmail account with a filter that adds the “I” label to all incoming e-mail (the e-mail I want to eventually download with getmail).
• getmail is configured to download mail from the “I” folder (Gmail labels are IMAP folders).
• getmail is configured to delete read mail. Gmail treats an IMAP delete as an “unlabel” operation (mail isn’t actually deleted unless it is moved to the “Trash” folder.

The result:

• getmail can download mail all day.
• The Gmail web interface can be used to read mail without confusing getmail about what has been seen and what has not been seen. Furthermore, normal Gmail filters can be used independently of getmail (as long as the magic “I” labeling is not interfered with).
• As a side-effect, the presence of the “I” labels is a visible indicator of getmail activity.

Postfix

For outbound mail (SMTP), most Linux (and Mac OS X) systems will use postfix. ISPs will direct you to use their SMTP servers. This is fine, but one loses the nicety that sent mail will not appear in the Gmail “Sent Mail” folder. Getting this kind of clean integrated “Sent Mail” behavior requires that outbound mail go through Google’s SMTP servers.

In /etc/postfix/main.cf:

relayhost = [smtp.gmail.com]:587
smtp_sasl_auth_enable = yes
smtp_sasl_security_options = noanonymous
smtp_sasl_password_maps = hash:/etc/postfix/sasl/sasl_password
smtp_use_tls = yes
smtp_tls_CAfile = /etc/ssl/certs/Thawte_Premium_Server_CA.pem

In /etc/postfix/sasl/sasl_password:

[smtp.gmail.com]:587 username@gmail.com:password

Update and restart postfix:

sudo postmap /etc/postfix/sasl/sasl_password
sudo /etc/init.d/postfix restart

The next project is to get DNS set up somewhere to work around 1&1′s pesky 5-subdomain limit.

I still use mutt for reading my e-mail. I’ve tried other stuff (Thunderbird, Entourage, Gmail), but the thing I can’t get over is the text editing. There are certainly things that aren’t as nice as graphical mail clients, but I do get by:

• PuTTY/iTerm and/or urlview/urlscan give me easy access to hyperlinks.
• Attachment viewing is a keystroke away.
• LDAP integration at work provides tab-completion on names and e-mail addresses.

In all other respects – mail filtering (procmail), editing, quick scanning, etc. – mutt wins hands down.

The one deficiency is a decent search facility (a la Gmail); mutt “out of the box” only supports folder-based search, and doesn’t provide a mechanism for searching all mail.

Enter mu – an indexing system for Maildir-based e-mail. Indexing itself is pretty fast: at home (AMD64 1.0GHz), it took about 8 minutes to index 109857 messages (1.1GB) going back to 1993. Being Maildir-based, mu can incrementally update itself with just the new messages simply by examining file mtimes.

The rub is that all my mail was stored in mbox-format (maildir wasn’t invented yet); I had to use mb2md to convert all the mboxes over to maildirs.

Maildir’s one-file-per-message scheme is nice for lots of things because applications can create a separate set of directories with links to the actual messages to implement features like search results (such as done with mu) and tagging (a.k.a. “virtual folders”).

The maildir format is bad for things like backup and other applications where it would be convenient to have fewer files to manipulate. Individually compressing many small files loses any benefit from compressing a large concatenated stream of files. Maildir filenames contain a ‘:’ character, which makes them un-copyable to a Windows machine (such as for backup).

Pause while laughter subsides.

I am a fan of the Logitech “marble” mice. The latest generation is the Trackman Marble Mouse (preceded by the “Optical Marble Mouse” and the “Marble Mouse”, all of which I’ve used and have loved).

The ergonomics are great:

• The trackball form factor means a fixed footprint. This is important for keyboard tray users, since “footprint space” is more limited.
• The “marble” design means manipulation with many fingers or the whole hand, and clicking with a more natural “grasping” motion with the biggest, strongest finger: the thumb. This is better than other so-called trackballs with smaller balls designed to be manipulated with just the thumb or one finger (and with buttons that require the traditional clicking motion).
• The symmetric design means rightie or lefty use. “Lefty” use is especially important for users of traditional keyboards with numeric keypads to the right.
• The smaller buttons are programmable. I program them as “PgUp” and “PgDn” keys, which provides equivalent functionality to, and faster use than, the small “scroll wheel” buttons found on many mice (which are also very ergonomically bad).

The only bad thing about these trackball mice is that they are bad for gaming (which I no longer do).

I had a problem with mine a few days ago (plugged into a MacBookPro) where the mouse cursor would just freeze at seemingly random times, with recovery requiring an unplug/replug of the mouse, or sometimes even a reboot of the whole computer.

With a coworker, we quickly narrowed the problem down to the mouse itself (we swapped mice and the problem moved with the mouse). Furthermore, the coworker discovered that recovery was much more conveniently achievable by simply popping out the “marble” and putting it back in.

This was almost acceptable, but I went ahead and called Logitech Phone Support to see about getting a replacement, just to see what would happen. The support representative told me he hadn’t heard of my problem before, but he gave me some “device reset” instructions:

1. Unplug the mouse from the computer.
2. Hold down both buttons for 90 seconds.
3. Release both buttons.
4. Plug the mouse back into the computer.

Voila! I haven’t had any more mysterious problem since! The tech support experience was great:

• After navigating a phone tree to enter my mouse model and operating system, I was speaking to a representative within 15 seconds.
• I did not get the standard script of rebooting the computer, unplugging/replugging the mouse, etc.

My only gripe is that the device-reset instructions should be available on the product support page; it would have saved all of us a phone call.