Thousands of Evernote users affected by data loss

Online note-saving service Evernote on Monday acknowledged that it had suffered a hardware fault at the beginning of July that resulted in potential data loss for more than 6,000 of its users worldwide.

The issue was first reported by blog Techwave, citing a report from Japanese newspaper Mainichi Shinbun. In a Monday note to Evernote users on the company's blog, Evernote CEO Phil Libin explained that the loss stemmed from bad server hardware:

"Every user's data is stored on a 'shard.' A shard is made up of a server together with a redundant fail-over server. If there is any problem with a server, the system automatically fails over to the second server in the shard. We currently have 37 shards. Shard 22 was the one that had problems last month."

Evernote's back-up system stores user data in up to six different places using both on- and off-site servers as well as locally on the user's copy of the software. Though in the case of the problem, which lasted four days, user data was simply being overwritten due to one of these systems not having a working failure routine. "Basically, the shard kept failing over back and forth between two servers over the time period causing some of the data created during that time to get overwritten," Libin explained.

In a call with CNET on Monday morning, Libin said that of the 6,323 users affected by the outage, approximately 70 percent were able to get their data back.

Evernote's software saves a copy of a work in progress before syncing it up with whatever was stored online, so the company was able to pull the complete copies of various files once the problem had been addressed and fixed. However, those who had been working purely on Evernote's site, and whose work was being stored on the faulty shard, had no such protection.

As an apology, Evernote has provided affected users with a free year of the company's $45-a-year premium service. Those who were already premium subscribers get an extra year.

As for whether this could happen again, Libin said it's extremely unlikely."This was a freak of hardware failures. But we've changed the fail-over process so it won't happen again."

Data loss on large-scale Web services is uncommon, but can be extremely hard to recover from. In 2009, social-bookmarking site Magnolia suffered a massive data corruption that resulted in the loss of all its user data. It has since started from scratch with a new version of the site. Prior to that, one of the most high-profile outages was a multi-hour downtime for Amazon's S3 cloud storage service, which many sites use as their built-in storage solution.

At a press conference three weeks ago, Evernote announced it has reached 3.7 million users since launching in June of 2008. In that time, its users have saved 145 million notes, which Libin said works out to 312 new ones every minute.

Evernote says affected users received the below e-mail from company CTO David Engberg, and that if they didn't receive it, they were not one of the 0.2 percent in the problem group.

From: David Engberg
Subject: Important information about your Evernote account

We're writing to tell you about a potential problem which may have occurred with your Evernote account a few days ago. One of our servers suffered a series of serious hardware failures over the Fourth of July weekend. Our automatic redundancy and backup systems recovered the vast majority of the data and most users did not experience any problems.

Unfortunately, the servers may have failed to properly record new notes and edits made by a small group of users (less than one-fifth of 1 percent) between July 1 and July 4. Your account is in the group potentially affected. We're writing to suggest that you attempt to recover or recreate any important notes that may be missing from that time using the information stored on your Evernote desktop application. Please see the instructions for each client, below.

Alternatively, if you created notes during those dates via Evernote's incoming e-mail gateway, you may be able to find the e-mails in your mail client's "sent mail" folder, and you could send them into Evernote again. If you created notes by clipping from the Web, you may be able to find those pages again from your browser's history to clip the pages again. If you created file attachments or PDFs, you may be able to find the original files and add them again.

This was an isolated, one-time occurrence so any notes from before July 1 or after July 4 will not be affected. We have revised our management, backup, and monitoring systems to prevent this type error in the future. We apologize for our mistake and for your inconvenience (even if you didn't lose any notes, you still had to read this e-mail).

As a further apology, we have upgraded your account to Evernote Premium for one year. In addition to a much higher monthly quota and some powerful new features, a Premium account will ensure that you get priority tech support should you have any lingering problems or questions from this disruption. If you were already a Premium subscriber--thanks!--we have extended your subscription by an additional year at no charge.

Thank you for finding us and for using Evernote. We are committed to constantly improving our product and providing you with the best possible experience now and in the future. Please do not hesitate to contact me directly if you want any additional information about this error, need help with the recovery procedures below, or just have ideas about how we can make Evernote more useful for you.

Sincerely,
David Engberg
Chief Technology Officer
Evernote Corporation
REDACTED

----------------------------------------------------------------------
* If you have installed Evernote for Windows: *
----------------------------------------------------------------------

1) Start Evernote on your Windows computer and press the Sync button.

2) Click on the "All Notes" button to view all of your notes.

3) Find all of the notes that were created between July 1st at 6:45am GMT and July 5th at 10:45am GMT. You can do this by sorting your list by the Date Created, or by copying the following line and then pasting it into the "Search" field in Evernote:
created:20100701T064500Z -created:20100705T104500Z

If you see any notes created during this time, then please continue, otherwise you can stop.

4) Select all of the notes that were created during this time by clicking on the first note and then holding the Shift key while you click on the last note to select them all.

5) Go to: File > Export...
Save the .enex archive file to a place on your computer.

6) If you have created any "Local" notebooks on your computer that you didn't want to synchronize to Evernote, then right-click on each of these in your list of Notebooks on the left side and use "Export Notes..." to export each of these to other .enex archive files on your computer.

7) Go to: Tools > Sign Out

8) Go to: Evernote > Options...

9) Click on: Open Database Folder...

10) Move the file with your username (ending in ".exb") to a different location on your computer. For example, move it to your computer's desktop.

11) Sign in to Evernote with your username and password. If you see a prompt to "transfer database", click "No".

12) Wait for your account to synchronize

13) Go to: File > Import > Evernote Export Files...

14) Import each of the .enex archive files that you Exported in steps 5 and 6

----------------------------------------------------------------------
* If you have installed Evernote for Mac OS X: *
----------------------------------------------------------------------

1) Start Evernote on your Mac and press the Sync button

2) Click on the "All Notes" button to view all of your notes

3) Find all of the notes that were created between July 1st at 6:45am GMT and July 5th at 10:45am GMT. You can do this by sorting your list by the Date Created, or by copying the following line and then pasting it into the "Search" field in Evernote:

created:20100701T064500Z -created:20100705T104500Z
If you see any notes created during this time, then please continue, otherwise you can stop.

4) Select all of the notes that were created during this time by clicking on the first note and then holding the Shift key while you click on the last note to select them all.

5) Go to: File > Export...
Save the .enex archive file to a place on your computer.

6) If you have created any "Local" notebooks on your computer that you didn't want to synchronize to Evernote, then Control-Click (or Right-Click) on each of these in the list of Notebooks on the left side and use "Export Notes..." to export each of these to other .enex archive files on your computer.

7) Click on the elephant icon in the menu bar and select "Quit" from the menu.

8) In the finder, open the "Library" folder in your home directory. (This is in the directory with your name, NOT at the top level of your hard drive.)

9) Open the "Application Support" folder

10) Move the "Evernote" folder from that location a different place (like your desktop).

11) Start Evernote, and sign in with your username and password if needed.

12) Wait for your account to synchronize

13) Go to: File > Import...

14) Import each of the .enex archive files that you Exported in steps 5 and 6

CNET Editor-at-Large Rafe Needleman contributed to this report.