How to recover Evolution data after an upgrade

Write tutorials for Linux Mint here
More tutorials on https://github.com/orgs/linuxmint/discu ... /tutorials and (archive) on https://community.linuxmint.com/tutorial
Forum rules
Don't add support questions to tutorials; start your own topic in the appropriate sub-forum instead. Before you post read forum rules
Post Reply
mpiter
Level 2
Level 2
Posts: 80
Joined: Tue Oct 25, 2011 3:24 pm

How to recover Evolution data after an upgrade

Post by mpiter »

1. Introduction

Like many others, I spent a lot of time trying to recover my Evolution data after upgrading to Mint 12. I could not find a Web page where everything was explained. The information was spread over too many Web pages. I therefore decided to write a small howto on this topic.

Evolution format changed from 2.32 to 3.x. In both cases, the data was stored in ~/.local/share/evolution/mail/local/ according to XDG Base Directory Specification (http://www.freedesktop.org/wiki/Specifi ... sedir-spec). Inside that data directory, the format and the tree structure changed.


2. The 2.32 format

In 2.32, each Evolution folder was stored on the hard disk with 4 files. One single file contained all the folder emails in Mbox format. This Mbox file and the Evolution folder had exactly the same name, e.g., Inbox, Outbox, Sent, Drafts... The 3 other files were metadata files related to the folder format and indexes. The 3 files had the same names than the Mbox file but with extensions .cmeta, .ibex.index, and .ibex.index.data. For example, you should have Inbox, Inbox.cmeta, Inbox.ibex.index, and Inbox.ibex.index.data in ~/.local/share/evolution/mail/local/.

When an Evolution folder folder_name contained or had previously contained another folder, a directory called folder_name.sbd was added to the 4 files. For example, for an Evolution folder called Mydir that contained other folders, you will find:
  • Mydir (the Mbox file containing the emails stored in Mydir);
  • Mydir.cmeta, Mydir.ibex.index, and Mydir.ibex.index.data (the 3 files associated to the Mydir Mbox file);
  • Mydir.sbd (a directory containing the other subdirectories).
This structure recursively repeated itself inside the subdirectories. For example, an Evolution folder Main containing 3 folders Sub1, Sub2, and Sub3, with Sub3 containing another folder Sub4 was structured like this:

Main folder-------------|-Content of Main.sbd--|-Content of Sub3.sbd
---------------------------+----------------------------+----------------------------
----------------------------|-----------------------------|
----------------------------| Sub1----------------------|
----------------------------| Sub1.cmeta-------------|
----------------------------| Sub1.ibex.index--------|
----------------------------| Sub1.ibex.index.data-|
---------------------------|------------------------------|
Main---------------------|-Sub2-----------------------|
Main.cmeta-------------|-Sub2.cmeta-------------|
Main.ibex.index--------|-Sub2.ibex.index-------|
Main.ibex.index.data-|-Sub2.ibex.index.data--|
Main.sbd----------------|-----------------------------|
----------------------------|-----------------------------|
-----------------------------|-Sub3---------------------|-Sub4
-----------------------------|-Sub3.cmeta-------------|-Sub4.cmeta
-----------------------------|-Sub3.ibex.index--------|-Sub4.ibex.index
-----------------------------|-Sub3.ibex.index.data--|-Sub4.ibex.index.data
-----------------------------|-Sub3.sbd-----------------|

3. The 3.x format

In Evolution 3.x, the emails were stored in Maildir instead of Mbox format.

In Maildir format, a mail folder is composed of one directory (often called Maildir in Linux) containing 3 subdirectories called cur, new, and tmp. The subdirectory cur contains one file for each email of the folder, i.e., one file per email instead of one file containing all emails as in Mbox format.

So the Evolution file structure was accordingly modified. Each Evolution folder was stored on the hard disk with the same 4 components than in 2.32 except that the Mbox file was replaced by a Maildir directory with the same name. Furthermore, all folder names were preceded by '.' to make them hidden files. For example, your Sent folder had become .Sent, .Sent.cmeta, .Sent.ibex.index, and .Sent.ibex.index.data with .Sent being a Maildir directory containing the subdirectories cur, new, and tmp.

An evolution subfolder was stored in the same way with the folder names separated by a dot. For example, in the last example of the previous section, you should find .Main.Sub1, .Main.Sub2, .Main.Sub3, and .Main.Sub3.Sub4 Maildir directories in ~/.local/share/evolution/mail/local/ with the 3 corresponding Evolution metadata files for each Maildir directory.

Finally, ~/.local/share/evolution/mail/local/ has become your Inbox Maildir, i.e., you will find there cur, new, and tmp directories with cur containing all your Inbox emails.


4. Converting data format from Evolution 2.32 to 3.x

To avoid leaving any leftover of the previous version, I think it is best to get rid of the whole previous data and to rebuild the new one from scratch.

4.1 The easy way

Exit Evolution and move all the data from ~/.local/share/evolution/mail/local/ to an easy-to-reach place such as ~/tmp. This data contains all the Mbox files mentioned in section 2. Run Evolution and recreate your previous folder structure with the "Folder / New..." menu item. It should not take long. Then, import each Mbox file from ~/tmp to the right Evolution folder (Menu File / Import...). The procedure is self-explained once you start it. Of course, you must choose "Import a single file" instead of "Import data and settings from older programs" when you are requested to select one of these options. It is a good way to restore your emails if you only have a few folders to import. When you have many folders to restore, it can be boring and you might prefer to use shell commands.


4.2 A faster way if you love shell commands

If you have many folders to restore, you might prefer to do everything in bash to automate the process. It is possible to convert Mbox files to Maildir format and to recreate the Evolution structure detailled in section 3 in bash. For example, mb2md is a nice tool to convert Mbox files to Maildir format. Open a terminal and:

Code: Select all

$ sudo aptitude install mb2md
$ mkdir ~/tmp
$ cd ~/tmp
$ pushd ~/.local/share/evolution/mail
$ mv local ~1
$ mkdir local
$ cd local
$ mb2md -s ~/tmp/local/folder1 -d ~/.local/share/evolution/mail/local/.folder1
$ mb2md -s ~/tmp/local/folder1.sbd/sub1 -d ~/.local/share/evolution/mail/local/.folder1.sub1
This will convert folder1 and folder1/sub1 in the new format. Even if you have a complex folder structure, it is easy to use "for i in blabla ; do mb2md ... ; done" to process all your data with just a few commands. It may be faster and more reliable than to manually import everything by hand in Evolution. Be careful with mb2md because it processes data relatively to your home directory if you do not give full paths. You also need to convert Inbox from your original data and to move the resulting cur, new, and tmp directories in ~/.local/share/evolution/mail/local/ since this directory is you Inbox Maildir directory.

When you have finished converting your data, you will need to build the metadata files. Evolution will take care of everything for you. Run Evolution then select each folder and let Evolution rebuilds its data and index metafiles. Just keep an eye on the bottom line of its window to be sure it has processed the data before leaving the folder for another one (when you leave a huge folder, you will see the process progress in the bottom line of Evolution window). Due to the lack of metadata, Evolution will duplicate all emails and it might also produce error messages telling you that he cannot read some emails. Do not pay attention, everything will be resolved soon. When you are done, exit Evolution to let it properly set all its metafiles. Rerun Evolution again and it will take care of the duplicated emails during this second pass. After a while, it will be over. You can exit Evolution again to let the software fix everything, then your are all set with Evolution 3.2 with all your mail folders :-)
giancon

Re: How to recover Evolution data after an upgrade

Post by giancon »

Thanks!
Everything very clear and it all worked like a charm. I battled a lot, initially, because once the automatic import script fails for the first time, evolution has already written a few config hidden directories (probably at each new install). Thereafter, every time I fired up evolution hoping to get it to give me an initial, "clean slate" folder structure, it kept popping-up the same dialog telling the user to import from mbox format. Which, of course, kept crashing. Eventually, I managed to fire up a clean instance of evolution and replicate my original folder structure, only after a went to seek-and-destroy manually all the hidden .local/mail/~ folders I could find.

Eventually, I imported each subfolder of my mailbox manually (via GUI) instead of using the shell.

Thanks for your effort in writing all this up.

ciao

G.
mpiter
Level 2
Level 2
Posts: 80
Joined: Tue Oct 25, 2011 3:24 pm

Re: How to recover Evolution data after an upgrade

Post by mpiter »

And thanks for your positive comment. At least I know now that I did not write the howto for nothing. It took me some time. :D
jan rinze

Re: How to recover Evolution data after an upgrade

Post by jan rinze »

Life saver!!

after upgrading from Debian 6 to Debian 7 (wheezy) i lost my mailbox but with this tutorial i was able to restore it.

Many, many thanks!!

Jan Rinze.
mpiter
Level 2
Level 2
Posts: 80
Joined: Tue Oct 25, 2011 3:24 pm

Re: How to recover Evolution data after an upgrade

Post by mpiter »

Welcome :-)
gramo

Re: How to recover Evolution data after an upgrade

Post by gramo »

Thanks for your help

I have a large structure of dirs and subdirs about 6 levels deep, my message's dir is about 7 GBytes, and, after dealing some time with scripts finally I have my evolution 3.2 with every one of the 2.28's messages.

I'm currently using Mageia 3, then I had to download mb2md from rpmfind but basically from command line everything were the same.

The only difference, that causes me hesitate, is that evolution's first run shows directory structure but no-messages at all... then I thought that something were wrong and when I turned off evolution it seems to be blocked. (I didn't realize that evolution were generating metadata for 7GBytes)...

When I summoned evolution again (After about 10 minutes blocked) everything works nicely.

Thanks a lot (And sorry for my bad english... maybe I sound like Tarzan, ¡Bwana!)
mpiter
Level 2
Level 2
Posts: 80
Joined: Tue Oct 25, 2011 3:24 pm

Re: How to recover Evolution data after an upgrade

Post by mpiter »

It is fun to see that almost two years after the first post people can still use it. It is pleasant to know.

I gave up Evolution one year ago because there were too many bugs. I think it was becoming more buggy with each new version. I have used Thunderbird for more than 6 months, but the support for movemail is really poor and the maildir storage is so bugged that it is very risky to use it. Mail can be lost. The last Thunderbird version cannot handle anymore movemail in maildir storage, so I am switching right now to KMail. It has become difficult to find a good mail client on Linux.
Genbar1

Re: How to recover Evolution data after an upgrade

Post by Genbar1 »

You can try Data Recovery, it's safe and effective, it can retrieves your lost videos, photos, music, documents, emails, etc. from your PC’s hard drive as well as from USB drives, external hard drives, mobile phones, digital cameras, iPods, MP3/MP4 players, and other storage media. Four unique recovery options guarantee to find everything you’re missing quickly and easily.
mpiter
Level 2
Level 2
Posts: 80
Joined: Tue Oct 25, 2011 3:24 pm

Re: How to recover Evolution data after an upgrade

Post by mpiter »

You missed the point. The problem was not lost emails but format change from one Evolution version to the next. The emails were still in the computer but the new Evolution version changed the default place where the emails were stored and the software could not read the old format.
Post Reply

Return to “Tutorials”