DEVONthink – Backups, Databases and Metadata

Problem / Outcome Summary

• This article will help you to understand how DEVONthink backups, databases and metadata are stored on your drives and understand how to best backup your DEVONthink databases

Why might I want to read this?

• To have confidence your DEVONthink database is backing up as you expect
• To learn how to move DEVONthink to a new computer
• To understand the DEVONthink terminology around backups.

To be clear, why shouldn’t I just use the DEVONthink backup?

Because, as per the DEVONthink warning in the application, the DEVONthink backup is not actually a real backup.  What it does is export the Metadata of your database, in place so that you can import it back again if you make a mistake with metadata.   It does not back up any files at all.  We’ll be reaching out to DEVONthink to try and understand why you might want to do that, but to be honest, we’d be just as happy if they renamed it to ‘Duplicate Metadata’ since that’s what it actually does and other than it’s name, it really shouldn’t have to enter into the backup conversation at all.

A word about backups.

There is a lot of confusing information about backups on the internet.  Of course, everyone should have at least one backup of their important data, but backups are not about keeping a single copy or two of your data, backups are about whole process, that if adhered to correctly, can save you from disaster.  DEVONthink is absolutely no exception to this, especially given it’s widespread use as a document management system.  If you’d like to learn more about the backup process and why you need one, click here for our article, Howto create a Proper Backup process and why you need one.

Before I go too much further, I’m quite a fan of DEVONthink.  If you’ve not come across it and need to check it out, have a look at their product page here.

OK, so what do I need to know about DEVONthink and backups

I’m glad you asked.

1. Depending on which version of DEVONthink you purchased, your databases could be stored in more than one location.  ‘DEVONthink personal’ only allows a single database in a single location, whereas ‘DEVONthink Pro’ and ‘DEVONthink Pro Office’ allow you to store multiple databases wherever you like.  By default, DEVONthink will store your database in ‘~/Library/Application Support/DEVONthink Pro 2/’.  This by the way, includes the default general Inbox for Pro and Pro Office users.
2. The database (I was surprised to find out), is not actually a database.  The interface in Mac see’s it a bit like a Mac dmg ‘package’ which is normally like a zip/iso file, except with DEVONthink the database is actually more like a traditional folder with files in it.  You’ll notice this if you try to copy ‘the database’ via the command line, because you will see it actually copies all the files inside the folder separately.  You can also ‘right click’ it and choose show package contents.  Very unexpected.
3. Within the database folder, you can see it’s comprised of three sections, these are a settings.plist file, a folder full of your database files and DEVONthink metadata.

Right, so can I perform any type of backup with DEVONthink?

Well, wouldn’t you know it, DEVONthink doesn’t actually have a backup tool, at all.  Given the extensive features in DEVONthink, this is a real disappointment as it would be easy to add one. There are however tools within DEVONthink that will give you a single up-to-date copy of your current database, which is sometimes called a  backup, however for reasons mostly outlined in our backup process article here, this is most definitely not a backup. The main reason for that is, if you make a mistake with your DEVONthink ‘database’, that mistake will be copied to the ‘backup’ as well, meaning that mistake may not be able to be undone, since DEVONthink has only kept a single copy.

It might not be quite as bad as you think.

It turns out DEVONthink aren’t completely negligent after all, when you understand and put all the information above together into a single context.  This is because of the following things, which are not understood well because, well, they’re hidden and not explained.

Consider this:

1. The database is not a database, it’s a folder full of files
2. The folder full of files includes the documents you’ve saved
3. The folder full of files includes the metadata you’ve worked so hard on creating
4. The folder ‘can’ include multiple backups of the metadata.

Say what?  OK yes, another undocumented feature is that the metadata backups get stored into the ‘database’ that’s not actually a database.  Each metadata backup is stored in it’s own separate folder within the ‘database’ with a date beside it, and this is configurable to how many copies to store and when to back them up within the ‘Preferences, Backup, archiving’ section.

Wow, that almost sounds like a backup doesn’t it?  So now we’re getting somewhere.  In reality, this is still NOT a complete and real backup, but it is a whole lot better than it seemed to be up front.  Because the database is not a database and it contains multiple copies of the metadata, it does in fact therefore protect you from a larger amount of problems that you might have originally thought.  However, this still, does not back up your files, should for example you accidentally delete one, one becomes corrupt (or something else).

So how do I do a proper backup of DEVONthink?

When you get down to it, you realise DEVONthink’s mentality.  The one mistake they made is calling their file structure a database, because it conjures up thoughts of a single flat file or an external application such as MySQL or PostgreSQL, which actually would have traditional backup / export capability.  From one perspective, Mac users are more creative, less concerned with technical detail and therefore rely on standard Mac tools such as time machine.  From another perspective, Mac users look for even more simplicity and expect the application to have it’s own backup given all the time and effort put into something like DEVONthink.  To this end, (simply speaking) you’d expect something called backup, to backup the something that it’s created in, but it unfortunately falls short both when you’re an average Mac user and when you’r emore technical and know what you’re talking about.

To be really clear, DEVONthink does not currently offer a database tool to backup the files in your database and your metadata into anything but an automatically synchronised single copy – this is absolutely NOT a backup you should rely upon.  While you may find the backup function does save you in certain circumstances, to do a real and proper backup of DEVONthink, you are going to need some external software, some scripting, an external NAS with it’s own backup regime or Apples’ built in Time Machine backup.  From this article you have learned where the important data to back up is located, so you can now do exactly that and be confident you have your DEVONthink data safe.

In summary the location of the DEVONthink data to backup is:

• The actual DEVONthink ‘database’ (which is a folder) in a location you choose AND/OR
• The default and inbox ‘databases’ ending in .dtbase2 which are located in the default database directory of ‘~/Library/Application Support/DEVONthink/’
• These locations include all of the metadata and files you need to back up, in order to restore from a failure of any sort.

I had worried about not having the right folders and files backed up for a long time because I did not understand (and could not find) how this all worked.  Now that I know where they are, I understand what I’m supposed to do with them.  As such, I’ll likely store them on my NAS and let the NAS backup the data with my favorite ‘client side’ encrypted cloud storage, namely Amazon’s ‘Glacier’ backup storage.

As always, we welcome your insights and opinions in the comments section below.

Marshalleq