Configuring mail in CiviCRM

Notes I’ve found useful as I set up mail in CiviCRM, part of a larger series on configuring CiviCRM.

CiviCRM is designed to work with users’ existing mail client, not replace it. Users will keep on using Thunderbird or webmail or whatever.

For a walkthrough of basic mail setup, see the administrator’s guide.

OUTGOING MAIL
When configuring outgoing mail at Administer – Configure – Configuration Checklist – Outgoing Email, if your server uses a secure connection, prepend ssl:// to the server name. For example, if your server is smtp.example.com, then enter ssl://smtp.example.com.

INCOMING MAIL
CiviCRM can process two kinds of incoming mail: bounce messages produced by outgoing mass mail, and anything in the user’s mail client he wants to copy to the activity records of the sender and of each recipient. First, let’s consider the workflow once everything is set up. Then we’ll see how to set them up.

Bounce processing: concept and workflow
You’ll need a dedicated mailbox for this, perhaps unsubscribe@example.org, on a mail server that supports subaddressing of incoming messages. Subaddressing support means that mail addressed (to continue with our example) to unsubscribe+subaddress@example.org will be delivered to the mailbox of unsubscribe@example.org without error.

The bounce processor mailbox can be named anything you like, but I prefer “unsubscribe”. CiviCRM will append unsubscribe instructions to all mass mailings, including an unsubscribe URL or mailto: link. In the latter case, the bounce processor mailbox is used. So to keep things clear for recipients (and for system administrators) I name the bounce processor “unsubscribe”.

TODO: Make bounce processing work and document. I have a dedicated mailbox and mass mailings go out with the bounce processor box in the Return-Path and X-CiviMail-Bounce headers. Yet bounce messages are delivered to the sender’s account, not the bounce processor mailbox.

Recording external emails: concept and workflow
Also known as email to activity processing. There are two ways to do this; I like to make both available for users.

Dedicated mailbox
Set up a dedicated mailbox, say activities@example.org. Configure each user’s preferred email client to BCC this address by default on outgoing correspondence; CiviCRM will acquire copies in the activity records of the sender and of each recipient. This is useful for outgoing mail.

Be aware that the dedicated address is not excepted from processing and will be added as a contact. For a better user experience, I add the dedicated address to the home organization’s contact record.

Dedicated IMAP folder
Create a dedicated IMAP folder, say Activities, in the users’ mailbox. Using his preferred mail client, the user copies desired messages into this folder; CiviCRM will acquire copies in the activity records of the sender and of each recipient.

To do its stuff, CiviCRM will create two subfolders here: CiviCRM/ignored and CiviCRM/processed. So while you can name the dedicated folder anything you like, for clarity’s sake I don’t recommend naming it “CiviCRM”.

Setting it up
Expanding on the general information in “Configuring inbound email processing” at Email System Configuration, I find the following helpful:

If this is a new CiviCRM installation, then at Administer – CiviMail – Mail Accounts, you will see one default account. If you are configuring the bounce processor, edit it. If you are setting up email to activity processing, then press the “Add a mail account” button. Whichever you are doing, continue thus:

• Name: Whatever you want. For clarity, I find it helpful to name each account one of “Bounce”, “BCC to activity,” or (in the case of a dedicated IMAP folder) the name of the associated CiviCRM user.
• IMAP server, username, and password: The same information you would enter when setting up incoming mail in a mail client.
• Local part: Leave this blank unless you are editing the bounce processor. In that case you enter the part of the mail address before the @, followed by a plus sign. For example if your bounce processor mailbox is return@example.org, you would enter return+.
• Email domain: The part of your mail address following the @. Following our example, that would be example.org.
• Return path: You may leave this empty.
• Protocol: Per your server, but preferably IMAP. If you’re setting up a dedicated IMAP folder, this obviously must be set to IMAP.
• Source: Leave this blank unless you are setting up a dedicated IMAP folder. In that case give the name of the dedicated folder you previously created for this purpose in the account. The exact syntax will depend upon the mail server and may require some trial and error. For example, on one recent installation I created the folder “Activities” in the main mail folder. Thunderbird reported the new folder’s path as “imap://user%40example.org@imap.example.org/INBOX/Activities”, and with a little trial and error I found that INBOX/Activities worked. YMMV.
• Uses SSL: Per your server, but preferably yes.
• Used for: If you are editing the bounce processor, choose that. If you are setting up either a BCC to activity account or a dedicated IMAP folder, choose “Email-to-Activity Processing”.

Press Save to save your edits. Repeat as necessary to add more mail accounts. You may not have more than one bounce processor. Your organization only needs one BCC to activity account. For dedicated IMAP folders, however, you will need to set up a separate account for each CiviCRM user with a mail account.

Testing it
First we’ll test CiviCRM’s ability to access the accounts. Go to Administer > System Settings > Scheduled Jobs. Locate the appropriate job(s) to test. If you configured one or more bounce processors, look for Fetch Bounces; if you configured mail to activity processing, look for Process Inbound Emails. Select More > Execute Now. Check the job log; there should be no errors.

Next we’ll put messages in the accounts and test CiviCRM’s ability to process them:

• If you configured a bounce processor, open any contact record for editing. Add a known invalid email address, one that you have confirmed will generate a bounce message. Save your edit. Now send a test mass mailing to that known invalid address.
• If you configured a BCC to activity account, send a test message from your preferred email client to a contact, BCCing the dedicated mailbox. Both the From: and the To: email addresses should correspond to contact records in CiviCRM.
• If you configured one or more dedicated IMAP folders, enter your preferred email client and copy a message into the dedicated folder. Again, both the From: and the To: email addresses should correspond to contact records in CiviCRM.

With the messages in place, repeat the test: Go to Administer > System Settings > Scheduled Jobs, locate Fetch Bounces and/or Process Inbound Emails, and select More > Execute Now. Once again, the job log should have no errors. More importantly, CiviCRM should have processed each message appropriately:

• If you configured a bounce processor, check the contact record to which you added the known invalid email address. You should see that the invalid address has been marked on hold.
• If you configured a BCC to activity account, check the contact records corresponding to the From: and To: addresses in your test message. Each contact record should have a copy of the test message as a new activity.
• If you configured one or more dedicated IMAP folders, likewise check the contact records corresponding to the From: and To: addresses in your test message. Each contact record should have a copy of the test message as a new activity.

AUTOMATING IT
Once you have verified that CiviCRM properly handles outgoing and incoming mail, set up scheduled jobs to automatically do these tasks in the background.

TIPS AND TRICKS
The dedicated IMAP folder is a high maintenance feature. CiviMail’s mail accounts need to be updated every time your staff changes or a user changes his email password. It will also break if the user starts mucking about with the dedicated folder. Most won’t, but we all know the kind of user that will — they’re also the ones who complain the most.

TODO: Activities created from emails cannot be deleted like normal activities. Is there any way to delete them?

These notes refer to CiviCRM 4.3 and were last updated 22 July 2014.

REFERENCES
User and Administrator Guide and Supplement