When you send one email, a sender interacts with a recipient at the same time. But when you deal with transactional emails or bulk mails in your app, they can’t all be sent at the same time. Emails are put on an SMTP queue that provides temporary storage before processing. When the recipient is able to receive email, they are sent. Here, we’ll figure out why you would opt for email queueing in your app, and how this can be done.
What is an email queue?
An email queue is a mandatory component of SMTP servers. It is a system that creates an array of emails that are waiting to be processed for delivery. Email queuing is a form of Message Queuing – an asynchronous service-to-service communication. A message queue is meant to decouple a producing process from a consuming one. An email queue decouples the sender from the recipient. It allows them to communicate without being connected. As such, the queued emails wait for processing until the recipient is available to receive them.
You can look at an email queue as a buffer where the emails are stored before they hit the endpoint. At the same time, the sender does not need to send each message separately. The communication between the sender and the recipient is asynchronous. Once the emails have been enqueued, they’re delivered step by step. Usually, the SMTP server starts from the beginning of the queue and goes forward.
How email queues work
Let’s say you’re launching an email campaign that contains 100 recipients. Your email client interacts with the SMTP server to send the message. The server, in turn, interacts with SMTP servers on the recipients’ hosts to relay the email. Since you’re sending 100 emails, the SMTP server on your host puts them on a queue. Most mail servers use a mail transfer agent (MTA) called sendmail to do the actual sending. We’ll touch upon the differences between these two terms in the next section. The MTA will periodically send all the messages in queue until completion. If the recipient’s SMTP server does not respond, sendmail will recurrently send emails. In this case, the email queue turns into the sendmail queue. Usually, your mail client will be notified about this sort of problem. If the sendmail queue fails to be delivered during a specific term (for example, five days), the email will be returned.
– A mail server a computer system that sends and receives electronic messages using email protocols. For the difference between email protocols, reference the blog post: SMTP vs. IMAP vs. POP3.
– An SMTP server is the part of the mail server that handles outgoing mail. That’s where email queuing is mostly implemented.
– A mail transfer agent or MTA is a specific type of software that queues email and moves it along a delivery chain until it hits a Mail Delivery Agent (MDA).
Why email queues get clogged and how to fix that
When doing mass mailing, the SMTP server puts your outgoing messages automatically into an email queue. They are sent one by one from this buffer. It is a regular process that is quite beneficial for email campaigns. At the same time, queued emails can become an issue when they are waiting to be sent for an abnormally long time (depends on the service you use). Queued emails won’t be bounced. They’ll be sent anyway, but the time of the delivery can be significantly reduced. And here are the two core reasons for that:
Exceeded volume of emails
Some mailbox providers (mostly the major ones like Gmail or Yahoo) enforce email rate limits on IP addresses. The limits are based on the sender reputation. If you exceeded this rate and queued too many emails, the delivery speed will decrease. You may also reach the maximum email attachment size, which can be a brake factor as well. The only solution is to contact the incoming server as frequently as possible to push the queue. Whenever an IP exceeds the limit of emails, the SMTP server replies with an error code (for example, 421) to any command. You can read more about SMTP commands and response codes in our blog post. Also, it’s advised to opt for a dedicated IP address instead of a shared option to maximize the speed of your email flow.
Spam-related issues
Another common reason is that your email has been busted by spam filters. Don’t panic! The filters will let the emails gradually pass to analyze how the rest of the recipients react to the message. If there is slow progress, it’s okay. Your email campaign is being observed and assessed. If it’s stuck, there could be different reasons including the blockage of your IP address. In this case, you need to unlist yourself and optimize your email campaign. Read our blog post on how to avoid emails going to spam.
Besides these fundamental causes, a queue can be clogged for other reasons that you should figure out with your mail provider.
Managing email queues
Managing an email queue is an easy task if you use a server control panel like cPanel. It provides WebHostManager (WHM) to handle a bunch of stuff. And what if you don’t? Since we have no idea which mail server you use, let’s check out commands for the most widely used mail transfer agents: Postfix and Exim.
| Command | Postfix | Exim |
| List the queued emails | postqueue -p | exim -bp |
| Reattempt delivery of all queued emails | postqueue -f | exim -q -v |
| Remove all queued emails | postsuper -d ALL | exiqgrep -z -i | xargs exim -Mrm |
| Remove a particular queued email | postsuper -d “Queue ID”postsuper -d <message-id> | exim -Mrm <message-id> |
Email queue in your app
Usually, SMTP servers have a built-in queue management system. This is an option if your email campaign has hundreds of recipients. But sometimes, it’s better to have an email queue right in your app. Here are some cases when it’s beneficial:
- If a user is making async actions like sending notifications to 1000 contacts, it would be really slow without using queue or background task.
- If your app is tracking, let’s say, the limit of megabytes per month, and you want to send a notification about 70%/80%/90% limit reach. It can potentially happen quickly for some high usage, so queue is needed to send only one email to one user instead of sending three emails per event.
- If you need to send more than 10K transactional or bulk emails every day.
The SMTP queue might not be enough to handle these tasks. That’s why you should opt for a refined email sending architecture. It’s based on asynchronous system to send mails, which will queue messages before they reach the mail server. That’s how it may look at a higher level:
Here you can see three major processes:
Expanding workers implement bulk requests for emails stored in database
If the same email is to be delivered to multiple recipients, the worker expands bulk requests for each recipient. This pushes emails to the email queue.
Email queue is implemented via the email queuing system
It’s not an SMTP queue. Emails are put on a queue but not on the mail server. Without a queue, your app will try to send out thousands of emails at once. As a result, a lack of memory or time to process the request would cause crash. When you use an email queuing system like ActiveMQ or RabbitMQ, the emails are queued up and processed in batches.
Mail sending worker sends queued emails to the mail server
This worker actually takes the emails from the queue and sends them to the mail server. If the server response is negative, the email is pushed to an error queue, which either resends them later or cancels the delivery. It’s up to you to set up the scenario. Successfully delivered emails can be archived.
Test your email queuing system with a fake SMTP server
Once you have set up this advanced email sending architecture, do not forget to test it. A fake SMTP server like Mailtrap is a perfect tool for this. It’s not a tool to test SMTP server. So, you won’t be able to use it to test the SMTP email queuing. But if you have your email queue implemented before the mail server, then Mailtrap can be used. It provides all the necessary credentials like port, authentication method, and so on. Also, you can benefit from ready-to-use integrations for most common tech stacks. So, just copy a piece of code and paste it into your app.
Failsafe is another benefit of using a fake SMTP server. Your transactional or marketing emails will be transmitted to a fake POP3 server. This excludes any spam to real users. Also, you avoid any issues related to improper configuration, authentication, and so on. In the end, if your app functions well, you’ll see your email in the Demo inbox.
Wrapping up
So, the key takeaway here is that the email queue has two sides of the coin. The first one is positive. Email and SMTP queuing uncouples the processes of sending and receiving of emails. This is useful for bulk or mass mailing cases. The negative side is that a queued email is often associated with some failure that delays the message sending. At the same time, now you know how to deal with these concerns and take advantage of email queuing at most.