How The Financial Times Got More Than 78,000 Replies to a Survey

These stories are presented thanks to Litmus, an all-in-one email marketing solution that enables your team to send error-free emails and drive immediate impact — regardless of their technical coding knowledge. Their tool helps you create messages that can cut through the clutter and rise above the rest. Learn why 700,000+ users across companies of all industries and sizes trust Litmus to create effective email campaigns that convert.

Newsletters play an unusually important role at the Financial Times. We have more than 40 newsletters in all, and we see them as crucial for driving engagement among our readers and delivering FT journalism to the widest possible audience. Our research has shown that 11 percent of our engaged subscribers would be disengaged if it wasn’t for FT newsletters.

And like other large publishers, we know how important newsletters can be to both convert and retain our paying subscribers. We have two different types of paid subscription offers: Standard, which includes access to FirstFT, our regular weekday news briefing, and a weekly Editor’s Choice newsletter; and Premium, which includes access to more than a dozen additional newsletters, like Unhedged, where commentator Robert Armstrong dives into market trends. We’ve found that access to these newsletters is one of the key reasons readers decide to stick with the Premium subscription — even though it costs several hundred dollars more per year than Standard access.

We’re also keen to invest in newsletters because they let us build a direct relationship with our readers. We encourage our newsletter readers to get in touch and share their thoughts, and the emails we receive in return are useful, insightful, and often great fun.

So we’ve long known that newsletters matter to the FT, but we’ve struggled to accurately measure the impact of email on our overall strategy. Some newsletters reach massive audiences, while others are sent to relatively small lists. We wondered: Was there something else we could do to measure the success of our products and how much our readers were enjoying them? Was there a way to find out more about our readers on a regular basis?

A reader survey was the answer to our questions, and based on our experience, more newsletter publishers should consider running them.

Moving Beyond Open Rates

The impetus for our reader survey came from the changes Apple introduced last fall with their Mail Privacy Protection update, which affected the accuracy of reporting around open rates. We take our metrics seriously, and although open rates aren’t the be-all and end-all when it comes to newsletters, the fact that they were going to be far less accurate meant we couldn’t rely on them as a basic measure of newsletter health. Apple’s changes made the idea of a reader survey increasingly desirable — it would give us another way to measure newsletter success and engagement.

We also wanted to track satisfaction across our newsletter suite, to understand what was resonating with our audience and what we could improve. A survey would enable us to identify top-performers and potential strugglers. From this, we could help build a better idea of “what good looks like” and also how weaker newsletters could be improved. 

Designing the Survey

When it came to producing the survey, we knew that, in order to end up with something genuinely useful, we needed as many people as possible to reply. There would obviously be limited interest in filling out a long survey that our audience had to click a link to get to, so we looked to combat this in two ways:

First, we created a one-click feedback mechanism that was actually embedded into the bottom of the newsletter, providing an opportunity for our readers to give feedback at the point where they finished reading.

Then, we added an incentive for respondents to complete the rest of the survey: Those who finished it would be entered into a monthly prize draw to win £100 (or $100) of book vouchers. This has proved to be incredibly successful, as it turns out that FT readers love books! In order to be entered into the prize draw, respondents have to fill out the whole survey (which opens in a separate tab), but we’ve found they’re happy to do so for the chance to win a prize. 

We wanted to ensure that if we were to collect only one piece of data from readers, it would be something that we could use as a basis for understanding our entire roster of newsletters. That’s why we ended up using a simple satisfaction scale from 1 to 5. This also enabled us to create a feedback mechanism that fitted easily onto phone screens.

These numbers were later crunched as part of a formula that converts the data into a satisfaction score from -100 to +100. 

The rest of the survey, which takes no longer than five minutes to complete, was designed to uncover even richer detail. After the satisfaction metric was captured, we asked some open-ended, freeform questions around why respondents gave that satisfaction score and what they thought we could improve. This enabled us to analyze their text responses for particular themes. 

We also developed questions to get feedback on specific ideas, such as how interested our audience was in networking, attending events, or engaging on social media. This has helped us better understand how we can grow each newsletter and engage with our audience in a way that suits them. 

Using the Survey Data

The results of the survey were beyond our most optimistic predictions, with over 78,000 responses (so far) since we launched in March. We can see the most popular newsletters in terms of scores given, and also the least popular. We can see which newsletters led to many surveys filled in (a good sign of engagement), and which haven’t. 

Some very opinionated newsletters might have lower overall satisfaction scores because they are naturally more divisive. However, these also have higher engagement than a summary or daily briefing type of newsletter. Dividing our newsletters by format or category can result in fairer comparisons.

Here’s how we’ve utilized learnings from the survey so far: 

We tried to match readers’ motivations for subscribing with the content and form of the newsletter. For example, if our audience indicated that their main reason for reading the newsletter was the writer him or herself, then this could mean that the writer could afford to be more verbose. However, if the motivation was to quickly get up to speed with the latest news on a particular topic, then we would need to focus on ensuring that the newsletter was concise and informative. We have implemented changes due to reader responses, cutting back certain sections, for example, so they aren’t as long, or making things clearer. 

Many of the most popular newsletters based on reader survey engagement also boast high open rates, but it’s not always an exact match. The newsletters that have scored the best are the ones with huge personality and expertise. However, there are other newsletters with strong open rates that aren’t in the top 5  — perhaps because, although people enjoy reading them, they are not as personality driven, and so don’t offer such a strong, direct relationship. The newsletter at the bottom of our reader survey is a middling performer in terms of open rate and click through rate, but we’ve realized from this survey that obviously readers don’t feel passionate about it. As a result, we’ll be making some changes to this newsletter to see if we can improve it for readers.

From an editorial point of view, the verbatim comments from readers are incredibly valuable. Not only do we get some brilliant (and very amusing) comments along the lines of “X writer is a genius, give him a pay rise,” but we have also had useful remarks suggesting that: 

• Certain parts of a newsletter are confusing. 
• The email is too long or too short.
• Some readers would prefer more insight and fewer links. 
• Promoting Premium content in Standard newsletters is extremely frustrating for readers who can’t access this exclusive content.
• We can implement better signposting — for example, ending a weekly newsletter by calling out that it will be sent again at the same time the next week, so people know when to expect it and can build up a regular reading habit.

From this feedback, we’ve been able to make several changes to improve our newsletters for all readers.

We are aware that it’s the most engaged readers who fill in surveys, and that, as time goes on, the numbers of monthly responses  has dropped a little, as have the overall satisfaction scores. With tracker studies like ours, there is usually some type of leveling out. At this point in the survey, we’re mostly keeping a lookout for significant dips in these scores.

Another benefit of having such an engaged group of survey respondents: Many of them can be helpful with qualitative research in the future. About nine percent of survey respondents have indicated that they would be happy to answer further questions from the FT. We have already used some of these extremely engaged participants for feedback on a newsletter redesign — we’ve found that it’s helpful to have readers who can act as a sounding board for new ideas. 

What’s Next for the Reader Survey

That the newsletter survey has received more responses than expected means we will have to reconsider how much data we collect over the year. One unexpected outcome of all this feedback: We have a set number of responses that we are allowed to collect as part of our survey license, and the fact that we have already collected so many has eaten into a considerable proportion of that allowance. This is something we are revisiting as we discuss the future of the survey.

However we do decide to move on, we both agree that the survey has been a huge success, giving us insight that we simply did not have before, sometimes confirming suspicions (our readers really do like the personality-led newsletters!) and sometimes not. Our readers are not afraid to share their opinions, and that can only help us produce better newsletters going forward.

Sarah Ebner is head of newsletters at the Financial Times and was formerly head of email and editorial newsletters at the Telegraph in London. She has been working in newsletters for the past eight years, across writing, editing, and strategy.

Michael Hoole heads up the Audience Feedback Research Team at the Financial Times. He helps implement research programs across the business to better understand how the FT can continue to provide compelling and relevant content to a diverse audience. 

Thanks to our sponsors
The stories you’re reading on inboxcollective.com are made possible thanks to the generous support of our fall sponsors: ∙ Litmus, an all-in-one email marketing solution to help you create campaigns that convert ∙ Who Sponsors Stuff, a tool to help you find relevant sponsors for your newsletter