5 things to consider when building a news feed

Here’s the central paradox of modern intelligence: We’ve never had more ‘news’, but never had less insight. 

Or perhaps slightly more accurately, it’s never been harder to generate real insight precisely because the number of news stories, opinion pieces, and blog posts published every single day runs into the millions.

That number represents a ‘wall of noise’ that overwhelms any attempt to make fast, accurate assessments around what is important, and which in turn can help drive effective decision-making. But attempt we must.

To start with the obvious, what certainly doesn’t work is manually consuming and evaluating everything within that wall. In the modern world, there is just too much data. When Samuel Taylor Coleridge died in 1834, it was claimed he was the last man on earth who had read every book in print at the time. Less than 200 years later, it would take hundreds of Coleridges hundreds of hours to manually evaluate and categorize just the news published in one 24 hour period. 

Of course, no organization today would take that exact approach. There are any number of services or tools that can, to some extent provide what we call a ‘news feed’: A more-or-less curated selection of stories that - in theory at least - puts what matters in front of us. To give the simple most well-understood example, a search on Google News does this job.

However, all news feeds are not created equal. And in this blog post, I’m going to look at precisely what attributes we might want to see in a news feed that really does the job. But before doing that, perhaps we should ask what that job is.

What is a news feed for?

This may seem like a stupid question. But it is only when we ask ourselves what precisely we are trying to achieve, that we can understand what the tool for the job might look like. And in fact, news feeds can be divided - roughly - into two groups.

The first might be classified as ‘news as entertainment’. These feeds provide some information about aspects of the world around us, often curated by a single site or publisher. And if we are being honest, the consumer of a feed like this isn’t looking for a huge amount more than a way to pass the time and keep up-to-date with the world. 

The second - and the one I will address - is ‘news for decision-making’. At Aylien, this is what we deal with, and the requirements in this scenario are qualitatively different. To put it simply, in this context ‘news’ is essential information that enables fast, accurate decision-making relating to risks, opportunities, investments and so on. 

The stakes, in this case, are sometimes higher than in pretty much any other context you can imagine (think of the importance of accurate, timely military intelligence in times of conflict or near-conflict). But across the board things that don’t matter a huge amount in the first category matter a lot more here. 

So what are those things that matter when it comes to delivering a news feed for decision-making? I would argue for five key criteria, some of which I have touched on already. These are, in no particular order:

1. Coverage 

Put simply: missing things isn’t an option. If it might matter, we need to see it. A news feed supporting decision-making must have not just adequate but exhaustive coverage, both in English and all other major global languages. Aylien, for example, continuously scans 80,000 sources, both behind and in front of the paywall. 

Of course, precisely how significant a news story has to be to feature in a news feed is ultimately up to your own individual requirements and configuration. But at the most basic level, the ability to catch everything is essential.

2. Timeliness

In the modern world, time to decision-making is everything. So a news feed relied on by professionals in fund management, for example, needs to surface adverse events as fast as it possibly can. Time, literally, is money. A good news feed needs to discover and distribute relevant news fast.

This in turn reminds us just how important it is to move away from manual processes and towards AI approaches that aggregate, understand, and deliver to the news feed as fast as possible.

3. Relevance

Coverage and timeliness are nothing without relevance. Unless the news feed puts the right information in front of the right people, everything else means nothing. 

It is here that so many news feeds fall down. To understand why that is, we need to understand the difference between a conventional keyword search and entity searching using Natural Language Processing (NLP).

The former is absolutely perfect if we want to aggregate news about Zurich insurance and Zurich the city in Switzerland. The latter is preferable when we just want one or the other (plus a whole host of more detailed information). NLP allows news to be understood, and filtered by meaningful entities that we have defined rather than simply names. 

But more than this, we can also evaluate sentiment, relationships between entities, and automatically aggregate similar stories in order to estimate importance. In other words, we can deliver a news feed that presents us with precisely the information we need, and only that information.

4. Scalability

Any news feed solution needs to be flexible and scalable. What a group of decision-makers cares about can change, and often changes quickly. And the content of our news feed needs to be able to do the same thing too.

We need to be able to add new entities of interest, without having to create long, cumbersome search strings. We need to add publications without building bespoke scraping tools. And we need to be able to adapt or expand the criteria we use around relevance on-the-fly.

Scalability also implies the ability to share a news feed across potentially millions of endpoints, particularly when embedded into customer-facing apps such as with Aylien customer Revolut. Whatever technology your news feed is built on must be ready to meet this challenge without extensive re-engineering.

Lastly and perhaps most obviously, we need to avoid manual intervention at all times. True scalability means that doubling the amount of news we discover and distribute to our news feed should have no impact on performance.

5. Trustworthiness

Last, but not least, we need to be able to trust what we read. In an era of fake news, the credibility of sources needs to be checked constantly, with only those that are most trusted surfaced to the feed. Precisely for this reason, Aylien constantly evaluates and vets all publishers - so our customers don’t have to.

On a similar note, the sheer volume of stories around a particular event will give a fast way to evaluate significance and establish the ‘truth’ of a story. A good news feed will communicate that information without showing every individual piece, but rather a selection of representative ones from most trusted publishers.

If you work in any industry that relies on up-to-the-minute insight powered by real-world events, it’s well worth asking yourself whether the feeds you rely on meet these criteria. 

And if they don’t, you might want to drop us a line, or try our 14 day free trial of AYLIEN News API.