This week I started playing around with Astra, and I’ve been genuinely impressed. I also implemented a GDELT toolkit for Toolforest, which gave me something to try it on beyond asking whether it was impressed with itself.
I wanted to see what Astra could do with news data alongside the prediction-market and Earth toolkits I already use. I asked it to help me work through a few experiments, find useful applications, and see which results would stand up to a closer look.
The most useful result came from looking at the coverage itself. I found eighty-two websites carrying what appeared to be the same account of an incident in Minna, Nigeria. Several had different headlines. They looked geographically diverse. The text was less well travelled than the mastheads suggested.
That gave me something worth writing about.
News you can query
GDELT processes news coverage into structured data: people, organizations, themes, places, and coded events. Toolforest now gives Astra access to a recent slice of that data, through a backend I run myself.
The toolkit can retrieve article metadata, plot coverage over time, compare geographic distributions, and follow coded events back to the articles that mention them. The current article interface filters GDELT’s extracted metadata. It does not search the full text of every article, and it does not certify that a reported event happened.
Eighty-two websites
The reporting concerned deaths in custody and the protests that followed in Minna. I wanted to examine how that story appeared across different outlets.
I asked Astra to query articles mentioning Minna, Nigeria, over September 18–24. After Astra followed pagination through the returned results, I had 551 distinct URLs. That was a collection of articles mentioning the place, not a clean count of reports about one incident.
Within it, eighty-two URLs on eighty-two domains shared the publisher’s story identifier, 279315982. Seventy-one carried one headline; eleven carried another. They formed a strong candidate for a syndication cluster.
I had Astra open three versions: Big News Network, New York Statesman, and Central Africa News. They carried substantially the same account, the same byline, and reporting attributed to Reuters.
I did not inspect all eighty-two bodies. The shared identifier is evidence of a distribution pattern; the three inspected pages support that interpretation. It would take more inspection to certify every member of the group.
Even with that limit, the result changes what the numbers mean. Eighty-two domains can look like extensive independent attention. Here, a substantial part of the apparent breadth came from distribution of a common account.
GDELT’s outlet-country mapping assigned the group to thirty-seven country codes, with some entries unmapped. That mapping comes from a dataset published in 2015. It cannot establish where the current newsroom sits, who read the story, or whether anyone in those countries independently reported it. The geography was worth examining precisely because it was so easy to overinterpret.
The eighty-two records appeared within a five-hour span of GDELT observations. Those are observation times. They do not tell me that the websites published within five hours, much less that readers encountered the story then.
A world map of this result would have looked impressive. A table of matching story identifiers was more informative.
Studying the coverage itself
This was the clearest answer to my recurring question: couldn’t I just use web search?
In an earlier experiment, GDELT found WCAX’s report of a Vermont governor’s race poll that helped explain a prediction-market move. Ordinary web search could find the same report, and GDELT picked it up after trading had already begun to react. I could explain the move, but I had not demonstrated an advantage over search.
Minna gave me a more useful comparison. Finding an article about the incident was easy. Examining the distribution of coverage required a collection of records.
Here I wanted to examine a defined set of coverage: collect the returned records, retain the query window, group domains and identifiers, and inspect the suspicious groups. GDELT supplied data I could work through systematically. Search could have found examples of the duplicated article, but I would still have needed a collection method to measure the pattern.
The toolkit did not perform the clustering for me. I had Astra group the returned URLs during the analysis and check sample pages separately. Clustering is a plausible backend addition, and this experiment provides a concrete reason to build it.
It also suggests useful media questions. How much of a coverage spike comes from republication? Which stories travel through a particular publishing network? Do different headlines put different emphasis on the same underlying text? Does an apparent increase in international attention survive grouping the copies?
These are narrower questions than “what does the world think?” They have the advantage of being answerable from the data I actually retrieved.
I also tried comparisons of local and international coverage across several countries. The examples did not support a tidy general account of how the two differ. International outlets sometimes carried the detail I expected to find only locally. Ordinary search found some of the supposedly hard-to-find material immediately. I would want a more carefully constructed sample before making claims about national media habits.
For forecasting, that distinction matters too. An apparent rush of corroborating reports should carry less weight if many are copies of a common account. Minna demonstrated a way to investigate that dependence. It did not demonstrate an improvement in forecast accuracy.
The backend gets an opinion
The Minna analysis depended on a few unglamorous capabilities: a defined time window, pagination through the matching records, source URLs, and observation timestamps whose meaning was explicit. These made it possible to collect and inspect the results instead of relying on whichever examples fitted in the first response.
The backend preserves those distinctions. Deduplicating URLs still leaves different websites carrying the same text, as this experiment showed. Grouping syndicated stories would be an additional operation, with its own evidence and uncertainty.
There are separate paths for article geography and coded event geography. The first describes places mentioned in coverage. The second exposes GDELT’s extraction of where an action occurred. Event mentions let me inspect the reporting behind a coded event. Extraction confidence helps prioritize that inspection; it does not settle the facts.
The ingestion system also keeps track of which source files were actually advertised and successfully loaded. A missing result needs that context. An empty interval might be quiet, or the data might be incomplete. The response should give Astra enough information to tell which explanation remains possible.
The repository work produced a few less philosophical surprises. During event verification, about ten percent of the sampled mentions referred to an event recorded in the other language stream. English and translated data share an event-ID space. Resolving references separately by stream had quietly lost valid connections. The service was changed to resolve by global event ID.
Historical references required their own work. A newly observed article can mention an older event, so a recent serving window needs access to older event records. The thirteen-month backfill completed all 73,341 planned archives without a failed or quarantined archive. Its completeness report still required investigation: recent events had not been kept current in the durable lookup used by that check. Finishing the download was one accomplishment. Making every layer agree about what had finished was another.
There was measurement work beneath the API as well. Direct queries over Parquet failed the representative geography performance gates. The service uses ClickHouse for recent queries and keeps canonical Parquet for archive and recovery. An attempted query optimization was discarded after it measured slower. This is a satisfying outcome for a benchmark, if an awkward one for the optimization.
Where the Earth toolkits could fit
I still want to connect this work to the Geo, Earth Observation, and Google Earth Engine toolkits. A report of flooding could lead to a comparison of water extent. A report of drought could lead to rainfall or vegetation history. GDELT could supply candidate reports, Geo could define the area, and an Earth toolkit could retrieve a measurement relevant to the claim.
Location needs checking first. An article might mention a port where something happened, where a ship departed, or near an offshore incident. Each can produce a point on a map. Before asking Astra to fetch satellite imagery, I need to establish what that point represents.
Once Astra has coordinates, it can calculate distances and request imagery quite efficiently. It can do both for the wrong place. Connecting news reports to Earth observations remains an application I want to test, with the location, dates, and available measurements checked along the way.
For now, studying the coverage has earned its place. GDELT gave me a collection I could examine, and Astra helped work through it. Eighty-two websites initially looked like a great deal of evidence. After inspection, they looked like a distribution system.
I had learned something. Fortunately, I only had to learn it once.
As always, ideas and suggestions are welcome at gerrit@toolforest.io.