Posts Making the Infinite Flight Heatmap and Line Graph
Post
Cancel

Making the Infinite Flight Heatmap and Line Graph

Recently, I decided to create a heatmap of all flights on Infinite Flight’s Expert Server over a 24 hour period initially, then a 7-day period. For the 24 hour period, I fetched location data from the New API every 10 minutes. The data format I produces from this was as follows.

1
2
3
4
5
6
7
8
9
10
11
12
13
{
    "2020-10-18T13:00:03.000Z": [
        {
          "latitude": 4.368342265170154,
          "longitude": -39.999846486846685
        },
        {
          "latitude": 38.20144013415538,
          "longitude": 145.13290374996524
        },
		...
    ]
}

I put this all into a huge JSON file. It’s not ideal, but I needed this to be quick and easy to parse into a variety of formats. For different uses of the data. By this stage, I was hoping to make:

  • A heatmap of a 24 hour period
  • An animated heatmap of a yet to be determined day period
  • A line graph of a 24 hour period

All of these would require different formats.

Line Graph

I’d worked with Chart.js before, so I knew the line graph would be the easiest, and could also be done with very little data. I started the cron job at around 12am on the 19th of October. When I woke up in the morning, I had a lovely 8 hours of data to work with - more than enough for a line graph.

The chart was an almost exact copy and paste from a previous project, modified slightly to fetch the data client-side. After loading up the dataset I’d just downloaded from my server, it worked alright. The graph drew itself and the results were interesting, but the page was very slow to load. The reason being, it was fetching 40,000 lines of JSON, parsing it, and converting it to a better format. This took a good 1 second - locally.

No way that would work on GitHub Pages, so I needed to write something to build this raw data from the server to a better format. A few minutes and some Node later, I had the data is a "date": count format.

1
2
3
4
5
{
    "2020-10-18T12:52:00.000Z": 640,
	"2020-10-18T13:00:03.000Z": 642,
	"2020-10-18T13:20:02.000Z": 649,
}

Much nicer. With some small changes to the Chart code, it was much faster. Add some pretty colors, and viola!

Infinite Flight Expert Flights Over 24 Hours

24-hour Heatmap

Heatmaps were something I hadn’t done before. Originally, I was going to generate a single image in Node. However, after exploring how this could work, I decided it would be easier and have a better result to do it using Leaflet and heatmap.js. Having worked with Leaflet before, it wasn’t long before I had a nice map on a webpage running locally. Now I just needed to set up heatmap.js. Heatmap requires yet another format for data, so after making yet another Node application to format the data, into [{lat: 0.00, lon: 0.00, count: 1}, ...] format it was time to get working on the heatmap. Heatmap.js creates a Leaflet layer which you pass to the L.map constructor. I found heatmap.js to be great to work with - super customizable and easy to use. My first result was this.

It looked cool, but something was off. The color was not based on the traffic density in that area. Instead, each aircraft produced a small red dot that was (to an extent) independent. After some tinkering, I found this was a gimmick of heatmap.js and was happening because I loaded the data after the configuration. Heatmap has getters and setters for all properties, so it was easy enough from here to just set the configuration after adding the data.

Much better. ATC the previous day had been in China and was at the time of downloading the data in Europe, hence the density in those areas. You also see what you’d largely expect, with LAX, SFO and JFK all getting large amounts of traffic. You can also see the Atlantic Tracks were widely used, as there are few flights paths that don’t follow those 2 clear lines of the Atlantic.

Finally, I wanted to make the overlay stand out more against the map. So I switched the tiles to a darker set.

I finished this map just 36 hours after starting the cron job, and at the time of writing haven’t moved onto the animated heatmap, but here’s my plan for it.

The data format will be much the same as the format for static map. In fact, when I wrote the formatter for the static map, I included the functionality to generate timestamped versions.

The idea for the animated version is to take this array of datasets, initialize the map and heatmap, then loop through the datasets, updating the map with each one every 0.5 seconds.

The Finished(ish) Result

The finished map and static heatmap can be found here.

This post is licensed under CC BY 4.0 by the author.