Sunday, September 27, 2015

Why Strava segment matching and timing sucks (and how to cheat)


I'm enjoying Strava segments. I find segments motivating and fun, and even if it should be considered just a game (and not serious competition), fact is that there is some prestige to be had for holding the KOM on certain segments. However, Strava doesn't really try to get accurate segment times, so if you lost your KOM (or gained a KOM) by a small margin, then there is a chance that the result is wrong. This is of course true also further down the leaderboard, so if your mate beat you with a small margin, there is a chance that in reality he didn't.

Strava's segment timing

The reason for this is no secret, it is because Strava makes no effort to calculate your actual segment time, they just choose the data point from your ride that is closest to the start (or end) point of the segment as the basis for their timing. Here's what Strava says:
Recording intervals vary between devices - for example, the Strava mobile app records every second while Garmin devices use either 1-second intervals or a smart recording which has a varied recording interval. Segment matching works the same on each GPS dataset, but depending on the device's recording interval, can yield different results. Segment matching uses the GPS points in the data closest to the start and endpoints of the segment, and as this can vary with each activity, timing on a segment can vary slightly because of this. At the present time, we don't interpolate or extrapolate GPS data to normalize for the exact start and end positions of the segment.
The highlight is mine, and it summarizes the issue in just one sentence. Strava also makes a note that Garmin devices has a recording mode called 'Smart Recording', which is a dynamic mode where the unit decides (depending on a few metrics) if it should write a data point to the log or not. If you ride in a straight line, the unit will typically increase the recording interval, waiting a few seconds between log points. Now, if a segment starts on a straight section, devices with smart recording (or otherwise a long recording interval) will most likely miss the segment start point by quite a large margin. The same is of course also true for the segment end point, and armed with the knowledge that Strava just chooses the closest point, we can see that there are four possible scenarios:

  1. The log start point is before segment start and the log end point is after segment end
  2. The log start point is before segment start and the log end point is before segment end
  3. The log start point is after segment start and the log end point is after segment end 
  4. The log start point is after segment start and the log end point is before segment end
For segment timing, scenario 1 will yield the worst segment time, while scenario 4 will give the best segment time. As for the two in the middle, whether you gain or lose all boils down to the actual distances by which you miss the segment start and end. For example, if your log start point is just before segment start, and the end point is quite a distance before segment end, then you clearly gain.

An example

I guess you might have become bored if you've read this far, so here's a visualization from a random segment (but one that I'm familiar with):

Segment start

This is the start of the segment, and the segment track itself is shown in red (I've added a virtual starting line perpendicular to the segment track) . The colored tracks are the efforts of the top 10 from the segment leaderboard, and as you can see (if you look closely), there are 4 efforts that start early, and 6 efforts that start late. The yellow rider is the one that gains the most, and his time effectively starts approx 17 m (55 ft) into the segment, gaining him ~1.2 seconds in this case.

Segment end

To the right you can see the end of the same
segment. Again, if you look closely, you can see that half the efforts end early, and the other half overshoots the finish. The green rider's effort ends approx 19 m (62 ft) after the segment end, causing him to lose ~1.4 s. The yellow rider's effort ends just before the finish, gaining him a couple of tenths, so in total he gained ~1.3 seconds. He was clearly using smart recording, because for this effort his device logged one data point each 3.1 seconds on average. The yellow rider's effort was a case of scenario 4, as listed above.

The green rider's effort was a case of scenario 3, and even though he lost some on the finish, he gained some at the start, so in total he gained just a couple of tenths.

How much error does Strava allow (and how to cheat)

I seem to recall that Strava requires your ride to match the segment 75% of the time, but I can't find any accurate references to this at the moment. If true, it means you could pass the segment start, and somewhere along the way make a shortcut and then pass the segment end and still get a segment time. I'm a bit skeptical that this still holds true, as I think perhaps Strava has improved this.

However, let's get back to the issue at the start and end of segments. We know that Strava chooses the closest data point, but are there any limits to how far off a point might be, and still be considered valid by Strava (clearly there is, but bear with me). To find out, I took one of my own rides where I rode this segment, and started removing data points on both sides of the segment start point, and then uploading to Strava for each iteration, until I no longer got a match for the segment. I didn't care to do this in small enough increments to find the exact limit, but I can say that it seems to be somewhere between 40 and 50 m (131 to 164 ft). In other words, as long as you have a data point that is within this distance of the segment start (and end) point, you will get a match for the segment (of course provided you also ride the segment, but that's stating the obvious).

Now, I'm sure you're all curious what effect this had on my times on this segment, and here they are:

Pos   1: Manipulated effort   time:  2:47.00
Pos   2: Manipulated effort   time:  2:50.00
Pos   3: Manipulated effort   time:  2:51.00
Pos   4: Genuine effort       time:  2:55.00
Pos   5: Manipulated effort   time:  2:57.00

Remember, I didn't really do anything but weed out some data points at strategic places, I didn't change any of the data (like speed or time) in the rest of the data set, I just removed some points, much like what happens naturally with devices that use 'Smart Recording'. The main difference is that with those devices the outcome is more random, and you'll both win some and lose some.

The worst result in this list was when I accidentally got it wrong, and the last data point before segment start was a wee bit closer than the first data point after. For one of these efforts, I removed data points both at the start and at the end, and as you can see, I then gained a whopping 8 seconds on my own genuine effort. From the same data set!

For reference, here's images showing what this (manipulated) effort looks like:

Segment start - manipulated
Segment end - manipulated

How Strava could fix this

At the start of this post, I quoted Strava where they state that they don't interpolate or extrapolate to calculate more accurate segment times. However, had they done so, they could have increased the probability of getting correct leaderboards many times. I'm not saying it would become entirely accurate because clearly it wouldn't, but why should that be an argument for not trying? I've created my own piece of software that can do this (with some limitations, since the Strava API doesn't allow downloading of the entire data stream for other athlete's activities). Here's the results for the exact same efforts as above (sorted, obviously):

Pos   4: Genuine effort       time:  2:54.97 (corr: -0.03)
Pos   5: Manipulated effort   time:  2:55.10 (corr: -1.89)
Pos   3: Manipulated effort   time:  2:55.13 (corr: +4.13)
Pos   1: Manipulated effort   time:  2:55.17 (corr: +8.18)
Pos   2: Manipulated effort   time:  2:55.25 (corr: +5.26)

The number in parentheses is the amount of correction that was applied as a result of the interpolation, and as you can see, the difference across the board is still within 3/10 of a second!

I've also done some more interesting experiments, like riding with two Edge units, one using 1 sec recording, the other set to smart recording. While the majority of the results (for segments I passed) were close (luckily), I also observed exactly what I have written about in this post: A big discrepancy (4 seconds on Strava) between the device using smart recording compared to the one that used 1 sec recording. After processing the data in my software, the difference was within a few tenths.

I might write some more about this, if time permits.