Mobile Quality of Service

MOBILE INTERNET: Significant increase in service quality in 2021 for every type of use, after a 2020 marked by the Covid-19 crisis

The quality of every operator’s mobile internet services (data metrics) improved significantly, and this in every type of area: rural, medium-density and high-density.

This trend is particularly visible in web browsing and online streaming results: in high and medium-density areas, the success rate for loading a web page in under 5 seconds stood at 98% for Orange, followed by Bouygues Telecom (96%), SFR (95%) then Free Mobile (93%); in rural areas Orange (90%) was followed by Free Mobile (83%) then Bouygues Telecom and SFR (81%). This indicator gained an average eight points for every operator in rural areas, compared to 2020. For video streaming in densely populated areas, Bouygues Telecom leads the rankings (94% of videos streamed with perfect quality), followed by Orange (91%), Free Mobile and SFR (90%). In medium-density and rural areas, Orange topped the rankings, with a four-point lead over its competitors (worth noting here is that Free was in second place for streaming service QoS in rural areas). For all operators combined, streaming quality improved by nine points compared to last year.

Downlink speeds on 2G/3G/4G networks reached an average 71 Mbit/s, compared to 49 Mbit/s last year, which marks a steady increase after the decline in average connection speeds in 2020 due to the Covid-19 crisis.

All types of area combined, the average connection speed measured for Orange stands at 110 Mbit/s, followed by SFR (69 Mbit/s), then Bouygues (55 Mbit/s) and Free Mobile (50 Mbit/s). In rural areas, Free Mobile provided the second fastest average speeds, behind Orange.

Close-up on the speed testing protocol: single vs. multiple parallel connections

Several years ago, Arcep made the choice of performing “single-thread” speed tests, in other words of opening and using a single connection between the mobile device and the server hosting the file to be downloaded. This is the most common configuration on the internet today. Even when several applications are open on a mobile, in practice, typically only a single stream is being used at any given time. At this stage, then, this protocol seems to most accurately mimic customers’ reality, and creates the ability to highlight operators’ network optimisation efforts in this area.

It is also possible to perform “multi-thread” tests which reveal the network’s “maximum capacity” in terms of speed. This is the approach taken by a great many crowdsourced speed test apps, which can explain the difference between the speeds measured by Arcep and those displayed by crowdsourcing apps.

Arcep is also publishing an indicator that was added to its enquiry last year: the percentage of connection speeds above the threshold of 3 Mbit/s[1]. This percentage is particularly high in high- and medium-density areas, with comparable performances from the operators (between 94% and 99%). In rural areas, the gap is more significant, with 90% of speeds measured for Orange exceeding 3 Mbit/s, compared to 84% for Free Mobile and 79% and 77% for SFR and Bouygues Telecom respectively.

This indicator, for all operators combined, rose by 8 points in rural areas: from 73.9% to 82.3% – reflecting operators’ rollout efforts in more sparsely populated parts of the country, which means that, for the most common types of use, the gap in quality of service between rural and urban areas continues to shrink.

SPOTLIGHT: First 5G QoS tests

For the first time, Arcep has implemented a protocol that creates the ability to test quality of service for a user with a 5G-compatible plan and phone, and so measure downstream and upstream speeds. The indicator being published here presents the average speed obtained with 5G-compatible tests across the whole of France, to measure the speeds that a user can expect to have for their daily use, regardless of whether they are connected to a 5G cell tower.

Orange provides the fastest downstream speeds, with an average of 142 Mbit/s across the whole of France. Orange 5G users in high-density areas, where most of the operator’s 5G cell sites are deployed, have access to an average connection speed of 227 Mbit/s. This is followed by SFR, with 84 Mbit/s on average in the whole of France and 145 Mbit/s in high-density areas, then Bouygues Telecom (71 Mbit/s on average, 130 Mbit/s in high-density areas). Free is in last place with 31 Mbit/s on average, with little difference in the speeds provided in high-density, medium-density and rural areas.

Upstream speed performances for 4G and 5G are comparable.

This indicator is completed by an “end-to-end” 5G rate, which represents the percentage of speed tests performed with 5G-compatible tests that actually used 5G to download files. According to this indicator, Bouygues Telecom leads the way (60%) in high-density areas, followed by Free (49%), Orange (46%) and SFR (36%). In medium-density and rural areas, the end-to-end” 5G rate is substantially higher for Free Mobile than for other operators (50% in medium-density, 27% in rural areas).

These are the first QoS measurements performed on 5G networks that operators are still in the process of deploying and optimising. Consumers are invited to consult operators’ own coverage maps to check whether the technology is available in the area where they live and/or work.

VOICE CALLS AND TEXTING: QoS in 2021 comparable to 2020

Regarding the quality of calls nationwide, Orange provides the highest quality of service: 86% success rate for maintaining a 2-minute call without audible interference, followed by Bouygues Telecom and SFR (82%) then Free Mobile (81%). The trend is similar for texting: the success rate for SMS received in under 10 seconds stands at 96% for Orange, 94% for Bouygues Telecom and SFR, and 92% for Free Mobile.

Call setup time: a new indicator measured in 2021

To fully factor in every aspect of quality of experience (QoE) for mobile calls, this year Arcep measured call setup time – i.e. the time between the moment when the caller places the call and when they hear the first ringtone.

The published indicator is the average call setup time. Bouygues Telecom and Orange lead the rankings for this indicator (1.9 seconds), followed by SFR with a very small, if not imperceptible difference for users (2 seconds). Free Mobile lags significantly behind with an average lag time of 3 seconds. Worth noting is the only slight difference between setup times in rural areas (2.4 seconds on average for the four operators) and high-density areas (2.1 seconds).

TRANSPORT CORRIDORS: QoS gaps between operators are narrowing

Following through on the trend seen in 2020, the gaps between operators’ quality of service on roadways continue to be small. Orange leads the way with a 97% success rate for web page loads in under 10 seconds, followed by Bouygues Telecom (95%), SFR (94%) then Free Mobile (93%). The trend is the same for the success rate of calls maintained for 2 minutes: 93% for Orange, compared to 91% for Bouygues Telecom, 88% for SFR and 84% for Free Mobile.

On TGV high-speed rail lines, Orange is widening the gap on web browsing quality, with close to 85% of pages loaded in under 10 seconds, or around four points more than last year, and 13 points ahead of Bouygues Telecom, Free Mobile (72%) and SFR (71%). Orange also scores highest for voice call QoS: 79% of calls maintained for 2 minutes, which is 13 points higher than Bouygues Telecom, 15 points higher than SFR and 16 more than Free Mobile.

Performance levels on city metros is similar to 2020, with the notable exception of Lille whose number 2 line was outfitted with 4G in early 2021, coming to join the 1 line which has been covered since early 2018: the success rate for web pages loaded in under 10 seconds has thus soared from 45% to more than 97% on the city of Lille’s metro.

The quality of service on “Intercités”, “Transiliens” and “RER” railway lines was measured once again in 2021, after having been impossible to perform in 2020 due to Covid-19. On Intercités and TER lines, voice calling and data service quality is comparable to what is provided on TGV lines, but Orange has a slightly smaller lead over its competitors – scoring higher than Bouygues Telecom, Free Mobile then SFR whose results are very similar. On Transiliens lines, while Orange provides a better web browsing experience (95% success rate, or 6 points ahead of the other operators), it is neck and neck with Bouygues Telecom when it comes to the success rate of maintaining calls for 2 minutes (90%, compared to 85% and 84% for SFR and Free Mobile).

All the measurements are available as open data

Arcep makes all the measurements obtained during this campaign available as open datasets on both its own website and on data.gouv.fr.

Annexes:

• Annex 1: Scope of enquiry in 2021
• Annex 2: Summary of 2021 findings

Links:

• Map-based visualisation tool: monreseaumobile.arcep.fr
• Open data: https://www.data.gouv.fr/fr/datasets/monreseaumobile

Note:
[1] In most cases, this speed supports all “standard” mobile internet uses (browsing, mail…)

Calls: the success rate for calls maintained for two minutes and of calls maintained for two minutes without audible disturbance. On modes of transport, calls are made between two SIM cards belonging to the same operator (on-net calls). In living environments, calls are made using every SIM card combination (on-net and off-net).

Texting: success rate of SMS received in under 10 seconds

Voice quality: Calls’ mean opinion score (MOS) – an automated assessment of voice quality, using the POLQA algorithm

Web: success rate of web pages loaded in under 10 seconds and in under 5 seconds

Streaming: success rate of videos streamed with a decent viewing quality and with perfect viewing quality

Downlink speed: average downstream speed, in Mbit/s

Uplink speed: average upstream speed, in Mbit/s

Speeds of ≥ 3 Mbit/s: percentage of speed tests that display an average connection speed equal to or above 3 Mbit/s