Arcep is publishing the results of its 21st annual audit evaluating the quality of the services provided by mobile operators in Metropolitan France. This edition was marked by the Covid-19 crisis, forcing the measurement campaign’s timetable to be delayed.
2020, public health crisis and mobile quality of service
Arcep’s 2020 audit of mobile QoS was not spared the effects of the Covid-19 crisis. First, the measurement campaign had to be postponed till the autumn, following the lockdown that lasted from March until May, although usage tends to vary from season to season.
Moreover, during the lockdown, operators’ work on the networks was confined to essential service maintenance operations. A great many ongoing improvements that operators would normally have undertaken had to be postponed. Added to which, operators adapted their networks to optimise quality of service for static, indoor use, as users were travelling outside the home far less.
Lastly, telecom service usage has undergone a massive spike over the past several months, in particular due to an explosion in voice calling traffic from the networks[1].
This then is the backdrop for the more than 1 million tests that were performed on 2G, 3G and 4G networks across the country, in every department, indoors and outdoors, and on various forms of transportation, from August to November 2020. The audit covered the most widely used mobile services: web browsing, playing videos, data transfer, texting and voice calls. The tests performed on 266 indicators sought to evaluate the performance of operators’ networks in a strictly comparable fashion, and this in an array of circumstances.
There are still significant disparities in quality levels depending on the location and the operator: Arcep invites everyone to compare the findings, each according to the type of area where they live (high-density, medium-density or rural) and the type of transport they use.
MOBILE INTERNET (“Data measurements”): Ongoing increase in all operators’ service quality, albeit at a slower pace
The quality of every operators’ mobile internet services (data metrics) continues to improve, and this in every type of area: rural, medium-density and high-density, albeit at a slower pace than in previous years. This can be attributed in particular to the Covid-19 crisis (see inset) and the introduction of stricter protocols designed to more accurately reflect the user experience (in particular, new configurations for the servers used to perform the speed tests). The average downlink speed measured in Metropolitan France stands at 49 Mbit/s, compared to 45 Mbit/s in 2019.
Orange continues to improve the quality of its mobile internet services, particularly thanks to an average speed that has increased by 15 points in rural, high-density and medium-density areas. SFR has progressed by one percentage point on all data services indicators (speed, web browsing, streaming, etc.), putting it on par with Bouygues Telecom which has maintained its performance levels. Lastly, Free Mobile performed significantly better than the year before, with a QoS that is nearing the one provided by Bouygues Telecom and SFR. For both video viewing and web browsing, Orange has a three to five-point lead over its competitors (depending on the indicators), which follow with one or two-point differences from one another.
A new data indicator: percentage of speed tests achieving more than 3 Mbit/s
An indicator showing average speed is an interesting piece of information, but reflects only one aspect of quality of service. For instance, an operator that provides very limited coverage but very fast connections to those users who are covered may have a similar average speed to an operator that provides very broad coverage, but slower connections. The quality of the user experience will therefore differ between these two operators.
To complete the information provided by average speeds, this year Arcep has introduced a new indicator: percentage of connection speeds above the minimum threshold. In the findings of this audit, this threshold is set at 3 Mbit/s as, in most cases, a connection of more than 3 Mbit/s is enough to support “standard” mobile internet use, such as web browsing, reading e-mails and watching most videos in 720p without any significant slowing.
In France as a whole, this speed is achieved in 82% to 88% of cases, depending on the operator. The percentage in rural areas stands at between 71% and 77%.
VOICE CALLS AND TEXTING: Fewer and fewer audible disturbances on calls
Regarding the quality of calls on a national level, operators have increased their QoS by six to 11 points compared to last year. Orange provides the highest quality of service: 90% of two-minute calls are of perfect quality, widening the gap with Bouygues Telecom (86%) and SFR (84%). Free Mobile is the operator that has made the most progress in this area, going from 70% to 81% thanks to increased use of its own network, which it continues to deploy.
Regarding texting: all four operators provide an excellent quality of service: depending on the operator, between 93% and 95% of SMS relayed are received in under 10 seconds.
TRANSPORT CORRIDORS: gaps between operators are narrowing
The improvements made to data service coverage on roadways vary depending on the operator. Orange continues to lead the way with a 96% success rate for web page loads in under 10 seconds, which is 2% higher than the year before. Bouygues Telecom is holding steady at 94% and SFR has his improved its YoY performance by no fewer than eight points. Meanwhile Free has made tremendous strides: increasing its performance by 23 percentage points to reach 92%.
Coverage of TGV high-speed rail lines is also improving, even if it is not clear in all of the results. The measurement protocol used has become substantially stricter, out of test control concerns: this year, the test for assessing voice calling quality is conducted between two mobile phones onboard the train – as opposed to the previous method of the caller being on the train but using a fixed line. In addition, new trains that are better insulated but also less “porous” to the incoming radio signal, have been put into service on some lines. Worth noting is the three-point decrease in voice call quality for Bouygues and SFR, which stands at 70% of received calls, and the six-point drop for Free, down to 63%. Orange fares best, having improved its network and losing only one point, scoring an 80% call success rate.
Regarding mobile internet mobile quality on TGV high-speed trains: as some lines still need to be tested, TGV indicators will be updated later this year, along with the findings for underground metro services.
This year, because of the public health situation and the small number of regional trains that were running, Arcep and its service providers were unable to assess quality of service on railway lines other than the TGV and city undergrounds.
5G in the sights for 2021
Arcep is already preparing for its next measurement campaign. It will be marked by the first 5G rollouts in Metropolitan France: Arcep will be introducing a protocol that creates the ability to compare quality of service for a “5G-compatible” user (with a 5G-compatible plan and phone) and a user whose plan or handset is only compatible with 2G, 3G and 4G technologies.
Viewing the findings in map form on Arcep’s “Mon réseau mobile” website
The “Quality of service” tab on monreseaumobile.fr has been updated with the results being published today. The website now provides a map-based representation of the findings, in addition to a growing number of open datasets, by living environment. It allows users to view the results based on their own circumstances: living in a rural area, a high-density area, indoors, when travelling, etc.
Arcep’s interactive map tool allows users to compare mobile operators’ performance, by providing two types of complementary information: “simulated coverage” and “measured quality”. All of the data used to feed the “Mon réseau mobile” (My mobile network) site are available as open datasets.
All of the measurements are available as open data
Arcep makes all of the measurements obtained during this campaign available as open datasets on both its own website and on data.gouv.fr.
[1] See the market observatory for more information: https://www.arcep.fr/cartes-et-donnees/nos-publications-chiffrees/observatoire-des-marches-des-communications-electroniques-en-france/marche-des-communications-electroniques-en-france-t2-2020.html
Links :
- Map-based tool: monreseaumobile.fr
- Open datasets: https://www.data.gouv.fr/fr/datasets/monreseaumobile
Annex 1 : Scope of Arcep’s 2020 audit of mobile service quality
