Arcep is publishing the results of its 23rd annual audit evaluating the quality of the services provided by mobile operators in Metropolitan France. The mobile QoS audit is based on more than a million measurements taken of 2G, 3G, 4G and 5G networks in every department of Metropolitan France, between late May and the end of August 20222, in living environments indoors and outdoors, and on various forms of transportation.
There are significant disparities in quality levels depending on the location and the operator: Arcep invites everyone to compare the findings using the “Mon réseau mobile” tool, each according to the type of area where they live (high-density, medium-density or rural) and the type of transport they use.
MOBILE INTERNET QoS: KEY FINDINGS
- All four operators provide a high quality of mobile internet service in densely populated areas. Orange provides the best quality of experience in medium-density and rural areas.
Regarding 2G/3G/4G web browsing in high-density areas, all four operators scored very high on the success rate for loading a web page in under 10 seconds: 98% for Orange, 97% for Bouygues Telecom, 95% for Free and 94% for SFR. In rural areas, Orange (90%) was followed by Free Mobile (86%) then Bouygues Telecom and SFR (81%).
For video streaming in densely populated areas, Orange and Bouygues Telecom lead the rankings with 98% and 97%, respectively, of videos streamed with perfect quality, followed by Free Mobile (94%) and SFR (93%). In medium-density and rural areas, Orange tops the rankings (96%), with a more than five-point lead over Free Mobile (91%), Bouygues Telecom (90%) and SFR (88%). Orange also scored highest (89%) in rural areas where Free Mobile ranks second (85%), ahead of Bouygues Telecom and SFR (78%).
Comparing use of these same applications in 2G/3G/4G/5G, the user experience appears to be similar to the QoE in 2G/3G/4G. 5G is still being deployed, and its immediate benefit lies chiefly in the additional capacity it provides in locations where mobile networks are heavily solicited, if not saturated. Moreover, the current 5G configuration, i.e. non standalone (NSA), means that 5G still relies heavily on 4G[1]. Standalone (SA) 5G is due to be deployed in the coming years, which will make it independent of 4G. In any event, operators will continue to optimise their 5G rollouts, as they have done with 4G which has undergone and constant series of improvements and adjustments for over close to a decade. This will improve the quality of new uses cases for consumers.
- 3G/4G/5G downlink speeds stand at 94 Mbits/s for all operators combined, and are well above 3G/4G speeds (63 Mbits/s).
Average downlink speeds on 3G/4G/5G networks are considerably higher than those obtained on 3G/4G networks, and improved more significantly than they did in 2021:
For 2G/3G/4G/5G users, Orange provides the fastest downlink speeds, averaging 143 Mbit/s across the whole of Metropolitan France and 217 Mbit/s in high-density areas. They are followed by SFR, averaging 84 Mbit/s across the whole of Metropolitan France and 163 Mbit/s in high-density areas, tied with Bouygues Telecom (84 Mbit/s in Metropolitan France, 167 Mbit/s in high-density areas). Free closes out the ranks across the whole of Metropolitan France with 64 Mbit/s on average, and providing virtually the same performance in high and medium-density areas, but ranking second in rural areas.
Worth noting is that Arcep’s downlink speed testing protocol was amended this year, to test the different congestion avoidance algorithms. To more accurately reflect changes in users’ consumption, 75% of the tests were performed using the Cubic algorithm and 25% with the BBR algorithm [2].
- After years of steady increases, several indicators measured in 2022 decreased compared to 2021: this may be attributable to the aftermath of the Covid crisis and the high temperatures during summer months when the audit was performed
Changes in several “mobile internet” indicators, in 2G/3G/4G in high-density, medium-density and rural areas, all operators combined, since 2018: success rate for loading a web page in under 10 seconds (left), rate of video viewing with perfect quality (centre), average downlink speed (right)
The difference in the performances observed in 2021 and in 2022 can be attributed to two phenomena:
- First, the measurement campaign in 2021 was carried out from May to July, on the tail end of a lockdown, a period during which people were still travelling little and mobile network traffic was lighter. The 2022 campaign was carried out during an “ordinary” year in terms of traffic load.
- Second, the measurement campaign in 2022 was carried out from June to August, in the heart of summer, during which networks are subject to local spikes in traffic, particularly in popular tourist destinations, which can deteriorate users’ quality of experience.
Good to know: 5G devices can’t stand the heat
The mobile internet QoS audit was carried out from June to August 2022, during which there were several heatwaves. To avoid overheating and reduce the device’s internal temperature, some 5G handsets are capable of running all apps in the background on 4G, and to switching over automatically from 5G to 4G, which alters the device’s performance. This phenomenon is especially common when the mobile’s temperature exceeds 42°C.
To take this 5G device protection mechanism into account, Arcep introduced preventive measures in the field during the audit, opting to perform the measurements in the morning in those locations enduring a heatwave (temperatures above 35°C), and to postpone testing in certain modes of transport that were not air conditioned, conducting them instead during cooler weeks.
In addition, to more accurately reflect the quality of service provided by 5G, Arcep elected to eliminate testing of the 5G-compatible chain when, at the same testing location, operators’ mobiles exceeded 42°C. Testing performed at these locations with the 2G/3G/4G measuring chain was not eliminated.
VOICE CALLS AND TEXTING: KEY FINDINGS
QoS in 2022 higher than in 2021
The quality of calls progressed nationwide in 2022. Orange (93%), Bouygues Telecom (93%) and SFR (91%) scored similarly high on the success rate for maintaining a 2-minute call without audible interference, followed by Free Mobile (87%). The trend is similar for medium-density areas (92% for Orange, 91% for Bouygues Telecom, 90% for SFR and 85% for Free Mobile) and in rural areas (83% for Orange, 80% for Bouygues Telecom, 79% for SFR and 75% for Free Mobile).
Operators delivered comparable performances in terms of average call quality: the gaps in MOS[3] were very small between SFR, Orange, Bouygues Telecom and Free Mobile (3.9 on average across all areas).
Average call setup times (between the moment when the call is placed and the first ringtone) stood at 2.8 seconds with SFR and Bouygues Telecom, 2.9 seconds with Orange and 3.3 seconds with Free Mobile.
All operators enable customers to receive a text in under 10 seconds, very often with very little difference between Orange (98%), Bouygues Telecom (98%), SFR (97%) and Free Mobile (97%).
TRANSPORT CORRIDORS: KEY FINDINGS
QoS improving on all transport corridors
On average, internet quality of service is very high, with a close to 95% success rate, all operators combined, for web page loads in under 10 seconds. The situation is more nuanced on railway lines where the success rate for web page loads in under 10 seconds stands at only 81%, on average, on TGV high-speed trains, Intercités and TER lines. Browsing is more fluid on RER and Transiliens commuter trains in the Paris region (90%) and in the metro (95%).
Results on an operator-by-operator basis reveal that Orange scores highest for web browsing quality on all types of road and railway line.
Orange had a 97% success rate for web page loads in under 10 seconds on roadways, Bouygues 96% and SFR 95%, followed by Free Mobile with 92%.
Orange also delivered the strongest performance on long distance railway lines, with an 88% success rate for web page loads in under 10 seconds on the TGV high-speed rail lines, ahead of Free Mobile (80%), Bouygues Telecom (79%) and SFR (78%). On Intercités and Regional Express networks, Orange had a success rate of 86%, followed by Bouygues Telecom (81%) SFR (79%) and Free Mobile (78%).
On commuter lines in the Paris region (RER and Transiliens), Orange scored highest as well, with a 95% success rate in page loads in under 10 seconds, this year only slightly higher than Bouygues Telecom (92%) and SFR (91%), and somewhat higher than Free Mobile with 83%. On metro lines, all four operators are providing a good quality of service, with Orange (96%), SFR and Bouygues Telecom (95%) ahead of Free Mobile (92%).
Regarding calling quality, Orange and SFR are neck and neck on roadways with the strongest performances (95%) for calls maintained for 2 minutes, compared to 93% for Bouygues Telecom and 89% for Free Mobile.
The QoS trend for calls on trains is the same as the mobile internet QoS trend, with Orange delivering the best performance on long distance railway lines, scoring 81% on call maintenance on TGV lines, compared to 71% for Bouygues Telecom, 68% for SFR and 61% for Free Mobile. On the RER and Transiliens commuter networks, Orange scored 91% and Bouygues Telecom 90%, ahead of SFR (85%) and Free Mobile (82%).
On city metros Bouygues Telecom had the highest success rate of calls maintained for 2 minutes (96%), outperforming Orange (94%) and SFR and Free Mobile (92%).
All the findings are available as open data
Arcep makes all the measurements obtained during this campaign available as open datasets on both its own website and on data.gouv.fr.
Annexes:
- Annex 1: Scope of enquiry in 2022
- Annex 2: Summary of 2022 findings
Links:
- Map-based visualisation tool: monreseaumobile.arcep.fr
- Open data: https://www.data.gouv.fr/fr/datasets/monreseaumobile
[1] For more details, go to: https://www.arcep.fr/fileadmin/cru-1664269931/user_upload/grands_dossiers/5G/aspects-techniques-5G-imbrication-entre-4G-et-5G.pdf
[2] For more information, see page 29 of the Arcep 2022 report on the State of the Internet in France: https://www.arcep.fr/uploads/tx_gspublication/rapport-etat-internet-2022-300622.pdf#page=29
[3] “MOS” (mean opinion score) for call maintenance which measures the difference between the call being tested and the benchmark sample
Annex 1: Methodology and scope of the mobile QoS enquiry
Mobile quality of service audit based on a more than a million measurements taken in 2G, 3G, 4G and 5G
More than 1 million measurements were taken of 2G, 3G, 4G and 5G networks in every department across the Metropolitan France, in living environments indoors and outdoors, and on various forms of transportation. The audit covered the most widely used mobile services: web browsing, video streaming, data transfer, texting, and voice calls. The tests that were performed sought to evaluate the performance of operators’ networks in a strictly comparable fashion, and this in an array of circumstances.
For mobile internet services, Arcep has implemented a protocol that creates the ability to test quality of service in a same location, for both a user employing only 2G, 3G and 4G networks, and a user with a 5G-compatible plan and phone. The “5G” results contained in the findings are those obtained from 5G-compatible tests performed across the whole of Metropolitan France, to measure the speeds that a user can expect to have for their daily use, regardless of whether they are connected to a 5G cell tower.
Annex 2: summary of 2G/3G/4G results
Calls: the success rate for calls maintained for two minutes and of calls maintained for two minutes without audible disturbance (dark colours). On modes of transport, calls are made between two SIM cards belonging to the same operator (on-net calls). In living environments, calls are made using every SIM card combination (on-net and off-net).
Texting: success rate of SMS received in under 10 seconds
Voice quality: Calls’ mean opinion score (MOS) – an automated assessment of voice quality, using the POLQA algorithm
Web: success rate of web pages loaded in under 10 seconds and in under 5 seconds (dark colours)
Streaming: success rate of videos streamed with a decent viewing quality and with perfect viewing quality (dark colours)
Downlink speed: average downstream speed, in Mbit/s
Uplink speed: average upstream speed, in Mbit/s
Speeds of ≥ 3 Mbit/s: percentage of speed tests that display an average connection speed equal to or above 3 Mbit/s