Arcep is publishing the results of its annual audit evaluating the quality of the services provided by mobile operators in Metropolitan France. The mobile QoS audit is based on more than a million measurements taken of 2G, 3G, 4G and 5G networks in every department of Metropolitan France, between mid-May and mid-August 2023, in living environments indoors and outdoors, and on various forms of transportation (roads, metros and trains).
There are significant disparities in quality levels depending on the location and the operator: Arcep invites everyone to compare the findings using the “Mon réseau mobile” (My mobile network) tool, each according to the type of area where they live (high-density, medium-density or rural) and the type of transport they use.
MEASURING MOBILE INTERNET QoS: KEY FINDINGS
- Arcep has expanded its indicators for downlink speeds: providing a more accurate reflection of users’ habits and experience
To more closely align with how users employ their mobile devices, Arcep wanted to rethink how mobile quality of service indicators are presented in its inquiry.
This new presentation creates the ability to better inform consumers about downlink speeds, tailored to their particular needs, for which three thresholds were established;
- 3 Mbit/s: speed suited to the least demanding mobile internet uses, such as web browsing;
- 8 Mbit/s: speed suited to the most widespread uses, such as video streaming;
- 30 Mbit/s: speed suited to the most demanding uses, such as collaborative tools for business purposes.
The new mobile QoS indicators reflect users’ experience and are broken down by type of area (high-density, medium-density or rural). This approach has the added benefit of not creating an incentive for operators to engage in a “speed race”, but rather to be part of the drive to achieve digital sustainability, initiated by Arcep.
For the “3 Mbit/s” and “8 Mbit/s” downlink speed thresholds, all types of living environment combined: Bouygues Telecom, Free Mobile and SFR are neck and neck behind Orange which scores the highest overall on these indicators.
In high-density areas, Orange and Bouygues Telecom stand out with equally strong performances
For all the other web browsing and video streaming indicators, all four operators provide a high quality of mobile internet service in densely populated areas.
For video streaming in 2G/3G/4G/5G in densely populated areas, Bouygues Telecom, Free Mobile, Orange and SFR all score a rate of 97% of videos streamed with perfect quality. In rural areas, Orange tops the rankings (89%), ahead of SFR (85%), Free Mobile (84%) and Bouygues Telecom (82%).
Regarding 2G/3G/4G web browsing[1] in high-density areas, Orange (97% of pages loaded in under 10 seconds) and Bouygues Telecom (96%) are followed by SFR (94%) and Free Mobile (93%). In rural areas, Orange (84%) is followed by Free Mobile (78%) and SFR (77%) then Bouygues Telecom (75%).
VOICE AND TEXTING QOS: KEY FINDINGS
QoS in 2023 higher than in 2022
Thequality of calls progressed nationwide in 2023
Regarding the indicator for perfect quality calls (the success rate for maintaining a two-minute call without audible interference):
- in high-density areas, Orange (96%) and Bouygues Telecom (95%) are very close, edging out SFR and Free Mobile (92%).
- in medium-density areas, Bouygues Telecom (92%), SFR and Free Mobile (91%) are now neck and neck, behind Orange (95%). Free gained six points in this type of area.
- in rural areas, the trend is slightly different, with 85% for Orange, 78% for SFR and Free Mobile, and 76% for Bouygues Telecom.
Every operator has made progress on the average call quality indicator, and all are delivering comparable performances: the gaps in MOS[2] were very small between SFR, Orange, Bouygues Telecom and Free Mobile (4.4 on average across all areas).
All operators enable customers to receive a text in under 10 seconds, all providing a very good level of connectivity. Free and Orange are tied for first (97%), followed by Bouygues Telecom and SFR (95%).
Arcep inquiry has incorporated new indicators for the quality of voice calls made with an instant messaging application[3]
In 2023, to more accurately reflect new user habits, Arcep has incorporated measurement of voice quality for calls made with an instant messaging app, in its series of measurements for the first time. Arcep wants to thereby provide a more complete assessment of the quality of voice calls on mobile networks.
For successful calls, all uses and all types of area combined, the four operators provide a high level of connectivity with a success rate of 97% for Orange, 95% for SFR, 94% for Bouygues Telecom and 93% for Free Mobile.
Performance gaps are widening for perfect quality calls (the success rate for maintaining a two-minute call without audible interference), with a 92% success rate for Orange, 86% for SFR, and 82% for Bouygues Telecom. Free lags behind on this indicator in every type of area and for all types of use, with a 54% success rate.
N.B.:
- The volume of measurements was adjusted proportionate to these uses whose adoption is still by a minority of users;
- Only one of the most popular platforms was tested;
- The quality of these voice calling services can vary depending on the version and users’ personal settings.
QOS ALONG TRANSPORT CORRIDORS: KEY FINDINGS
QoS still good on roads, but is middling on high-speed (TGV) trains, Intercités and TER commuter network lines
Internet quality of service
On average, all operators combined, internet quality of service is very high, with a close to 92% success rate for web page loads in under 10 seconds on the roads measured. The situation is more nuanced on railway lines: a web page can be loaded in under 10 seconds in only 70% of cases, on average, on TGV high-speed trains, Intercités and TER (regional express) lines. Browsing is more fluid on RER and Transiliens commuter trains in the Paris region (85%) and in the metro (96%).
Results on web browsing quality tests (web page loaded in under 10 seconds) on roads, SFR (92%), Bouygues Telecom and Free Mobile (91%) are all close behind Orange (95%).
Orange delivered the strongest performance on long distance railway lines, with a 79% success rate for web page loads in under 10 seconds on the TGV high-speed rail lines, ahead of Free Mobile (73%), SFR (65%) and Bouygues Telecom (64%). On Intercités and TER regional express networks, Orange had a success rate of 77%, followed by Free (70%) then Bouygues Telecom and SFR (66%).
On commuter lines in the Paris region (RER and Transiliens), Bouygues Telecom (84%), SFR and Free Mobile (83%) are behind Orange which scored highest with a 90% success rate for web browsing. On metro lines, three operators are providing a good quality of service, with Orange (98%), SFR and Bouygues Telecom (97%) leading Free Mobile (94%).
Calling quality (calls maintained for two minutes)
Regarding calling quality on the road, Orange (with 95% success rate) is ahead of SFR (92%) followed by Bouygues Telecom (90%), with Free Mobile lagging behind with 85% success rate.
On trains, Orange also delivers the strongest performance on TGV (high-speed trains), trains in the Intercités network and TER (regional express) trains, with 78% of calls maintained on TGV lines, compared to 68% for Bouygues Telecom, 65% for SFR and Free Mobile.
On the RER and Transiliens commuter rail networks, Orange scored 86% and Bouygues Telecom 84%, ahead of SFR (80%) and Free Mobile (78%).
On city metros, Bouygues Telecom, Orange (98%) and SFR (97%) have the highest success rate of calls maintained for 2 minutes, followed by Free Mobile with 95%.
All the findings are available as open data
Arcep makes all the measurements obtained during this campaign available as open datasets on both its own website and on data.gouv.fr.
Links:
- Map-based visualisation tool: monreseaumobile.arcep.fr
- Open datasets: https://www.data.gouv.fr/fr/datasets/monreseaumobile
[1] The web pages tested vary from year to year, which can result in a more or less lengthy load time due to the presence of high definition images, complex scripts, plugins, widgets, and other interactive elements.
[2] “MOS” (mean opinion score) for call maintenance which measures the difference between the call being tested and the benchmark sample
[3] Over the top (OTT) application
Annex 1: Methodology and scope of the mobile QoS enquiry
Mobile quality of service audit based on a more than a million measurements taken in 2G, 3G, 4G and 5G
More than one million measurements were taken of 2G, 3G, 4G and 5G networks in every department across Metropolitan France, between mid-May and mid-August 2023, in living environments indoors and outdoors, and on various forms of transportation. The audit covered the most widely used mobile services: web browsing, video streaming, data transfer, texting, and voice calls. The tests that were performed sought to evaluate the performance of operators’ networks in a strictly comparable fashion, and this in an array of circumstances.
Annex 2: Summary of 2G/3G/4G/5G findings
Key finding: in the 15 biggest cities in France, 98% of Orange calls were maintained for two minutes, and 93% with no audible disturbance
Calls: the success rate for calls maintained for two minutes and of calls maintained for two minutes without audible disturbance (dark colours). Calls are made between two SIM cards belonging to the same operator (on-net calls).
Texting: success rate of SMS received in under 10 seconds
Voice quality: Calls’ mean opinion score (MOS) – an automated assessment of voice quality, using the POLQA algorithm
Web: success rate of web pages loaded in under 10 seconds (light colours) and in under 5 seconds (dark colours)
Streaming: success rate of videos streamed with a decent viewing quality (light colours) and with perfect viewing quality (dark colours)
Uplink speed: average upstream speed, in Mbit/s
Downlink speed: average downstream speed, in Mbit/s
Speeds of ≥ 3 Mbit/s: percentage of speed tests that display an average connection speed equal to or above 3 Mbit/s
Speeds of ≥ 8 Mbit/s: percentage of speed tests that display an average connection speed equal to or above 8 Mbit/s
Speeds of ≥ 30 Mbit/s: percentage of speed tests that display an average connection speed equal to or above 30 Mbit/s