Today Arcep is publishing the results of its 2024 annual audit assessing the quality of the services provided by mobile operators in Metropolitan France. The mobile QoS audit is based on more than a million measurements taken in every department of Metropolitan France, between May and July 2024, in living environments, indoors and outdoors, and on various forms of transportation (roads, metros and trains).
The results of this audit reveal a high quality of service for calling/SMS and the mobile internet: they confirm the overall trend of steadily increasing speeds in every high-density, medium-density and rural area, even if there continue to be gaps between them.
SUPPLEMENTARY ANALYSES AND POSTPONED PUBLICATION TO ENSURE THAT THE PUBLISHED DATA ARE REPRESENTATIVE
Arcep attaches great importance to the quality and reliability of the data that it provides to users, notably through its annual publication on the quality of mobile services.
Ordinarily published late in the year, this audit was postponed from late 2024. It emerged that supplementary analyses were required for the 2024 QoS audit to ensure that the results of the tests were representative of the user experience. It was within this context that the Arcep Body responsible for settling disputes, legal proceedings and investigations (RDPI) launched an administrative inquiry, pursuant to Article L. 32-4 of the French Postal and Electronic Communications Code (CPCE).
In the course of this inquiry, which closed with a Decision that is being published today[1], evidence was found that the activation by Orange of a specific standby/sleep setting for the devices used during the 2023 and 2024 audits could affect the representativeness of the tests carried out on voice calls made using instant messaging[2] (OTT calling) applications, and for this operator alone.
In light of the conclusions of this inquiry, the Arcep Executive Board’s Plenary body decided to remove the OTT calling[3] indicators for operator Orange[4] from the publication of its 2024 audit, and to publish the other indicators for all operators.
Arcep also implemented increased oversight when preparing for the 2025 audit: operators were invited to provide the Authority in advance with information on their network’s configuration. This information gives the Authority the ability to better understand the new features offered on 4G and 5G networks for managing quality of service and, ultimately, to ensure that QoS audits will produce an objective and transparent assessment of networks’ performance, and one that is representative of the user experience.
“VOICE AND TEXTING” QoS: KEY FINDINGS
An overall satisfactory level of satisfaction in the country as a whole, even if calling quality is significantly lower in rural areas for all operators
Regarding the indicator for perfect quality calls (the success rate for maintaining a two-minute call without audible interference):
- At the national level, 90% of calls are considered to be of perfect quality for Orange, ahead of Bouygues Telecom (87%), SFR (85%), and Free Mobile (84%).
- In high-density areas, Bouygues Telecom (95%) and Orange (94%)[5] are ahead of SFR (92%) and Free Mobile (90%).
- In medium-density areas, Bouygues Telecom (91%) and SFR (90%) are tied behind Orange (93%), with Free Mobile scoring 88% for perfect quality calls.
- In rural areas, 82% of calls are considered to be of perfect quality for Orange, followed by SFR and Bouygues Telecom (75%) and Free Mobile (73%).
Regarding the success rate for calls (maintaining a two-minute call without audible interference): Orange leads at the national level with 96%, ahead of Bouygues Telecom (94%), SFR and Free Mobile (92%).
The average quality indicator is high for every operator, and their performance levels are very close: there is a very small gap between the MOS[6] for Orange, SFR, Bouygues Telecom and Free Mobile (4.44 on average for all areas).
Texts delivered rapidly with high-level performance for all four operators
The success rate for receiving an SMS in under 10 seconds with a high level of performance reveals very high and very close QoS for the four operators on this indicator. The results are as follows: Orange (96%), Bouygues Telecom (95%), SFR and Free Mobile (94%).
For the second consecutive year, Arcep’s mobile quality of service audit includes QoS indicators for voice calls made using an instant messaging[7] (OTT calling) application.
In light of the conclusions of the administrative inquiry conducted by Arcep’s RDPI Body, the Authority is only publishing the results for Bouygues Telecom, Free Mobile and SFR this year. For successfully completed calls, all applications and all residential locations combined, the three operators are tied, all with strong performances, scoring an overall success rate of 92%.
The quality of OTT voice calls (the success rate for maintaining a two-minute call without audible interference) varies depending on the operator: Bouygues Telecom and SFR stand out with a respective success rate of 88% and 87%, compared to 76% for Free Mobile.
MOBILE INTERNET QoS: KEY FINDINGS
The download speed indicator that Arcep uses includes three thresholds – 3, 8 and 30 Mbit/s – that correspond to different requirement levels depending on the application:
- 3 Mbit/s: speed suited to the least demanding mobile internet uses, such as web browsing;
- 8 Mbit/s: speed suited to the most widespread uses, such as video streaming;
- 30 Mbit/s: speed suited to the most demanding uses, such as collaborative tools for business purposes.
The different mobile QoS indicators reflect users’ experience and are broken down by type of area(high-density, medium-density or rural). This approach has the added benefit of not creating an incentive for operators to engage in a “speed race”, but rather to be part of the drive to achieve digital sustainability, initiated by Arcep.
For the three downstream speed thresholds – all types of residential location combined – SFR, Bouygues Telecom and Free Mobile are close behind Orange
In rural areas where every operators’ connection speeds are below those found in other parts of the country, for every threshold in this indicator, Free Mobile and SFR are tied behind Orange and ahead of Bouygues Telecom. In high-density areas, Bouygues Telecom, Orange and SFR have similar results, and score higher than Free Mobile.
Video streaming and web browsing tests revealed very high and almost comparable quality for the four operators in very high-density areas.
For perfect quality video streaming over 2G/3G/4G/5G, in very high-density areas: Bouygues Telecom and Orange have a success rate of 98%, SFR of 97% followed by Free with 96%. In rural areas, Orange leads the way (92%), ahead Free Mobile (86%), SFR (85%) and Bouygues Telecom (81%).
Regarding web browsing[8], in very high-density areas, respectively 98% and 97% of web pages are loaded in under 10 seconds with Bouygues Telecom and Orange, ahead of SFR with 96%, and Free Mobile with 95%5. In rural areas, Orange (88%) is followed by Free Mobile (85%), SFR (81%) then Bouygues Telecom (80%).
QOS ALONG TRANSPORT CORRIDORS: KEY FINDINGS
Quality of service remains high on roads. But is still average on TGV high-speed rail lines and on Intercités network and TER regional line trains.
- Call quality (success rate for calls completed and maintained for two minutes):
For calls made while on roadways, Orange (with a 92% success rate), Bouygues Telecom (91%), SFR (90%) score higher than Free Mobile (87%).
Results for calling on trains are more varied, particularly on TGV high-speed trains where Orange leads the way with a 78% success rate for calls maintained for two minutes, followed by Free Mobile (60%), Bouygues Telecom (59%), and SFR (56%).
On Intercités and TER trains, Orange has a 67% success rate, followed by Bouygues Telecom (60%), SFR and Free Mobile (59%).
On RER and Transiliens commuter lines, Orange has an 86% success rate for calls completed and maintained for two minutes, followed by Bouygues Telecom 84%, SFR, and Free Mobile (80%).
Performance levels on metro lines are very good overall, with Bouygues Telecom reaching a 98% success rate, Orange 96%, SFR 95%, and Free Mobile 94%.
- Internet quality of service:
On average, all operators combined, internet quality of service is very high, with a close to 94% success rate for web pages loaded in under 10 seconds on the roads that were tested. The situation is far more nuanced on railway lines: a web page can be loaded in under 10 seconds in only 73% of cases, on average, on TGV high-speed trains, and 74% on Intercités and TER (regional express) lines. Browsing is more fluid on RER and Transiliens commuter trains in the Paris region (87%) and on metro lines (97%).
Results on web browsing quality tests (web page loaded in under 10 seconds) are as follows:
- On roads, SFR (94%), Bouygues Telecom and Free Mobile (93%) have comparable success rates, behind Orange with 96%.
- On long distance, TGV high-speed railway lines, Orange stands out again, with 82% of pages loaded in under 10 seconds, compared to 72% for Free Mobile, 69% for SFR, and 68% for Bouygues Telecom.
- On the Intercités and TER regional express networks, Orange has a success rate of 79%, followed by Free Mobile (74%), SFR (73%), and Bouygues Telecom (71%).
- On RER and Transiliens commuter lines, Orange performed the strongest with a 91% success rate, ahead of Bouygues Telecom (87%) and SFR and Free Mobile (86%).
- On metro lines, all four operators perform very strongly: Orange and SFR with a 98% success rate, Bouygues Telecom and Free Mobile with 97%.
All the findings are available as open data
Arcep makes all the measurements obtained during this audit available as open datasets on both its own website and on data.gouv.fr. These findings will also be available shortly on the “Mon réseau mobile” tool.
Annexes:
- Annex 1: Scope of enquiry in 2024
- Annex 2: Summary of 2024 findings
Links:
- Map-based visualisation tool: monreseaumobile.arcep.fr
- Open datasets: https://www.data.gouv.fr/fr/datasets/monreseaumobile
[2] Over the top – OTT: application providing a service (calling, streaming, messaging, etc.) over the internet, without going through an ISP’s classic services
[3] Or 8 calling indicators of the total 258 audit indicators published as open data
[4] Readers’ attention to this matter in the published results of the 2023 audit.
[5] These results factor in the statistical accuracy of the margin of error for each estimate. This means that, if the gap between operators remains within that margin, they are considered as being tied. In this instance, Bouygues Telecom and Orange are considered as being tied. It should be noted that this statistical accuracy can vary from one indicator to the next, depending on the volume of testing.
[6] The mean opinion score (MOS) for maintained calls, which measures the different between the live call and the baseline sample.
[7] Over the top (OTT) application: application that provides service (calling, streaming, messaging, etc.) over the internet, without using a telecom operator’s classic services.
[8] The web pages tested vary from year to year, which can result in a shorter or long load time due to the presence of high-definition images, complex scripts, plugins, widgets, and other interactive elements.
Annex 1: Methodology and scope of the mobile QoS audit
Mobile quality of service audit based on a more than a million measurements taken in 2G, 3G, 4G and 5G
More than one million measurements were taken of 2G, 3G, 4G and 5G networks in every department across Metropolitan France, between mid-May and mid-July 2024, in residential locations, indoors and outdoors, and on various forms of transportation. The audit covered the most widely used mobile services: web browsing, video streaming, data transfer, texting, and voice calls (notably OTT calls). The tests that were performed sought to evaluate the performance of operators’ networks in a strictly comparable fashion, and this under a wide range of conditions.
