28 November 2019: Arcep publishes data on calling quality along transport corridors.
Five of the total 243 indicators were not included in the published figures for the 2019 measurement campaign (press release of 22 October 2019) as their accuracy needed to be improved. These figures are now available and published by Arcep as open data on monreseaumobile.fr. These are data pertaining to success rates for making a two-minute call on transport corridors: roads, TGV high-speed rail lines, Intercités/TER (regional rail) lines, metro lines, RER (commuter train) and Transiliens (Paris suburban rail) lines.
Arcep is publishing the findings of its 20th annual audit to assess the quality of the services provided by mobile operators in Metropolitan France. More than 1.5 million tests were performed on 2G, 3G and 4G networks across the country, in every department, indoors and outdoors, and on various forms of transportation. The audit covered the most widely used mobile services: web browsing, playing videos, data transfer, texting and voice calls. The tests performed on 243 indicators sought to evaluate the performance of operators’ networks in a strictly comparable fashion, and this in an array of circumstances.
There are still significant disparities in quality levels between the operators: Arcep invites everyone to compare the findings, each according to the type of area where they live (high-density, medium-density or rural) and the type of transport they use.
MOBILE INTERNET (“Data measurements”): Strong increase in all operators’ service quality
On all four operators’ mobile networks, the quality of data services is improving, and this in every type of area: rural, medium-density and high-density. The average downlink speed measured in Metropolitan France stands at 45 Mbit/s, compared to 30 Mbit/s in 2018.
Orange scored highest on all mobile internet measurement campaigns. Bouygues Telecom continued to score higher than SFR. Lastly, Free Mobile still trails behind but has made considerable progress, and narrowed the gap with the other three operators.
In rural areas, Bouygues Telecom scored the highest across the board, just ahead of Orange and SFR. Free Mobile has made significant progress in terms of connection speed, but the quality of the (video streaming and web browsing) experience the operator provides in rural areas remains far below the other operators’.
VOICE CALLS AND TEXTING: Orange, SFR and Bouygues Telecom provide comparable levels of quality, ahead of Free
Regarding the quality of calls on a national level: Orange provides the highest quality of service: 80% of 2-minute calls are of perfect quality, just ahead of Bouygues Telecom (79%) and SFR (78%). Free Mobile is in last place (70%), penalised by having no high definition calls in 4G.
Lastly, regarding texting: SFR, Orange and Bouygues all provide an excellent quality of service: 95% of SMS relayed by these operators are received in under 10 seconds, compared to 92% for Free Mobile.
Note that Arcep improved its methodology this year to more accurately reflect users’ actual experience. This “change in thermometer” resulted in lower absolute values than the previous methodology. This does not, however, reflect any decrease in actual quality of service (see Annex 3 for details).
TRANSPORT CORRIDORS: Orange is the best operator on roads and railway lines, followed by SFR and Bouygues Telecom
Data service coverage on roadways is improving, with a 94% success rate for web page loads in under 10 seconds for Orange and Bouygues Telecom, and 91% for SFR – which marks an increase of 3% to 7% compared to 2018, depending on the operator.
Coverage of railway lines has also improved substantially, notably for Orange RER (commuter train) and Transiliens (Paris suburban rail) lines, for Orange and Bouygues Telecom on Intercités/TER (regional rail) lines, and for both of these operators and SFR on TGV high-speed rail lines.
Free Mobile lags behind considerably on all of these indicators, despite making real strides: a 9 to 18-point improvement on the different transport routes.
Lastly, all four operators have massively improved their coverage inside the Paris metro. The success rate for downloading a web page stands at 78% in 2019, versus 42% in 2018 and 18% in 2017.
New: measurements taken in the 50 most popular tourist spots in France
A new tourist-related dimension was added to the list of living environments audited, which already included “high density, “medium density” and “rural” areas. The tourist category comprises the 50 most popular tourist sites in France (museums, chateaux, theme parks, religious monuments…) [1]. Its purpose is to evaluate the evenness of the quality of service provided when faced with the saturation caused by an influx of large crowds into a small location.
Viewing the findings in map form on Arcep’s “Mon réseau mobile” site
The “Quality of service” tab on monreseaumobile.fr has been updated with the results being published today. The website now provides a map-based representation of the findings, in addition to open data, by living environment. It allows users to view the results based on their own circumstances: living in a rural area, a high-density area, indoors, when travelling, etc.
[1] This list is published by the General Directorate for Enterprise (DGE) https://www.entreprises.gouv.fr/files/files/directions_services/etudes-et-statistiques/stats-tourisme/memento/2017/2017-11-MEMENTO-TOURISME-CHAP7-les-sites-touristiques-en-france.pdf
Arcep’s interactive map tool allows users to compare mobile operators’ performance, by providing two types of complementary information: “simulated coverage” and “measured quality”. All of the data used to feed the “Mon réseau mobile” (My mobile network) site are available as open data sets.
All of the measurements are available as open data
Arcep makes all of the measurements obtained during this campaign available as open datasets on both its own website and on data.gouv.fr. Of the 243 indicators measured, the five that pertain to voice calls on modes of transport will be the subject of a future publication, to further improve their accuracy.
- Annex 1 : Scope of Arcep’s 2019 audit of mobile service quality
- Annex 2: Summary of the 2019 findings
- Annex 3: Arcep tightens its methodology to more accurately reflect the user experience. Mobile operators continue to provide a high quality of service
For the 20th edition of this measurement campaign, Arcep wanted to increase the accuracy of the method used to assess the quality of mobile calling services in the different environments. Three new features were therefore added:
- In previous campaigns, the assessment made was of operators’ mobile lines calling fixed lines. This year all of the tests were of calls made between two mobile phones, which more closely reflects French users’ habits.
- In addition, to be as representative as possible, 25% of the calls were assessed between two mobiles attached to the same operator (on-net) and 75% between mobiles attached to two different operators (off-net).
- To assess the quality of the calls, the method used up to now had been the “golden ear” method: auditors were tasked with assigning a score. This year, the automatic classification of call QoS was employed: it consists of measuring the difference between the test call and a benchmark sample [2], for a short 8-second voice sample. A software programme then assigns a score that corresponds to the signal’s distortion. A two-minute call contains a dozen samples, and therefore receives as many scores.
Scoring an entire call: lowest or average score method
There are two possible approaches to establishing an overall QoS score for the entire call, based on the series of scores obtained: retain the average of all of the scores, or the lowest of all the scores. To remain consistent with human assessments, which tend to point to any disturbance on a call rather than the overall quality, Arcep proposes relying primarily on the lowest score to establish its findings. For the sake of transparency, Arcep does, however, also provide the results obtained using the average score method. The results obtained using the two methods are comparable, and do not affect operator rankings.
The introduction of these three new features make the QoS testing for mobile calls much stricter than in Arcep’s previous campaigns. This “change in thermometer” results in lower success rates for test calls, but this in no way means that the quality of the networks has decreased from last year to this. A pilot was conducted during the 2018 campaign to test mobile-to-mobile calls maintained for two minutes. Applied to all calls during the 2019 campaign, this test makes it possible to ensure that the quality being provided is similar to that of previous years.
[2] Sample including calls by females and males, in compliance with ITU P.863 standards on the objective listening quality assessment