ARCEP publishes a beta version of its new QoS scorecard for fixed internet access

Paris, 25 November 2014

Background

Today, ARCEP is publishing the beta version of its new quality of service (QoS) scorecard for fixed internet access provided in Metropolitan France. It comes to complete the observatories that have already been published on a regular basis: on the quality of fixed network access and wireline calling, on the one hand, and on mobile service quality and coverage on the other.

ARCEP is releasing the results of measurements that were taken as part of the system introduced by its Decision of 29 January 2013, whose aim is to improve the information available to internet users, and to provide the Authority with the means to satisfy the responsibilities assigned to it by Law, i.e. to supervise the overall quality of internet access services. This new observatory is part of the broader work and investigation that ARCEP has been carrying out since 2010 on the technical and economic aspects of net neutrality.

The system that is being introduced will eventually contain two parts: measurements that are taken in a controlled environment on dedicated test lines (the "main" tool) and measurements taken by users themselves ("supplementary" tool). At this stage, only the main tool is in place.

For the sake of transparency, ARCEP wanted to provide the public with a first summary of the information that was gathered during this initial series of tests, in a report and in a computer-processable format, namely an attached spreadsheet. Given the limited hindsight, however, and the inherent risks of error when any such system is launched, the Authority nevertheless asks readers to apply caution when interpreting the data contained in this first summary report. It also recommends that any dissemination of this data be accompanied by the disclaimers on the methodology mentioned in the report.

Additional information is provided on the background to and framework for the implementation of this new observatory, in the section of the report entitled, "General introduction to the system".

Once the system is fully up and running, two series of tests will be carried out each year: the results will be published in April (for second half of the previous year) and in October (for the first half of the current year). This first publication must be considered a beta version, which is an integral part of the process of fine-tuning the system's reliability.

The process of breaking in the system in fact shorted the test period, exceptionally, to one month (26 May to 30 June 2014) instead of six. Moreover, several malfunctions were found during this first series of tests, and needed to be corrected. Lastly, several measurement points had to be excluded from the scope of calculation for the published findings.

For all of these reasons, ARCEP decided not to publish in detail the technical indicators (1) that were particularly affected by the malfunctions during the system's running-in stage. Only the aggregated values for all operators combined are being published for these indicators.

The measurement tool

The measurement tool is based on dedicated test lines, inside a technically-controlled environment whose conditions make it possible to achieve a high degree of comparison between operators, and a broad representation of the various network access conditions that users encounter. It concerns the five main internet service providers (ISP) in France: Bouygues Telecom, Free, Numericable, Orange and SFR.

The design and creation of this tool were the fruit of close to three years of preparatory work, which was subject to very close consultation with stakeholders: ARCEP involved all of the main stakeholders in the decision-making process from the outset, and held several public consultations and a large number of multilateral meetings on the subject.

A technical committee, managed by ARCEP and composed of the affected ISPs, consumer associations and independent technical experts, drafted the technical specifications for the measurement protocol. These specifications include reference points that are common to all of the ISPs. The committee continues to meet on a regular basis to improve and enhance the measurement protocol.

ARCEP believes that, over time, regular use of the measurement protocol will make it possible to fine-tune the system. The results being published today, including the individual data, must thus be interpreted cautiously, as this is the first time the system has been used.

The measurements concern the three main fixed internet access configurations (or technologies) that were being used by consumers in France when the process was launched: ADSL (2 and3), hybrid fibre-coax (HFC) and FttH (4). Several categories of access have been taken into account for the same configuration, to reflect the wide range of performances that can be observed, according to the configurations and the access products on offer.

 

In the above-mentioned decision of January 2013, ARCEP elected to apply four technical indicators and three indicators relating to specific types of usage, or seven performance indicators in all.

For this first publication, only the usage indicators are published in detail. The methodology used to measure the technical indicators could not be stabilised during the first round of tests, and the ensuing results do not yet appear to be reliable enough to deliver relevant information on each individual ISP. The technical committee continues to work on these indicators, with a view to having them fully integrated into the observatory for the first half of 2015 (publication of the results of tests carried out in the second half of 2014). Aggregated results for all ISPs combined are nevertheless being provided as a yardstick (5). They make it possible to illustrate the performance gaps between the different types of access.

Measurements were taken on three to six lines for each type of access, spread out over eight sites in Metropolitan France. The measurement protocol was defined by taking into account the geographical representativeness of the tested lines, the technical environment and the access products available to consumers. In particular, the products that were tested were the most popular triple play bundles - or, at the very least, which represent a significant percentage of sales - during the six months prior to the testing period.

The measurements were carried out by an independent company which was selected through a tender process.

Main findings

The technical committee has worked to achieve clear and easily understandable results. For every one of the indicators, each of the following graphs represents average performance levels for each type of access, for all of the lines tested.

TECHNICAL INDICATORS

Download speed - heavy traffic periods, nearby facilities

Observations (6): on the whole, the measured values reflect ISPs' advertised speeds for the access products tested.

Déplacez le curseur pour consulter le contenu du tableau

Upload speed (7) - heavy traffic periods, nearby facilities

Observations (8) : with ADSL, upload speeds are much less affected by the length of the line than download speeds.
Upload speed measurements on lines over 5 Mbps have been affected by a malfunction of the system.
Déplacez le curseur pour consulter le contenu du tableau

Latency - heavy traffic periods, nearby facilities

Observations (9) : it appears that upgrades to hybrid fibre-coax lines that make it possible to deliver a throughput of more than 100 Mbps, have not improved latency (10).
Disparate performances in terms of latency contribute a great deal to the web browsing and video streaming results that were obtained (see table below).
Déplacez le curseur pour consulter le contenu du tableau

USAGE INDICATORS (11)

Web browsing - heavy traffic periods

Observations: if the time it takes to load a web page appears to be significantly shorter for an FttH connection than for the other types of access technology, this reduction is nevertheless not proportionate to throughput. Page load times also depend on other technical metrics, and particularly latency.
Déplacez le curseur pour consulter le contenu du tableau

Video streaming - heavy traffic periods, excluding long lines

 

 

Observations: because of the video format tested (720p, which corresponds to a stream of around 4 Mbps) the playback quality of the streamed video makes it impossible to establish a hierarchy of the different access technologies. Quality appears to depend largely on the interconnection capacities between the ISP and the video website, which can improve as the different interconnection links are upgraded. Higher quality formats will be tested during future measurement campaigns.

Déplacez le curseur pour consulter le contenu du tableau

P2P file download - heavy traffic periods

Observations: P2P file download performance is heavily affected by throughput. This type of application nonetheless remains largely dependant on factors outside ISPs' control, and particularly on available peers, their capacity and location, which explains why gains are not proportionate to the throughput supplied.

Déplacez le curseur pour consulter le contenu du tableau

What's next?

To provide more detailed information, ARCEP staff will be available for a live chat with members of the public, on 4 December 2014 at 5 pm.

ARCEP also continues to work with all of the stakeholders (consumer associations, ISPs and technical experts) on improving and fine tuning the measurement system, which is to result in regular, twice-yearly publications.

___________________________
(1) The list of usage and technical indicators is presented below.

(2) Asymmetric Digital Subscriber Line

(3) VDSL will be included in subsequent tests

(4) Fibre to the Home

(5) Measurements for the "packet loss" indicator were significantly diminished by a malfunction of the system during the first round of testing. ARCEP thus decided not to publish this indicator in this first report.

(6) The methodology and the scope of the QoS indicators for internet access services can vary significantly between the different systems.
For the purposes of comparison, however, the DegroupTest barometer of fixed internet connections for the first half of 2014 (http://www.degrouptest.com/publications/16/barometre-connexions-internet-fixes-s1-2014/1) measured average download speeds of between 5.6 Mbps and 7.2 Mbps for xDSL; between 18 Mbps and 19.3 Mbps for hybrid fibre-coax lines with an advertised speed of 30 Mbps; between 46.3 Mbps and 47.2 Mbps for hybrid fibre-coax lines with an advertised speed of 100 Mbps and up, and between 61.9 Mbps and 77.8 Mbps for FttH/FttB products.
For the period running from 1 May 2013 to 13 November 2014, the Ariase observatory (http://www.ariase.com/fr/vitesse/observatoire-debits.html) measured download speeds of between 6.5 Mbps and 8 Mbps for ADSL, and between 50.1 Mbps and 125.1 Mbps for superfast access lines.
The FAI 01Net (http://www.01net.com/rub/actualites/10506/actualites/fai/banc-dessai-fai/) ISP benchmark, whose latest tests ran from 31 March to 6 April 2014, revealed the following results for lines in the Paris region: download speeds of between 12 Mbps and 14.9 Mbps for ADSL and 25.3 Mbps for hybrid fibre-coax lines with an advertised speed of 30 Mbps.

(7) Measurements of upload speeds on lines with an advertised upload speed of more than 5 Mbps were affected by a configuration flaw in the measuring system, and have thus not been published.

(8) The methodology and the scope of the QoS indicators for internet access services can vary significantly between the different systems.
For the purposes of comparison, however, the DegroupTest barometer of fixed internet connections for the first half of 2014 (http://www.degrouptest.com/publications/16/barometre-connexions-internet-fixes-s1-2014/1) reveal average latency of between 68 ms and 103 ms for xDSL; between 42 ms and 74 ms for hybrid fibre-coax lines with an advertised speed of 30 Mbps; between 40 ms and 41 ms for hybrid fibre-coax lines with an advertised speed of 100 Mbps and up, and between 24 ms and 39 ms for FttH/FttB products.
For the period running from 1 May 2013 to 13 November 2014, the Ariase observatory (http://www.ariase.com/fr/vitesse/observatoire-debits.html) measured latency of between 51 ms and 65 ms for ADSL, and between 22 ms and 35 ms for superfast access lines.
The FAI 01Net (http://www.01net.com/rub/actualites/10506/actualites/fai/banc-dessai-fai/) ISP benchmark, whose latest tests ran from 31 March to 6 April 2014, revealed the following results for lines in the Paris region: latency of between 9 ms and 22 ms for ADSL, and of 12 ms for hybrid fibre-coax lines with an advertised speed of 30 Mbps.
Access products with an advertised speed of 30 Mbps and 100 Mbps are not based on the same standard: the first employ DOCSIS 2.0 and the second employ DOCSIS 3.0.

(9) The methodology and the scope of the QoS indicators for internet access services can vary significantly between the different systems.
For the purposes of comparison, however, the DegroupTest barometer of fixed internet connections for the first half of 2014 (http://www.degrouptest.com/publications/16/barometre-connexions-internet-fixes-s1-2014/1) reveal average latency of between 68 ms and 103 ms for xDSL; between 42 ms and 74 ms for hybrid fibre-coax lines with an advertised speed of 30 Mbps; between 40 ms and 41 ms for hybrid fibre-coax lines with an advertised speed of 100 Mbps and up, and between 24 ms and 39 ms for FttH/FttB products.
For the period running from 1 May 2013 to 13 November 2014, the Ariase observatory (http://www.ariase.com/fr/vitesse/observatoire-debits.html) measured latency of between 51 ms and 65 ms for ADSL, and between 22 ms and 35 ms for superfast access lines.
The FAI 01Net (http://www.01net.com/rub/actualites/10506/actualites/fai/banc-dessai-fai/) ISP benchmark, whose latest tests ran from 31 March to 6 April 2014, revealed the following results for lines in the Paris region: latency of between 9 ms and 22 ms for ADSL, and of 12 ms for hybrid fibre-coax lines with an advertised speed of 30 Mbps.

(10) Access products with an advertised speed of 30 Mbps and 100 Mbps are not based on the same standard: the first employ DOCSIS 2.0 and the second employ DOCSIS 3.0.

(11) The methodology and the scope of the QoS indicators for internet access services can vary significantly between the different systems. Usage indicators are all the more sensitive to these differences in methodology as there is no standardised way to measure them. There does not appear to be an observatory that compares ISPs in France on the same scale and using the same methodology.


Linked documents

The complete report (pdf - 2.08 Mo) (in French)

Aggregated data (zip - 92 Ko) (in French - zip)