ARCEP publishes a beta version of its new QoS scorecard for fixed internet access

Paris, 25 November 2014

Background

Today, ARCEP is publishing the beta version of its new quality of service (QoS) scorecard for fixed internet access provided in Metropolitan France. It comes to complete the observatories that have already been published on a regular basis: on the quality of fixed network access and wireline calling, on the one hand, and on mobile service quality and coverage on the other.

ARCEP is releasing the results of measurements that were taken as part of the system introduced by its Decision of 29 January 2013, whose aim is to improve the information available to internet users, and to provide the Authority with the means to satisfy the responsibilities assigned to it by Law, i.e. to supervise the overall quality of internet access services. This new observatory is part of the broader work and investigation that ARCEP has been carrying out since 2010 on the technical and economic aspects of net neutrality.

The system that is being introduced will eventually contain two parts: measurements that are taken in a controlled environment on dedicated test lines (the "main" tool) and measurements taken by users themselves ("supplementary" tool). At this stage, only the main tool is in place.

For the sake of transparency, ARCEP wanted to provide the public with a first summary of the information that was gathered during this initial series of tests, in a report and in a computer-processable format, namely an attached spreadsheet. Given the limited hindsight, however, and the inherent risks of error when any such system is launched, the Authority nevertheless asks readers to apply caution when interpreting the data contained in this first summary report. It also recommends that any dissemination of this data be accompanied by the disclaimers on the methodology mentioned in the report.

Additional information is provided on the background to and framework for the implementation of this new observatory, in the section of the report entitled, "General introduction to the system".

Once the system is fully up and running, two series of tests will be carried out each year: the results will be published in April (for second half of the previous year) and in October (for the first half of the current year). This first publication must be considered a beta version, which is an integral part of the process of fine-tuning the system's reliability.

The process of breaking in the system in fact shorted the test period, exceptionally, to one month (26 May to 30 June 2014) instead of six. Moreover, several malfunctions were found during this first series of tests, and needed to be corrected. Lastly, several measurement points had to be excluded from the scope of calculation for the published findings.

For all of these reasons, ARCEP decided not to publish in detail the technical indicators (1) that were particularly affected by the malfunctions during the system's running-in stage. Only the aggregated values for all operators combined are being published for these indicators.

The measurement tool

The measurement tool is based on dedicated test lines, inside a technically-controlled environment whose conditions make it possible to achieve a high degree of comparison between operators, and a broad representation of the various network access conditions that users encounter. It concerns the five main internet service providers (ISP) in France: Bouygues Telecom, Free, Numericable, Orange and SFR.

The design and creation of this tool were the fruit of close to three years of preparatory work, which was subject to very close consultation with stakeholders: ARCEP involved all of the main stakeholders in the decision-making process from the outset, and held several public consultations and a large number of multilateral meetings on the subject.

A technical committee, managed by ARCEP and composed of the affected ISPs, consumer associations and independent technical experts, drafted the technical specifications for the measurement protocol. These specifications include reference points that are common to all of the ISPs. The committee continues to meet on a regular basis to improve and enhance the measurement protocol.

ARCEP believes that, over time, regular use of the measurement protocol will make it possible to fine-tune the system. The results being published today, including the individual data, must thus be interpreted cautiously, as this is the first time the system has been used.

The measurements concern the three main fixed internet access configurations (or technologies) that were being used by consumers in France when the process was launched: ADSL (2 and3), hybrid fibre-coax (HFC) and FttH (4). Several categories of access have been taken into account for the same configuration, to reflect the wide range of performances that can be observed, according to the configurations and the access products on offer.

In the above-mentioned decision of January 2013, ARCEP elected to apply four technical indicators and three indicators relating to specific types of usage, or seven performance indicators in all.

For this first publication, only the usage indicators are published in detail. The methodology used to measure the technical indicators could not be stabilised during the first round of tests, and the ensuing results do not yet appear to be reliable enough to deliver relevant information on each individual ISP. The technical committee continues to work on these indicators, with a view to having them fully integrated into the observatory for the first half of 2015 (publication of the results of tests carried out in the second half of 2014). Aggregated results for all ISPs combined are nevertheless being provided as a yardstick (5). They make it possible to illustrate the performance gaps between the different types of access.

Measurements were taken on three to six lines for each type of access, spread out over eight sites in Metropolitan France. The measurement protocol was defined by taking into account the geographical representativeness of the tested lines, the technical environment and the access products available to consumers. In particular, the products that were tested were the most popular triple play bundles - or, at the very least, which represent a significant percentage of sales - during the six months prior to the testing period.

The measurements were carried out by an independent company which was selected through a tender process.

Main findings

The technical committee has worked to achieve clear and easily understandable results. For every one of the indicators, each of the following graphs represents average performance levels for each type of access, for all of the lines tested.

TECHNICAL INDICATORS

Download speed - heavy traffic periods, nearby facilities

Observations (6): on the whole, the measured values reflect ISPs' advertised speeds for the access products tested.