Data-Driven Regulation

To be able to measure the quality of their fixed and mobile internet connections, users need to have access to crowdsourcing testing and measuring tools. It is equally vital to have a clear understanding of the kind of tests these tools perform and of their limitations, but also of how their findings are presented, so that users can conduct these tests under the best possible conditions, and properly interpret the results.

In 2018, Arcep drafted a Code of conduct for measuring internet quality of service, whose updated version is being published today

It is with these imperatives in mind, and in keeping with its data-driven approach to regulation that Arcep published the inaugural version of its Code of conduct on internet quality of service [1] in late 2018, whose main purpose was to promote best practices for measuring quality of service, and to encourage testing tools to:

• accentuate transparency on their methodological choices, so that any third party can analyse their results;
• do away with any questionable practices, in terms of both test protocols and published findings.

In 2019, the following entities declared themselves in compliance with the 2018 version of Arcep’s Code of conduct:

• For fixed line access: nPerf (developed by nPerf), Speedtest UFC-Que Choisir (developed by UFC-Que Choisir), DébiTest 60: the internet speed test from 60 millions de consommateurs (developed by QoSi), 5GMark (developed by QoSi) and IPv6-test (developed by IPv6-test) ;
• For mobile access: nPerf (developed by nPerf), DébiTest 60: the internet speed test from 60 millions de consommateurs (developed by QoSi) and 5GMark (developed by QoSi).

The Code of conduct on internet quality of service was designed to evolve over time and be updated on a regular basis, not only to strengthen existing criteria, but also to supplement them with elements pertaining to other types of issue.

As announced in its annual report on the State of the internet in France, in June 2020, today Arcep is publishing the new version of this Code of conduct.

The 2020 version of the Code of conduct on internet quality of service increases the transparency and robustness requirements for testing protocols and for publications of the results.

The co-construction approach taken to drafting the 2018 Code of conduct continued to be used to produce this new version. To this end, Arcep hosted a series of bilateral and multilateral meetings with some twenty stakeholders, including the publishers of crowdsourcing testing and measurement tools, consumer protection organisations, operators and members of academia. The 2020 Code of conduct is the fruit of this work.

Several areas will be strengthened in the new version of the Code of conduct, to provide the QoS measurement ecosystem with ongoing support to continue to develop their knowledge and abilities. In particular, QoS testing and measuring tools are being required to:

• provide users with information on the different factors that might affect the measurement, such as the use of and properties of Wi-Fi, and the model and version of their operating system and web browser, all of which can have a considerable influence on quality of service measurement;
• display a median value for certain parameters, notably latency. This information is more relevant than averages in reflecting the user experience, particularly in cases where the measured results contain extreme values;
• introduce a minimum capacity for test servers, to ensure that the servers will not hamper testing;
• specify the capacity for test servers conducting tests in IPv6, as the protocol used can impact the outcome of speed tests.

This Code of conduct also underscores a number of potential sources of bias that must be made clear in measurement and testing tools’ aggregate publications. Lastly, it takes greater account of the specific considerations when measuring internet quality of service on mobile networks.

Arcep invites any testing or measuring tool wanting to declare their compliance with the new Code of conduct to do so by committing publicly to meeting the Code’s requirements. Arcep will then publish a list of testing or measuring tools who have declared themselves compliant.

The Code of conduct will continue to evolve with the implementation of the “Access ID card” API

In October 2019, Arcep published a Decision on implementing an API [2] designed by operators and accessible to any Code-compliant testing or measuring tool, whose purpose is to provide valuable details on a fixed network user’s testing environment. In July 2022, the “Access ID card” API is due to be implemented and activated in virtually all of the internet boxes covered by the Arcep decision, after several beta testing and implementation stages.

The co-construction with stakeholders will also continue, to further improve practices and strengthen the Code of conduct once the “Access ID card” API is in place. Testing or measuring tools’ ability to take into account the features offered by this API will result in increasingly accurate QoS tests and aggregate publications. All of these developments will naturally be implemented in concert with the players involved.