Market surveillance and AI: Who will monitor whom in the future?

The Federal Ministry for Economic Affairs and Climate Protection and the Federal Ministry of Justice recently published the draft bill for a law to implement the European Regulation on the implementation of the Regulation on the protection of personal data. Although everyone is already talking about the AI Regulation and the first regulations are to be complied with from February of this year, various regulations still require implementation at national level. For example, each member state must appoint a notifying authority and a market surveillance authority. 

Like the regulation, the draft bill aims to create a standardised internal market for AI-based goods and services. Special attention is paid to promoting innovation, while at the same time ensuring a high level of protection in terms of health, safety and the fundamental rights enshrined in the Charter, including democracy, the rule of law and environmental protection.

Regulatory content of the planned implementation law

As part of the AI Implementation Act, the Federal Network Agency is to be established as the central market surveillance authority and contact point in accordance with Article 70 (para. 2 sentence 3) of the AI Regulation (KI-VO). The authority would then also function as a central complaints office. It would also operate the sandboxes referred to in Articles 57 and 58 of the AI Regulation and set up a coordination centre to support other competent authorities in their tasks. Furthermore, an independent body will be set up at the authority, which will also be responsible for market surveillance in the cases referred to in Article 74(8) of the AI Regulation. The Chamber will play a key role in monitoring the implementation of tests for high-risk AIs. The aim here is to pool expertise in relation to artificial intelligence (AI) and make it available to the existing authorities in a resource-efficient manner. 

Various implementation models were examined with regard to ensuring the independence of market surveillance. The decision in favour of the current solution, which is based on the principle of the existing Board of Appeal, should be the most appropriate solution based on a consideration of the legal requirements and practical feasibility.

Section 7 of the draft stipulates that those authorities that have been designated by federal or state law as market surveillance authorities for the implementation of the harmonisation legislation listed in Annex I Section A of the Regulation (e.g. Directive 2009/48/EC, Directive 2013/53/EU or Directive 2014/53/EU) are to be transferred to the competent authorities under national law or, for example, the Federal Office of Consumer Protection and Food Safety or the Federal Ministry of Food Safety. Directive 2009/48/EC, Directive 2013/53/EU or Directive 2014/53/EU, which apply to the competent authorities under national law or (such as the Federal Office of Consumer Protection and Food Safety or the Federal Ministry of Defence) should also perform the tasks as competent market surveillance authorities if AI systems are associated with products to which the provisions listed in Annex I Section A apply. This is intended to ensure a seamless integration of the existing regulatory structures into the implementation of the AI Regulation. 

The sanctioning provisions of the AI Regulation (Article 99 (3) to (5) AI Regulation) will be transposed into national law in that the provisions of the Act on Regulatory Offences (with the exception of Section 17 OWiG) will apply accordingly.

Open points and criticism

However, the draft bill raises various questions and concerns. At the centre of these are the selection of the authority and the practical implementation of the proposed responsibilities, which entail both legal problems and challenges related to the federal structure in Germany. 

1) Federal Network Agency as the central supervisory authority 

A key point of the draft is the decision to appoint the Federal Network Agency (BNetzA) as the central supervisory authority for the implementation of the AI Regulation. This choice is not uncontroversial and has led to discussions regarding practical feasibility and the legal framework. In the past, the Federal Data Protection Authority (the Federal Commissioner for Data Protection and Freedom of Information [BfDI]) has offered itself as an alternative for this task.

It is argued against the BfDI that its expertise is primarily focussed on data protection and not on the technical market surveillance of AI. Arguments in favour of the BNetzA, on the other hand, include technical expertise and the standardised interpretation of regulations. The BNetzA already has extensive expertise in the area of product safety, particularly with regard to AI models and how they work. In addition, the transfer of competences to a single authority would lead to a uniform interpretation and application of the requirements of the AI Regulation. Another aspect cited was that this would avoid sectoral fragmentation and ensure a practical, low-bureaucracy and innovation-friendly implementation of the AI Regulation. In particular, the fact that there was a lack of sufficient AI specialists argued against splitting the process between different authorities.

2) Compatibility with Article 74(8) of the AI Regulation 

However, there is uncertainty as to whether the requirements of Article 74(8) of the AI Regulation have been adequately observed. This stipulates that, with regard to the market surveillance of high-risk AI systems used in security-relevant areas such as law enforcement, border management, justice and democracy, the competent authority must implement the conditions of the JHA Data Protection Directive (EU) 2016/680, in particular Articles 41 to 44. This directive stipulates that the processing of personal data by the authorities must be lawful, purposeful and secure. It explicitly stipulates that independent supervisory authorities must be appointed to monitor the application of the directive.

According to the explanatory memorandum to the AI Implementation Act, the provision in Section 3 (5), according to which the Market Surveillance Chamber acts completely independently and is not subject to any direct or indirect influences or instructions, would guarantee the independence of the authority in accordance with the requirements of Article 41 (2) of the JHA Data Protection Directive. In this respect, the implementation of the requirements of Articles 41 - 44 of the JHA Data Protection Directive is guaranteed by Section 3 (3) and (4). The draft bill therefore takes the JHA Data Protection Directive into account, but consistent application and review of the high requirements would still be necessary in order to ensure the appropriate standard of data protection when monitoring high-risk AI.

3) Conflict with the federal structure 

Another challenge that the planned regulations will have to face concerns the federalist distribution of competences and the constitutionally guaranteed autonomy of the federal states in Germany. As the central market surveillance authority, the Federal Network Agency would have far-reaching powers of control over state authorities. In practice, this is usually implemented through a federally structured organisation of authorities that preserves the responsibilities at federal and state level, such as in data protection.

The high level of complexity and intricacy of AI regulation may stand in the way of an implementation similar to data protection. However, the planned establishment of joint AI supervision would require an international treaty between the government and the states in order to create a sufficient legal basis for breaking through the federal system. The federal and state governments can regulate and distribute competences and the coordination of task fulfilment by means of an international treaty. However, this would require a considerable amount of justification and administration, so it is questionable whether this would really be a sensible solution that could be implemented in a timely manner.

What happens next?

The implementation of the AI Implementation Act is urgent: The EU member states are obliged to complete the implementing legislation by 2 August 2025. In view of the upcoming change of government, it remains to be seen how the new government will deal with the draft and when it will come into force. It therefore remains exciting!