As artificial intelligence (AI) promises economic returns on a scale comparable to the Industrial Revolution, growing concerns surrounding its impact on privacy and data protection are intensifying. The core challenges—lack of transparency, absence of legitimization, and unrestrained data volumes—directly challenge the fundamental principles of the General Data Protection Regulation (GDPR). This comprehensive article explores how the development and deployment of AI systems are regulated by the GDPR, detailing the critical phases, the roles of data controllers and processors, and the necessary legal and ethical compliance frameworks established by European authorities, including the forthcoming AI Act.
The Dual Challenge of AI: Innovation vs. Privacy
Artificial intelligence, fueled by decades of research and the booming output from mobile phone use and the Internet of Things (IoT), has moved from theoretical minimum viable products (MVPs) into practical, widespread usability. However, this technological leap carries inherent risks and ethical dilemmas. One of the most critical risks is the impact of AI on personal privacy, particularly concerning the vast datasets used for development and the results generated by those systems. Key issues include the lack of transparency in how algorithms function, the absence of a clear legal basis for data processing, the repurposing of data for new goals, and the insatiable demand for unrestrained data volumes.
The European Union has responded to these challenges by preparing the AI Act, but even before its full enforcement, current AI development practices must be strictly governed by the GDPR. Understanding compliance requires differentiating between the two distinct phases of an AI system’s lifecycle, as outlined by guidance from the French Data Protection Authority (CNIL).
- The Development Phase: This stage encompasses the design, development, and training of the AI model, including data set creation.
- The Deployment Phase: This deals with the eventual utilization of the developed AI system in operational business contexts.
Defining Roles: Controller, Processor, and Joint Controllership
The determination of roles under the GDPR—Data Controller or Data Processor—is the foundational step for compliance in AI development.
The Data Controller
A provider of an AI system or the entity responsible for the development and training of the data set often qualifies as the data controller under the GDPR. The controller determines the “why” (purpose) and “how” (means) of the processing. It is also possible to have joint controllers if the training of an AI system is done by more than one entity for a jointly defined purpose. For example, the CNIL highlights scenarios where academic hospitals develop an AI system for analyzing medical imaging data and choose to use the same federated learning protocol. By mutualizing data they originally controlled to train a medical AI system, they jointly determine the purposes and means of processing, thus becoming joint controllers.
The Data Processor
The CNIL guidance suggests that an AI system developer contracted by an organization to process data (controlled by that organization) to build a solution, and who then returns the data without assuming ownership, likely qualifies as a data processor. However, if the AI developer pools that data with data from other organizations, or intends to develop a solution for resale to other clients, the developer most likely assumes ownership of the processing purpose and becomes a data controller instead. This distinction is vital as it dictates the legal obligations and accountability mechanisms.
The Foundational Principles for AI Development
Beyond determining GDPR roles, three primary data protection principles must be considered when utilizing personal data for training AI algorithms:
- Transparency: Requires comprehensively informing data subjects about what data is being collected and how it is being used for AI training purposes.
- Data Minimization: Stipulates that only the data that is necessary, adequate, and relevant for the defined purpose should be collected and processed.
- Storage Limitation: Demands the definition and enforcement of a specific period for which the personal data may be retained.
The Purpose Limitation Dilemma
A common pitfall in AI development is the purpose limitation principle, which is closely linked to the fairness principle. While developers are quick to justify data hoarding by focusing on the greater good for humanity—such as developing a cure—the GDPR mandates that data processing only takes place for documented, explicit, and legitimate purposes. Collecting or hoarding data that has no immediate purpose is a direct violation of both the fairness and storage limitation principles.
For AI, the more data the better the training, which creates massive datasets that beg the question of data acquisition legality. When a dataset is reused, the CNIL distinguishes between the data diffuser (the entity that uploads personal data or a dataset online) and the reuser of the data (the entity who processes the data for their own purposes).
Reusing data, even if it is your own data as a controller, requires new legitimization because the GDPR does not allow for the arbitrary repurposing of data. If the original collection purpose did not foresee AI training, a new legal basis must be found and communicated to the data subjects along with other communication requirements.
Legitimizing Training Data: The Legal Bases
Legitimizing the processing of training data is arguably the most complex challenge for AI developers. Vague indications that data will be used “to improve the services offered to the data subject” are likely to be seen as too vague by regulators. Until a clear operational use case is defined—a specific problem the AI system is poised to solve—legitimizing the processing remains impossible.
Available Legal Grounds (Article 6.1)
For data that belongs to data subjects who will benefit from the AI system, several options are possible:
- Consent (Article 6.1a): This is a robust legal base that provides the most control to data subjects, as revoking consent indicates the organization must stop processing the data for that purpose. However, it is also highly volatile, as consent can be withdrawn at any time.
- Performance of a Contract (Article 6.1b): This is generally unlikely for training data, as the data is used for the controller’s development purpose, not the execution of a contract with the data subject.
- Legitimate Interest (Article 6.1f): This may be sufficient where the data is already controlled by the organization, the nature of the data is low risk, and supplementary safeguards are implemented. However, relying on this base is impossible if the controller has no contact with the data subject, as it prevents the data subject from exercising their right to object.
Compatibility Assessment (Article 6.4)
While outright repurposing data requires new consent, the GDPR provides in Article 6.4 the ability to submit data to a secondary use, provided that use is compatible with the initial purpose of the collection. The CNIL has provided guidance on how to make use of this provision by conducting a compatibility assessment that considers:
- The link between the original purpose and the new purpose (e.g., the second processing operation was already implicitly included).
- The context in which the data was collected (e.g., whether the data subject could reasonably expect the re-use).
- The nature of the personal data used (less favorable for sensitive data).
- The consequences of the second processing for individuals (risks to their rights and freedoms).
Furthermore, Norway’s data protection authority highlights that under GDPR Article 89 and Recitals 50 & 159, further processing is often presumed compatible for archiving purposes in the public interest, scientific or historical research, or statistical purposes. While universities may be able to claim this exemption, it is likely not sufficient for private organizations without clear public interest mandates, meaning acquisition and use of data for training typically requires its own independent legitimization.
The Challenge of Storage Limitation and Anonymization
Personal data cannot be processed indefinitely. This means that training data, when it qualifies as personal data, must be deleted once it has served its training purpose to satisfy the storage limitation principle. The most rigorous way to achieve this is through anonymization—a process that must ensure data subjects cannot be identified by any person, in any way, and at any time. Merely de-identifying or further pseudonymizing data does not constitute anonymization from a legal perspective; the GDPR still applies unless the data is truly anonymous.
Anonymization Techniques
Developers must look into and implement a combination of techniques to demonstrate they have anonymized their dataset, including:
- Top and bottom coding (capping values).
- Controlled rounding and imputation (replacing values with statistical estimates).
- Data swapping and generalization.
- Noise addition.
- Differential privacy techniques like k-anonymity, l-diversity, and t-closeness.
The Reproducibility Conflict
However, deleting or truly anonymizing training data once it has served its primary purpose creates a conflict with fundamental principles of trustworthy AI: reliability and reproducibility. The need to purge data to comply with GDPR storage and purpose limitation principles often violates basic principles of traceability in quality management and product development. The EU Commission’s High-Level Expert Group on Artificial Intelligence recognizes this tension in its Ethics Guidelines for Trustworthy AI.
For developers, the processing of data that is seen as sensitive (special categories of data in GDPR Article 9, such as racial, health, or religious data) is generally prohibited with very few exceptions, placing an even higher burden of proof and risk assessment on the controller.
Risk Assessment and Ethical Oversight
Since AI systems can lead to high risks for data subjects, a Data Protection Impact Assessment (DPIA) is often required. The DPIA maps and assesses risks from the data subject’s perspective and helps establish mitigation measures as described in GDPR Article 25.
DPIA Triggers (The Nine Criteria)
A DPIA is generally required when two of the nine EDPB Working Party criteria are met:
- Evaluation or scoring, including profiling.
- Automated decision-making with legal or similarly significant effects.
- Systematic monitoring of public areas on a large scale.
- Processing of sensitive data or data of a highly personal nature.
- Data processed on a large scale.
- The matching or combining of data sets from different sources.
- Data concerning vulnerable data subjects (e.g., children, employees, patients).
- The innovative use of new technology.
- When the processing prevents data subjects from exercising a right or using a service.
The CNIL highlights that while the creation of a standard AI system may not be deemed innovative, utilizing deep learning may still be seen as innovative because the risks of that technology are not fully understood yet, potentially triggering the need for a DPIA.
Ethical Committees and Trustworthy AI Principles
When validating design choices, consulting an ethical committee is highly recommended. These independent, multidisciplinary bodies provide guidance on potential ethical problems surrounding an AI system.
According to the independent High-Level Expert Group on Artificial Intelligence, organizations performing AI assessments for trustworthiness must consider seven key ethical requirements: Ethics Guidelines for Trustworthy AI
- Human Agency and Oversight.
- Technical Robustness and Safety.
- Privacy and Data Governance.
- Transparency.
- Diversity, Non-discrimination, and Fairness.
- Societal and Environmental Well-being.
- Accountability.
Considering these aspects allows an organization to prioritize compliance by fully understanding the ethical implications of the AI system, regardless of whether they develop or only deploy the system.
The Forthcoming AI Act: A New Layer of Regulation
The AI Act represents a massive regulatory step, imposing requirements that complement the existing GDPR framework. While the CNIL guidance focuses heavily on AI development, the AI Act will apply to developers, distributors, implementers, and users alike. Key objectives of the proposed regulation include:
- Mandating that developers effectively oversee the development and implementation of AI systems.
- Outlining prohibited AI practices, such as the real-time identification of individuals in public spaces.
- Providing a classification of high-risk systems (e.g., those used in critical infrastructure, education, or employment) and laying out strict compliance requirements for them.
- Mandating a European Artificial Intelligence Board and national competent authorities.
- Laying out expectations for post-market monitoring, information sharing, and market surveillance.
- Outlining penalties and providing necessary complimentary information, such as a list of high-risk AI systems and the EU declaration of conformity in its annexes.
With the expected passing of the AI Act, future guidance from European data protection authorities will focus heavily on how the AI Act and the GDPR interact, providing crucial rulings, guidance, and case law necessary to complete the regulatory puzzle.
Conclusion: Building Trust by Design
The development of artificial intelligence systems presents a unique collision between technological ambition and fundamental data protection rights. Success in this field requires more than just technical expertise; it demands Privacy by Design and Ethics by Design—ensuring that data minimization, purpose limitation, and transparency are built into the AI lifecycle from the initial design phase. By clearly defining GDPR roles, rigorously conducting compatibility assessments, enforcing genuine anonymization, and consulting with independent ethical committees, organizations can move beyond mere compliance to build trustworthy AI. The confluence of the stringent GDPR principles and the comprehensive requirements of the forthcoming AI Act defines a clear, albeit challenging, path forward. Ultimately, the future of AI will be shaped not only by its computational power but by the ethical intelligence and legal accountability of its creators.
