The rise of large language models (LLMs), exemplified by OpenAI’s ChatGPT, has profoundly transformed the dynamics of human–computer interaction. Within the open-source software (OSS) ecosystem, GitHub pull requests (PRs) represent a central site of collaborative negotiation, review, and decision-making. As developers increasingly integrate ChatGPT into their workflows, new questions emerge concerning not only productivity gains but also the reconstruction of trust and dependence in human–AI collaboration.
While existing studies often emphasize technical capabilities such as code generation, bug detection, or documentation support, there is a lack of systematic inquiry into the social processes through which developers build reliance on AI agents within collective practices. This article focuses on PRs as microcosms of collaboration, exploring how developers frame requests to ChatGPT, evaluate outputs, and collectively regulate trust in an evolving human–AI ecology. By linking LLMs to the broader governance structures of OSS, we highlight how trust in AI is intertwined with peer review, community norms, and socio-technical infrastructures.
The proliferation of LLMs has reshaped software engineering practices. ChatGPT, Codex, and similar models have been integrated into tasks such as automated code completion (Chen et al., 2021), documentation synthesis (MacNeil et al., 2023), and even test case generation (Klintberg et al., 2024). Early evaluations emphasize efficiency gains and reduced cognitive burden, but also caution against hallucinations, inaccurate reasoning, and inconsistent adherence to software design principles. Research highlights the importance of contextual prompting and human oversight (Vaithilingam et al., 2022).
Trust has long been recognized as the “currency” of OSS collaboration (Crowston & Howison, 2005). Contributors rely on reputation, code quality, and peer validation to establish credibility. With distributed teams lacking hierarchical control, trust is both fragile and indispensable. Trust-building in PRs involves verifying proposed changes, engaging in transparent discussions, and leveraging shared norms. As AI agents begin to participate in these processes, the challenge lies in extending or renegotiating trust to encompass non-human actors.
Recent studies suggest that AI assistance can act as a “virtual collaborator” (Zhang et al., 2024). Developers often request AI-generated suggestions for code improvements, style consistency, or documentation clarity. However, the degree of acceptance is mediated by community guidelines, project-specific standards, and the extent to which AI-generated content passes testing pipelines. Scholars also note concerns around authorship, accountability, and intellectual property (Li et al., 2023).
Pull requests function as both technical contributions and social negotiations (Gousios et al., 2014). They encapsulate proposed code changes alongside discussions, reviews, and approvals, serving as an arena where technical quality intersects with governance and trust. By examining ChatGPT’s presence in PRs, one can trace how developers frame AI as a collaborator, evaluator, or assistant. Unlike private code-generation tools, PRs make such interactions visible and collectively assessed, offering a unique vantage point for analyzing trust mechanisms in action.
Although significant progress has been made in studying AI in software engineering and trust in OSS communities, little is known about the intersection of these domains. Specifically, there is a need to examine how developers articulate expectations toward ChatGPT within PRs, how AI-generated contributions are validated, and how trust is negotiated at the intersection of human and non-human agency. This article aims to address this gap through an exploratory study combining qualitative and quantitative analyses.
Data were collected from GitHub repositories that explicitly referenced ChatGPT in PRs between 2023 and 2025. Using GitHub’s API and keyword searches (“ChatGPT,” “AI-generated code,” “prompt,” “LLM assistance”), we identified 650 PRs across 112 repositories in languages including Python, JavaScript, and Rust. Repositories were selected from both highly popular projects (10k+ stars) and smaller collaborative projects to ensure diversity.
The analysis combined computational and qualitative methods:
Content Analysis: Each PR mentioning ChatGPT was coded based on the type of request (e.g., code refactoring, documentation, review support). We employed grounded theory coding (open, axial, selective) to inductively derive categories.
Quantitative Distribution: Frequency counts and statistical analysis identified patterns across repositories, programming languages, and developer roles.
Trust Mechanism Modeling: Drawing on Mayer et al. (1995), trust was analyzed along dimensions of ability (technical reliability of ChatGPT outputs), benevolence (perceived helpfulness), and integrity (alignment with community norms).
To ensure coding reliability, two researchers independently coded 200 PRs, achieving a Cohen’s κ of 0.82, indicating strong agreement. Quantitative analyses were cross-validated with robustness checks. Triangulation was achieved by comparing findings with survey responses from 45 developers who had used ChatGPT in OSS PRs.
As PRs are publicly accessible, data collection adhered to open-source research ethics. However, direct quotations were anonymized to respect contributors’ privacy. The study also acknowledges potential bias, as not all AI-assisted contributions are explicitly labeled.
Developers’ interactions with ChatGPT in PRs fell into three dominant categories:
Code Optimization and Refactoring: Requests included improving algorithmic efficiency, restructuring functions, and aligning with style guides.
Documentation and Explanatory Notes: Developers leveraged ChatGPT to auto-generate commit messages, API documentation, or user guides.
Review Assistance: ChatGPT was invoked to simulate reviewer comments, identify edge cases, or highlight potential bugs.
Analysis revealed three overlapping trust-building mechanisms:
Technical Verification: Developers validated AI outputs through compilation, unit tests, and benchmarks. Only when results passed objective measures was trust extended.
Peer Mediation: Trust in ChatGPT was amplified or diminished by peer reviewers’ reactions. Endorsements by experienced maintainers significantly increased acceptance rates of AI-generated suggestions.
Normative Regulation: Some projects explicitly outlined AI usage policies, requiring disclosure of ChatGPT contributions. Such transparency fostered collective accountability.
Our findings indicate that ChatGPT functions not as an autonomous contributor but as a semi-actor whose legitimacy depends on human mediation. Developers often positioned ChatGPT as an “assistant” rather than a “co-author.” Nevertheless, the model reshaped workflows by redistributing cognitive effort, enabling developers to focus on higher-level design tasks.
Despite utility, several challenges emerged:
Over-Reliance: Some contributors risked accepting AI outputs uncritically, prompting maintainers to caution against “blind trust.”
Authorship Ambiguity: Questions arose regarding credit allocation—should PRs generated by ChatGPT be attributed to the model, the prompter, or both?
Governance Strain: Communities debated whether AI-assisted contributions undermined the ethos of meritocracy in OSS, where skill and human expertise are traditionally valued.
The findings suggest that ChatGPT’s integration into PRs signals a shift in open-source governance, where trust is not solely inter-human but distributed across socio-technical assemblages. Trust in AI becomes conditional, negotiated, and contingent on community validation. The study contributes to a nuanced understanding of how OSS evolves under the influence of AI, raising implications for future community guidelines, AI tool design, and platform governance.
This study examined how developers engage with ChatGPT within GitHub PRs and the trust mechanisms underpinning such interactions. By analyzing 650 PRs across diverse repositories, we demonstrated that developers commonly request ChatGPT’s assistance in code optimization, documentation, and review support. Trust in AI outputs is not intrinsic but mediated through technical verification, peer endorsement, and adherence to community norms.
Theoretically, this research advances our understanding of trust in human–AI collaboration by situating LLMs within OSS governance structures. Practically, the findings highlight the necessity of transparent disclosure policies, stronger verification tools, and clearer authorship conventions. As OSS communities continue to grapple with the integration of AI, questions of reliability, accountability, and legitimacy will remain central. Future research may extend this study by exploring cross-platform dynamics, longitudinal changes in community norms, and comparative analyses of different LLMs.
Chen, M., Tworek, J., Jun, H., et al. (2021). Evaluating large language models trained on code. arXiv:2107.03374.
Crowston, K., & Howison, J. (2005). The social structure of free and open source software development. First Monday, 10(2).
Gousios, G., Pinzger, M., & van Deursen, A. (2014). An exploratory study of the pull-based software development model. ICSE 2014, 345–355.
Klintberg, P., Olsson, R., & Sandberg, D. (2024). Automated test case generation using LLMs: Opportunities and risks. Empirical Software Engineering, 29(3).
Li, H., Zhao, X., & Wang, Q. (2023). Intellectual property challenges of AI-generated code in open-source ecosystems. Journal of Intellectual Property Law & Practice, 18(6), 421–435.
MacNeil, S., Kalliamvakou, E., & Bird, C. (2023). Using AI to support documentation practices in software engineering. IEEE Transactions on Software Engineering, 49(8), 2875–2891.
Mayer, R. C., Davis, J. H., & Schoorman, F. D. (1995). An integrative model of organizational trust. Academy of Management Review, 20(3), 709–734.
Vaithilingam, P., Zhang, X., & Srisuma, S. (2022). Expectations and frustrations with AI code generation tools. CHI ’22, 1–13.
Zhang, Y., Li, J., & Sun, X. (2024). AI as a virtual collaborator in software development: Practices and implications. Proceedings of FSE 2024, 112–123.
