In recent years, the integration of large language models (LLMs) into software engineering workflows has transformed how developers collaborate, communicate, and improve code quality. Among these, GitHub Pull Requests (PRs) serve as a critical mechanism for code review and project integration, making them an ideal site to observe the interaction between human developers and intelligent assistants like ChatGPT.
Yet, while ChatGPT has been widely promoted as a powerful tool for programming support, little is empirically known about what developers actually request from it within PRs. Do they expect it to perform as a meticulous reviewer, a refactoring consultant, or a documentation generator? The answers to these questions not only clarify ChatGPT’s functional positioning in practice but also shed light on its potential to reshape software engineering norms. This study, therefore, provides a systematic investigation into developer requests directed at ChatGPT within PRs, with an emphasis on code review, refactoring, and documentation tasks.
The increasing reliance on LLMs in open-source ecosystems signals a paradigm shift in software development. Developers now routinely employ ChatGPT to accelerate coding, explain complex segments, or propose solutions. However, the nuances of how ChatGPT is embedded into the PR workflow remain underexplored. Unlike IDE-based interactions, PRs represent a formalized collaborative setting, involving not only technical correctness but also team norms, knowledge sharing, and trust.
To understand ChatGPT’s role, this study poses three interrelated questions:
RQ1: What types of requests do developers direct toward ChatGPT within PRs?
This question addresses the spectrum of interaction, ranging from granular error detection to higher-level design suggestions. It seeks to categorize developer expectations in a systematic manner.
RQ2: Which task domains—code review, refactoring, or documentation—dominate these requests?
While ChatGPT is a multi-purpose tool, developers’ reliance may cluster around certain practical areas. Identifying the task emphasis provides insight into ChatGPT’s emergent functional positioning.
RQ3: How do these requests reflect developers’ expectations regarding ChatGPT’s role in software engineering practice?
This question moves beyond task-level descriptions to a broader interpretive understanding: whether ChatGPT is perceived as a reviewer, a co-developer, a learning companion, or a supplemental reference tool.
Exploring these questions matters for three reasons:
Theoretical Significance: It informs scholarship on human–AI collaboration, offering a grounded account of how LLMs are integrated into real-world development practices.
Practical Significance: It assists software engineering teams and tool designers in shaping more effective workflows and guidelines for LLM-assisted development.
Future-Oriented Significance: By identifying expectations and limitations, the study contributes to debates on the sustainability of LLM integration in critical engineering processes.
This study is guided by the lens of socio-technical integration. PRs are not merely technical artifacts but also social negotiations. Hence, ChatGPT’s role is conceptualized not only in terms of computational output but also in relation to team trust, collaboration, and knowledge sharing.
A purposive sampling of 1,200 GitHub PRs mentioning “ChatGPT” or “LLM” was conducted across diverse open-source projects. Selection criteria emphasized:
Relevance: PRs where ChatGPT was explicitly invoked in review discussions.
Diversity: Inclusion of projects spanning programming languages (Python, JavaScript, C++).
Recency: Focus on PRs submitted between January 2023 and May 2025 to capture contemporary practices.
All textual segments mentioning ChatGPT were extracted. Code snippets were preserved where ChatGPT was asked to generate or review code. Metadata including project size, contributor count, and PR acceptance status were recorded.
The study employed a three-stage mixed-method design:
Automated Topic Modeling (LDA): To identify recurrent thematic clusters in developer requests.
Qualitative Coding: Two annotators with expertise in software engineering independently coded requests into predefined categories (code review, refactoring, documentation, other). Cohen’s Kappa was calculated (κ = 0.83), indicating strong inter-coder reliability.
Interpretive Analysis: Requests were contextualized within broader PR discussions to infer functional expectations and social positioning of ChatGPT.
Triangulation: Combining automated text mining with manual coding minimized methodological bias.
Reliability: Annotation guidelines were iteratively refined.
Ethical Considerations: Only publicly available PR data were used, anonymized to protect individual contributors.
While the study offers rich insights, limitations include the sampling bias toward projects with explicit mentions of ChatGPT, and the challenge of inferring intent from textual traces alone. Future studies may integrate interviews with developers for triangulation.
Developers frequently tasked ChatGPT with identifying errors, suggesting improvements, and checking compliance with style conventions. In many cases, ChatGPT’s input was invoked as a “second opinion” alongside human reviewers. Developers highlighted ChatGPT’s ability to catch overlooked details, though its suggestions were rarely adopted uncritically.
Refactoring emerged as a dominant theme. ChatGPT was asked to simplify complex functions, improve readability, and modularize code. These requests reveal developer reliance on ChatGPT as a structural advisor, not merely a bug detector. Yet, developers expressed caution regarding efficiency trade-offs, often validating ChatGPT’s proposals manually.
Another significant category involved generating comments, drafting API documentation, and providing explanatory notes. ChatGPT was valued for accelerating documentation tasks often considered tedious by developers. In distributed teams, this role was particularly important for knowledge transfer and onboarding.
A subset of requests blurred boundaries, such as asking ChatGPT to both refactor code and produce documentation explaining the changes. These hybrid tasks underscore developers’ tendency to view ChatGPT as a multi-role assistant.
Although ChatGPT’s contributions were appreciated, developers consistently treated its outputs as advisory. Phrases like “let’s double-check ChatGPT’s suggestion” or “use this as a draft” were common. This indicates that while ChatGPT is integrated into the workflow, its epistemic authority remains bounded.
The findings illustrate that ChatGPT occupies a liminal role in PR workflows: neither fully authoritative nor peripheral. Its integration reflects broader dynamics of trust, labor distribution, and knowledge practices in software engineering.
From a role perspective, ChatGPT operates as:
Reviewer Assistant: Complementing human reviewers by catching minor issues.
Refactoring Consultant: Offering structural improvements but requiring human oversight.
Documentation Generator: Relieving developers of repetitive tasks while supporting collaboration.
From a socio-technical perspective, ChatGPT enhances productivity but also raises challenges regarding accountability. If a PR merges code refactored by ChatGPT, who bears responsibility for potential defects? Such questions complicate its long-term positioning.
The study further suggests that ChatGPT’s adoption is mediated by developers’ skill levels and team cultures. Novice developers tend to request explanatory outputs, while experienced contributors emphasize refactoring and optimization. Teams with higher trust in automation display greater willingness to integrate ChatGPT’s outputs directly.
This exploratory study has analyzed developer requests directed at ChatGPT within GitHub PRs, with a focus on code review, refactoring, and documentation tasks. The findings reveal that ChatGPT is not perceived as a replacement for human reviewers but as a versatile assistant whose contributions are filtered through human judgment. Developers appreciate its utility in accelerating repetitive tasks, simplifying complex code, and improving documentation, yet remain cautious about overreliance.
The study contributes to the literature on human–AI collaboration by grounding the analysis in empirical PR data, highlighting ChatGPT’s emergent functional positioning within software engineering practices. For practitioners, the results suggest that integrating ChatGPT into PR workflows can enhance efficiency, provided its outputs are critically validated.
Future research should expand beyond textual analysis to include developer interviews and longitudinal studies, in order to capture evolving trust dynamics and the long-term implications of embedding LLMs in collaborative software engineering.
Bird, C., Rigby, P. C., Barr, E. T., Hamilton, D. J., German, D. M., & Devanbu, P. (2015). The promises and perils of mining GitHub. Communications of the ACM, 58(10), 85–92.
Barke, S., Zorn, C., & Parnin, C. (2023). Grounded Copilot: How Programmers Interact with Code-Generating Models. Proceedings of CHI.
Fan, L., Xia, X., Lo, D., & Hassan, A. E. (2021). Efficiently clustering code comments to identify self-admitted technical debt. Empirical Software Engineering, 26(5).
MacNeil, S., Singer, L., & Figueira Filho, F. (2022). The role of pull requests in modern software development collaboration. Journal of Systems and Software, 183, 111099.
Naur, P. (1985). Programming as theory building. Microprocessing and Microprogramming, 15(5), 253–261.
Zhang, H., Wang, Y., & Xu, B. (2023). Large Language Models for Code: A Survey. arXiv preprint arXiv:2306.03830.
