In Part 1 of this series, I instructed AI machines from three different companies to create a literature review for the same subject/review title. In this second part, I evaluate the literature cited by the AI tools: does it exist and indeed, is it relevant. Surprisingly, some references don’t exist and some are not relevant!
You might recall that I created a table of the all referrenced papers in Part 1 of this series. It included the titles of 13 papers in each of the literature reviews by the three AI tools from: Google’s Gemini, Open AI’s Chat GPT, and Anthropic’s Claude.
Does the cited paper exist?
In summary: Of the 13 papers then only 11 of them actually existed. Two of the papers could not be found at all, despite searching in Scopus, ScienceDirect and Google.
Both of the citations for the non-existent papers were in the Gemini literature review. This was particular unfortunate as it included a table which quoted a figure for an increase in potato yields and then cited it in the a non-existent papers.
Firstly “Zhang, Y., et al. (2019) ‘Effects of biochar application on potato yield and nitrogen utilization efficiency‘, Journal of Plant Nutrition and Fertilizers, 25(6), pp. 980–989)”; and secondly: “Liu, X., et al. (2021) ‘Biochar’s effect on crop productivity and greenhouse gas emissions under field conditions‘, Scientific Reports, 11(1), pp. 1–12.”. Neither of these papers exist – or can be found!
Relevance of the cited papers
Where a cited papers existed then it was catagorised as ‘not relevant‘, ‘relevant‘ or ‘less relevant‘ depending on their relevance to the subject of the literature review.
There were two papers cited which were ‘not relevant’. For example, one study was about wheat seedling germination with and without biochar.
Therefore, nine of the papers had some relevance to the instruction to ‘Write a literature review with references to peer reviewed and published papers of no more than 750 words showing the effect on potato plant yields of cultivation using charcoal (wood biochar) as a fertilizer’.
But of these nine papers then I regarded only seven of them as ‘relevant’ and the remainder as ‘less relevant’.
To be ‘relevant’ then the paper title had to at least include ‘potato’ and ‘biochar’! Both Claude and Chat GPT had some relevant papers but Gemini had none. But remember that this only one trial of the three AI tools.
You may wonder the value of a literature review – like the one produced by Gemini – where there are no relevant papers cited! The difficulty for the reader is that if a literature review makes claims about facts and those facts cannot be cited in peer reviewed literature then we must wonder about the validity of the claim, no matter how convincing the text and its arrangement of words! It’s surely no better than hearsay?
Summary
This table summarises the results of requesting each of the AI tools to write a literature review using the directions given above. See Evaluation Results below for details.
| Chat GPT Review | Gemini Review | Claude Review | |
|---|---|---|---|
| Number of papers referenced | 5 | 4 | 6 |
| Relevant | 5 | 4 | |
| Less relevant | 2 | ||
| Not relevant | 2 | ||
| Doesn’t exist | 2 | ||
| Link to literature review generated (PDF) and version of AI software used on 25 March 2026 | OpenAI (2026) ChatGPT (GPT-5.3) | Gemini (Gemini 3 Flash version) [Large language model] | Anthropic (2025) Claude Sonnet 4.6 [Large language model |
Conclusion
Based on the limited case study carried out here then on the references alone, none of the literature reviews written by AI machines are particularly good and indeed citing papers which don’t exist – which Google’s Gemini did – is a particular failure! Gemini performs worse in terms of relevance than Chat GPT or Claude.
Note that this article is only about references in the three literature reviews. A further review of the claims made in the three generated literature reviews needs to be undertaken to fully assess their value.
However, if the AI machines literature reviews cannot justify their selection of facts based on citations that exist and that are relevant then the text generated – while it may be reable and convincing – is of little value, and arguably is no better than hearsay.
You might also like: Manifesto for Embodiment: a riposte to machine capture
See all articles with tag crop trial
Evaluation Results
To assist me in finding the papers, I asked Google’s Gemini to produce a Digital Object Identifier (DOI) where available (week of Monday 19 May 2026), for each paper in my table of references. The DOI’s which it provided where very unreliable. There were seven where the DOI didn’t exist or linked to the wrong article! I’ve included this result in the evaluation table below.
Each row of this table matches the corresponding one from the table in part 1 of this series. As previously noted: there were only two referenced papers which were identified by more than one of the AI machine tools (Chat GPT, Claude, Gemini).
| Article Citation | DOI; AI tool(s) using reference; Comments |
| Biswas, B.R., et al. (2024) ‘Synergistic effects of biochar and vermicompost on yield and quality of potato’, Bangladesh Agronomy Journal. | DOI 10.3329/baj.v28i2.85870 Chat GPT Relevant: I don’t know the the compost – including vermicompost if any – was applied by gardeners in my research paper. Therefore this is relevant as it can’t be excluded. |
| Blackwell, P., et al. (2010) ‘Effect of biochar on germination and early growth of seedlings‘, Annals of Applied Biology, 156(1), pp. 125–133. | DOI 10.1111/j.1744-7348.2009.00367.x (This DOI is not correct. Directs to wrong paper) Gemini Not relevant: as due to growth and germination of wheat, mung bean, and subterranean clover rather than potato. |
| Carter, S., et al. (2013) ‘The Impact of Biochar Application on Winter Wheat and Starchy Tuber Yields over Two Cropping Seasons‘, Agronomy, 3(2), pp. 338–372. | DOI 10.3390/agronomy3020338 (This DOI cannot be found by DOI) Gemini Not relevant: as (from abstract) ‘we conducted a pot experiment using a wheat-millet rotation to examine the effects of two successive rice-straw biochar applications. Not a field experiment and also not potatoes. |
| Dawerasha, S.S., et al. (2024) ‘Effect of coffee husk biochar and inorganic NP fertilizer on soil properties, growth and yield of potato (Solanum tuberosum L.) on acidic soil of southwest Ethiopia’, CABI Agriculture and Bioscience, 5, article 56. | DOI 10.1186/s43170-024-00261-y Claude Less relevant: as only wood biochar was used by the gardeners, not coffee husk biochar … and also ‘Coffee husk biochar and wood biochar share the basic characteristics of being carbon-rich, porous materials used for soil amendment and carbon sequestration, but they differ significantly in nutrient content, pH, and surface chemistry, with coffee husk generally acting as a more nutrient-dense, high-ash soil conditioner’ (See: Properties of biochar derived from wood and high-nutrient biomasses with the aim of agronomic and environmental benefits https://doi.org/10.1371/journal.pone.0176884 in Plus One ‘) |
| Guo, J., et al. (2025) ‘The effects of biochar on soil quality and potato yield in arid and semi-arid regions’, Research Study (preprint). | DOI For the DOI Gemini says ‘No DOI assigned (Preprint)’ yet the paper is published … and not a pre-print. (Gemini says ‘Note: For the Guo et al. (2025) article, as it is listed as a “Research Study (preprint),” it may not have a permanent DOI until it completes formal peer review and publication. The DOI provided for Paswan et al. (2025) is based on current digital indexing for the Arab Universities Journal.’) Chat GPT Relevant |
| Hou, J., et al. (2024) ‘Increase in potato yield by the combined application of biochar and organic fertilizer’, Frontiers in Plant Science. | DOI 10.3389/fpls.2024.1389864 Chat GPT & Claude Relevant |
| Liu, X., et al. (2021) ‘Biochar’s effect on crop productivity and greenhouse gas emissions under field conditions‘, Scientific Reports, 11(1), pp. 1–12. | DOI 10.1038/s41598-021-91552-w (This DOI cannot be found in the DOI System) Gemini Doesn’t exist: The paper title and author cannot be found in Sciencedirect nor Scopus nor Google Scholar |
| Marat, A.K. (2023) ‘Development of biochar to improve soil health and increase potato yields’, International Journal of Design & Nature and Ecodynamics, 18(1), pp. 221–228. | DOI 10.18280/ijdne.180129 Claude Relevant: used wood biochar in my study |
| Mawof, A., et al. (2021) ‘Effects of biochar and biochar-compost mix as soil amendments on soil quality and yield of potatoes irrigated with wastewater’, Journal of Soil Science and Plant Nutrition, 21, pp. 2600–2612. | DOI 10.1007/s42729-021-00549-2 Claude Less relevant: as used barley straw biochar in paper. |
| Mollick, M.O.A., et al. (2020) ‘Effect of biochar on yield and quality of potato (Solanum tuberosum) tuber’, International Journal of Bio-resource and Stress Management. | DOI 10.23910/1.2020.2140 Chat GPT & Claude Relevant |
| Paswan, S., et al. (2025) ‘Effect of biochar with different fertilizers on growth and yield of potato (Solanum tuberosum L.)’, Arab Universities Journal of Agricultural Sciences. | DOI 10.21608/AJS.2025.321156.1578 – This DOI cannot be found in the DOI System The actual title of this paper is ‘Effect of Biochar with Different Fertilizers on Growth and Yield of Bajhang Local Variety of Potato (Solanum tuberosum L.)’ Chat GPT Relevant: Paper says ‘Biochar, sourced from local households and produced from wood, tree branches, and agricultural residues’ was used in this study. |
| Wang, H., et al. (2025) ‘Effects of biochar from different pyrolysis temperatures on soil physical properties and hydraulic characteristics in potato farmland of arid and semi-arid regions’, Agricultural Water Management, 313, article 109501. | DOI 10.1016/j.agwat.2025.109501 This DOI links to wrong paper (‘Evaluation of the LI-710 evapotranspiration sensor in comparison to full eddy covariance for monitoring energy fluxes in perennial and annual crops‘) Claude Relevant. While this paper considers soil and hydraulic characteristics, it also compared yields under application of biochar, finding ‘27.15 % increase in potato yield compared to the control’. Although uses wheat straw as the biochar rather than wood. |
| Zhang, Y., et al. (2019) ‘Effects of biochar application on potato yield and nitrogen utilization efficiency‘, Journal of Plant Nutrition and Fertilizers, 25(6), pp. 980–989. | DOI 10.11674/zwyf.18283 This DOI produced ‘Whitelabel Error Page. Gemini Doesn’t exist: Is there a mismatch in the author and the title? Google search on title gives this … https://www.frontiersin.org/journals/plant-science/articles/10.3389/fpls.2024.1389864/full Front. Plant Sci., 15 May 2024 Sec. Plant Nutrition Volume 15 – 2024 | https://doi.org/10.3389/fpls.2024.1389864 Increase in potato yield by the combined application of biochar and organic fertilizer: key role of rhizosphere microbial diversity Jianwei Hou, CunFang Xing, Jun Zhang, Zuhua Wang, Min Liu, Yu Duan, Hui Zhao … which is not the same! Further Google search on author ‘Zhang, Y., et al. (2019)’ gives this article … but it doesn’t have either the name of author nor indeed the correct year: https://www.sciencedirect.com/science/article/pii/S0960852421007914 Bioresource Technology Volume 337, October 2021, 125451 Review Production and beneficial impact of biochar for environmental application: A comprehensive review Yuwen Zhou, Shiyi Qin, Shivpal Verma,,Taner Sar, Surendra Sarsaiya, Balasubramani Ravindran, Tao Liu,, Raveendran Sindhu, Anil Kumar Patel, Parameswaran Binod, Sunita Varjani, Reeta Rani Singhnia, Zengqiang Zhang, Mukesh Kumar Awasthi Also … searching on the website of Journal of Plant Nutrition and Fertilizers (https://www.plantnutrifert.org/en/to_advance_search) does not find a paper with this title. Nor looking at https://www.plantnutrifert.org/en/article/2019/6 which is the 2019 Vol. 25 No. 6 publication and for which a search in the page with ‘Zhang, Y’ produces no match although there is a biochar paper but not with the title nor authors!! |