ChatGPT as a Grammar Checker for Japanese English Language Learners: A Comparison with Grammarly and ProWritingAid

105-119


Introduction
Grammar check software has been available for quite some time. Grammarly was first released in 2009 (Lytvyn, 2022) and ProWritingAid in 2012 (Blogging Tips, n.d.). The development of grammar-checking software has allowed for the ability to detect errors in writing and provide corrective feedback automatically and efficiently (Yang Hye Jin, 2018). Over the past ten years, the popularity of these programs has increased, with Grammarly now having 30 million users ("About Us," n.d.). Both language learners and native speakers have found the use of such software to help improve their writing skills and gain increased confidence in their writing (O'Neill & Russell, 2019).

Literature Review
Many researchers (O'Neill, 2019;Perdana & Farida, 2019;Dizon & Gayed, 2021;Wang & Zhong, 2022) have investigated the use of Grammar check software as a helpful aid for English language learners (ELLs) in improving their writing skills. Tran and Nguyen (2021) found that the use of the grammar checker ProWritingAid benefited students in terms of improvement in the quality of their writing as well as confidence. According to Ghufron and Rosyida (2018), ELLs often struggle with grammatical accuracy in their writing. Based on his experience in teaching Japanese university students for over the past 20 years, the author supports this finding, as many students make common grammatical mistakes in areas such as the use of articles, verb agreement, and tense. The author's experience supports Tokunaga's finding (2021) that there is a widely held, clichéd, and inaccurate belief that Japanese learners of English possess high grammatical accuracy. Japanese ELLs may find grammar correction software a helpful resource since errors can quickly and easily be located in a text, making it easier for learners to recognize and correct them (Fitriana & Nurazni, 2022). ELLs may additionally become better aware of the conventions of English grammar and punctuation, which can be beneficial for their overall language development as well as confidence (Cavaleri & Dianati, 2016). Based on a systematic review to examine the applications of grammar checkers in academic writing, Perdana and Farida (2019) found that such tools can be useful for detecting errors and providing corrective feedback. Ghaemi and Bayati (2021) came to a similar conclusion, with their research indicating that grammar-checking software benefitted EFL learners' writing ability and overall language development. An investigation by  into the impact of using grammar correction software on the writing performance of Chinese EFL learners also showed a positive effect on the learners' grammatical accuracy and fluency.
The autonomy and independence of ELLs may also be developed through the use of grammar check software (Cavaleri & Dianati, 2016: John & Woll, 2020. The use of technology in English language learning to foster autonomy, which can enhance student confidence, determination, and learning skills, is encouraged by Pham (2022). Grammar check software may be incorporated as part of a process writing approach at the draft stage before final submission. This may lead students to be less dependent on a teacher to check simple mistakes such as missing articles, verb agreement, and spelling, which they would be able to correct on their own before final submission. This would allow students to reflect on their writing to improve before final submission. Often it has been the case that students only receive grammar and spelling feedback after their writing assignment's final submission. This type of feedback at this stage may only end up receiving a cursory glance from students who may feel no further need to apply the feedback since the assignment, in their eyes, has already been completed. This may not be as deep a learning experience as when using a grammar checker to reflect on the errors found before submitting a final draft.
Teachers also benefit from the use of grammar checkers as considerable time is saved in not needing to correct and give feedback on basic grammatical errors and spelling mistakes when a grammar checker could easily be used to check such items (Toncic, 2020;Al-Ahdal, 2020). According to Nguyen et al. (2022), the amount of time available to lecturers influenced their choice of focusing on correcting common mistakes as the most popular method of written corrective feedback (WCF) at Van Lang University. Having students use a grammar checker to check for common mistakes would allow teachers the time to evaluate student writing on a more holistic level rather than individual linguistic elements. The ability to convey ideas and meaning, the organization of the writing, and the use of language, voice, and tone could be concentrated upon, giving students deeper and more meaningful feedback.

Student use of grammar checkers questionnaire
Before the availability of ChatGPT in November 2022, during the spring semester of 2022, students were encouraged to use grammar checkers when working on the final draft of their writing assignment. The author introduced students to Grammarly, ProWritingAid, Quilbot, Whitesmoke, and Ginger. Students were told that they might try any of the mentioned grammar checkers or any other that they come across in an internet search. Based on previous experience in using grammar checkers and a review of the literature (John & Wolf, 2020: Wang, 2022, the author cautioned students that they should not consider the resulting analysis as being infallible and as carefully considering whether the feedback is appropriate to the intended context of their writing.
At the end of the spring term, a short questionnaire was given to students regarding their experience in using grammar checkers. The questionnaire was administered ethically by explaining the purpose of the questionnaire. Participation was voluntary. The responses were collected anonymously. The replies were as follows: Results of the short questionnaire indicate that nearly all students found the use of grammar checkers helpful and easy to use. Although a large majority of students felt that the grammar checkers found many mistakes and helped them understand them, some students did not agree with this. In the case of not finding many mistakes, it may be that the student's writing was already sufficiently correct to begin with, or the number of errors was few. In the case of not helping to understand mistakes, it may be that the grammar checkers gave insufficient explanations as to why there were mistakes, or the explanations were unclear.

Comparison of Grammarly, ProWritingAid, and ChatGPT
Based on the results of a short questionnaire administered in the spring semester of 2021, students found the use of grammar checkers to help find grammar mistakes as well as be easy to use. The launch of ChatGPT in November 2022 sparked the author's interest in how it may be used as a grammar checker and how it would compare to other available grammar checkers. In the following section, a comparison is made between the more established grammar checkers Grammarly and ProWritingAid with that of ChatGPT, which is not strictly speaking a grammar checker, but whose functionality allows it to be used as one.
Wahyuda's (2022) comparison of Grammarly with ProWritingAid found that both could significantly improve the writing accuracy and fluency of the participants. Grammarly was found to be more effective for lower-level students, while ProWritingAid was found to be more appropriate for higher-level students. Although ChatGPT is not strictly speaking grammar check software, it can check grammar and provide advice in a similar way to that of Grammarly and ProWritingAid. The grammar checkers, along with ChatGPT, will be used in assessing a writing assignment by a first-year Japanese university student to determine their effectiveness and usefulness.
The use of the student's writing assignment in comparing the grammar check software with ChatGPT was conducted in an ethical manner through informed consent. The purpose and steps of the experiment were clearly explained to the student, including that their privacy and anonymity would be respected. To ensure that the student had a comprehensive understanding of the experiment, their consent to use their assignment as part of the analysis was requested, and they willingly consented. The following is the original student text used in the analysis: I experienced a big earthquake when I was an elementary school student. Then, it was very dangerous that things that were around me fell in. Also, it cracks in the ground and causes power outages. I thought of two things to solve these problems. First, we should equip suitable things, that is water and preserved foods, etc. Second, we should know the evacuation route. Because it is too late after the disaster occurs, and there is a possibility of injury due to delay in evacuation.

Grammarly
Grammarly's interface consists of a simple and straightforward single page comprising of a text editor window, a sidebar with a list of potential errors, and a results section that summarizes all the errors found. The errors are divided into categories such as grammar, spelling, and punctuation. Clicking on each error brings up a detailed explanation of the mistake, as well as suggested corrections. The free version of Grammarly offers grammar and spelling checking along with basic reports and writing suggestions. The premium version offers a more advanced set of features, including plagiarism detection, advanced reports, and over 400 additional grammar rules (Grammarly, n.d.). The latest pricing of the premium version may be confirmed by visiting the Grammarly website. For the analysis, the premium version was used.
Grammarly indicated most of the simple mistakes in the text correctly. The explanations were short but easily understandable. However, it did not give any suggestions for the mistakes found in the sentence parts "very dangerous that thing that is around me fall in," and "we should equip suitable thing." ProWritingAid ProWritingAid can be installed by using the Google Chrome extension. Once installed, users will be able to use ProWritingAid directly in the browser. It was initially unclear how the extension is to be used when analyzing text, which made its use slightly more confusing than that of Grammarly.
The ProWritingAid interface is similar to that of Grammarly. However, the interface is more detailed and includes a toolbar with a choice of options and customization settings. The text editor, where users can paste their text, is in the middle of the screen. The results are shown on the left side of the screen. As with Grammarly, the user can simply copy and paste their text into the window. The software scans the text and provides feedback on issues such as grammar, spelling, punctuation, readability, and style. An overall score is provided, as well as scores for each type of error and an overview of the document's readability and style. It also provides a summary of the errors, including the number of errors, the type of errors, and the number of words affected. The results it gives are potentially more comprehensive and detailed than those of Grammarly.
ProWritingAid offers several different pricing options. The free version offers basic features such as in-depth text analysis, style and readability checks, and writing improvement suggestions. The Premium additionally offers unlimited word count, advanced style improvements, customizable writing suggestions, citations, and collaboration. The latest pricing for the Premium version may be confirmed by visiting the ProWritingAid website. The Premium version was used for the analysis.

Results of ProWritingAid Analysis
An analysis of the student text resulted in the finding of six errors. These were: 1. a -a possible wrong determiner 2. power -a missing determiner 3. .(period) -The punctuation mark '.' may require a space after it. Consider adding the space. 4. is -Possible verb agreement error 5. etc. -Consider adding a comma before 'etc' 6. evacuation -Possible missing determiner.
ProWritingAid found the same but fewer issues than Grammarly despite the greater range of and detail of analysis offered. It also missed the same mistakes as Grammarly had, as described in the previous section. The impression given by ProWritingAid was that it was more suited to the analysis of longer passages to which its larger range of check options could better be applied. In comparison to Grammarly, it seemed slightly less capable of finding errors in short texts. The explanations given in the suggestions were less clear and direct than in Grammarly.

ChatGPT
ChatGPT (Generative Pre-trained Transformer) is an AI-powered natural language processing (NLP) platform. It uses a combination of machine learning and natural language processing to generate human-like responses. Its abilities are far beyond that of a grammar checker. It can write blogs, summarize text, write advertisements, and even code. Its range of capabilities has been met with both a sense of apprehension as well as excitement, as mentioned at the beginning of the article. For the purpose of this article, its application in assisting Japanese ELLs with their grammar will be described in comparison with Grammarly and ProWritingAid.
The ChatGPT interface is simple and intuitive. The main page consists of a text box where users can enter their prompts or questions and a 'send' button to activate a response. Chat GPT is available as a free service during times of low demand. It also has a paid version that offers access at times of high demand, faster response speed, and priority access to new features. The free version was used for the student text analysis. Although there were times of high demand when the software could not be used, these periods were short in duration, usually only lasting a few minutes.

Results of ChatGPT Analysis
In analyzing the student text, the following prompt was used, followed by pasting the student text: What grammatical, spelling, or vocabulary mistakes are in the following text?: ChatGPT found seven issues in the student text. The output was as follows: 1. "I experienced a big earthquake" because "I have experienced a big earthquake" should be corrected. The reason is "a" should be used before "big" to indicate that the earthquake was a single, specific event.
2. "when I was an elementary school student" because "when I was an elementary school student" should be corrected. The reason is "an" should be used before a word that starts with a vowel sound (e.g., "an elephant," "an apple"). 3. "it was very dangerous because things around me were falling" because "it was very dangerous that thing that is around me fall in" should be corrected. The reason is "things" is the plural form of "thing" and should be used to describe more than one object, and "was falling" is the correct verb tense to describe events that happened in the past. 4. "cracks in the ground appeared" because "it cracks in the ground" should be corrected. The reason is "appeared" is a more appropriate verb to describe the development of cracks in the ground. 5. "we should equip ourselves with suitable things such as water and preserved foods, etc." because "we should equip suitable thing that is water and preserved foods, etc." should be corrected. The reason is "ourselves" should be included to indicate that the subject of the sentence (we) is performing the action, "things" is the plural form of "thing" and should be used to describe more than one object, and "such as" is a more appropriate conjunction to introduce a list of examples than "that is". 6. "because it is often too late after a disaster occurs" because "because it is too late after the disaster occurs" should be corrected. The reason is "often" is a more appropriate adverb to describe the frequency of events than "always" or "never," and "occurs" is the correct verb tense to describe events that happen in the present. 7. "and there is a possibility of injury if there is a delay in the evacuation" because "and there is a possibility of injury due to delay in the evacuation" should be corrected. The reason is "if" is a more appropriate conjunction to introduce a condition than "due to." The ChatGPT analysis found fewer errors, but this was primarily due to Grammarly and ProWritingAid automatically including formatting and punctuation errors in the initial analysis. A subsequent follow-up prompt, "What are the formatting and punctuation errors in the following" was given to ChatGPT to further analyze the student text, which found additional errors. In order to focus on the main purpose of the article in terms of checking grammar, the formatting and punctuation errors found will not be discussed. The grammar mistakes found were described in much more detail than that of either Grammarly or ProWritingAid. The explanations regarding errors are clearer and direct, making the specific reasons behind why they are considered to be errors much more understandable.
ChatGPT not only provides more detailed explanations but for lower-level Japanese English language learners, it is possible to have the explanations translated by ChatGPT so that the student may easily understand them. The following prompt was used followed by pasting the analysis in English: Translate the following explanations into Japanese, but only translate the explanations so that a Japanese person can understand their mistakes: Below is a translation of the analysis given by ChatGPT: 1.「I have experienced big earthquake」は、「a」を「big」の前に使用することで、地震 が単一の特定のイベントであることを示し、「I experienced a big earthquake」であるべ きです。 2.「when I was a elementary school student」は、母音音から始まる単語の前に「an」を使 用することで、「when I was an elementary school student」であるべきです。 3.「it was very dangerous that thing that is around me fall in」は、「things」が「thing」の複 数形であり、複数のオブジェクトを表すために使用されるべきであり、「were falling」 が過去のイベントを表すのに適した過去形であるため、「it was very dangerous because things around me were falling」であるべきです。 4.「it cracks in the ground」は、「appeared」が地面に割れ目が現れることを表すのに適 した動詞であるため、「cracks in the ground appeared」であるべきです。 5.「we should equip suitable thing that is water and preserved foods etc.」は、「ourselves」を 含むことで、文の主語(we)が行動を行うことを示し、「things」が「thing」の複数形 であり、複数のオブジェクトを表すために使用されるべきであり、「such as」が例を示 すために「that is」よりも適した接続詞であるため、「we should equip ourselves with suitable things such as water and preserved foods, etc.」であるべきです。 6.「because it is too late after the disaster occurs」は、「often」が「always」や「never」よ りも頻度を表すのに適した副詞であり、「occurs」が現在起こっているイベントを表す のに適した現在形であるため、「because it is often too late after a disaster occurs」である べきです。 7.「and there is a possibility of injury due to delay in evacuation」は、「if」が条件を導入す るために「due to」よりも適した接続「and there is a possibility of injury if there is a delay in evacuation」であるべきです。 The subsequent investigation demonstrated that prompts need not only to be in English; they may also be given in Japanese. The prompt '次のテキストの文法、スペル、語彙の間違いは何で すか?' ('What are the grammatical, spelling, and vocabulary mistakes in the following text?') resulted in an English output similar to that described above. When the Japanese prompt was appended with '日本語で答える' ('answer in Japanese'), the output was similar to the translated version previously described.
As ChatGPT requires the use of prompts, teachers should provide students with example prompts, not only to use ChatGPT correctly but also so that all students receive similar types of responses and feedback from the software. Teachers may also ask students to submit the responses received as part of the process of writing approach as well as to see if the responses are appropriate. The teacher may then add any further suggestions to the responses, which the students can use in writing their final submission. After trying the examples, students may experiment with the creation of their own prompts, depending on the requirements of the analysis, such as checking further for formatting, punctuation, and sentence structure.
The fear with the use of ChatGPT may be that students skip entirely the use of the software only for grammatical analysis and have a text written entirely by ChatGPT of which it is fully capable (Jimenez, 2023). This may be prevented by having students submit drafts of their writing before using the software. ChatGPT may be used as part of a process writing approach. This may begin with brainstorming, writing an outline, writing one or more drafts, and then having their draft checked using ChatGPT as a grammar and writing checker. This process writing approach may be part of a portfolio submitted by a student. By ascertaining each of these steps along the process writing approach process, the teacher may be able to see the steps that the student has gone through in reaching their final submission, thereby circumventing cheating in having the final submission written by ChatGPT rather than through their own effort.

Introduction of ChatGPT to Japanese university students
Based on a comparison of ChatGPT with Grammarly and ProWritingAid as grammar checkers and considering the potential use of ChatGPT in a classroom setting, the application was integrated into a paragraph writing assignment at the beginning of the new semester to assist in checking student writing. The participants were first-year Japanese university students.
For the first assignment of the new semester, the students were asked to compose a paragraph. Prior to beginning their writing, the structure of a paragraph was explained, including the importance of a topic sentence, supporting sentences with details, and a concluding sentence. A model paragraph was provided as an example. Next, the students brainstormed ideas for their own paragraphs and created outlines for their topic sentences, supporting sentences with details, and conclusions. The students were then instructed to complete their paragraphs for the next class.
In the subsequent class, students engaged in a peer review activity to assess and give feedback on each other's paragraphs. Following the peer review activity, ChatGPT was introduced to the students as a tool to conduct a further check of their written work. The instructions for the sign-up process were explained to the students. This was followed by how to use prompts to have their paragraphs reviewed. To facilitate this, the following model prompt was given to students to copy and paste: For the following paragraph, 1. Check the spelling, grammar, and format. What spelling, grammar, and formatting mistakes were found? 2. Explain all the spelling, grammar, and formatting mistakes in detail: The students were instructed to copy and paste their paragraph following the above prompt and then click the 'send' button to view the results. The students were told that after viewing the results, they could have them translated to Japanese by copying and pasting the prompt "Translate the following to Japanese," followed by copying and pasting the English explanation and then clicking the 'send' button.

Questionnaire related to the use of ChatGPT to check writing
Students were asked to complete a questionnaire regarding their experience in using ChatGPT to check their writing. The questionnaire was administered ethically by explaining the purpose of the questionnaire. Participation was voluntary. The responses were collected anonymously. The close-ended and open-ended responses are given in the following sections. The results of the questionnaire overwhelmingly indicate that students found ChatGPT easy to use and very helpful in checking their paragraph writing for spelling, grammar, and formatting mistakes. Students were able to find many errors even after a peer check was conducted. Almost all students found the indicated errors to be easy to understand, particularly with the option to have the English explanations translated, which some students did not require or preferred to only use the original English explanation for further English study.

Summary and Conclusion
In summary, Grammarly, ProWritingAid, and ChatGPT all offer the opportunity for ELLs to have their writing checked for grammatical mistakes to improve their writing. Grammarly, ProWritingAid, and ChatGPT can be used to check for errors in grammar, punctuation, and spelling. Grammarly and ProWritingAid give explanations and advice, which is short but helpful, with ProWritingAid being less direct and clear than Grammarly and ChatGPT. The visual interfaces of Grammarly and ChatGPT are straightforward and intuitive. ProWritingAid is a more sophisticated grammar and spelling checker than Grammarly. It has a more complex visual interface, with multiple tabs and menus for different types of checks. It also offers detailed explanations and advice, as well as features such as text comparison and style guides. The use is more complex than Grammarly and English language learners may find it more difficult to pick up. Grammarly is better suited for shorter pieces of writing and ProWritingAid for longer pieces due to its greater range of check options. ChatGPT provides far more detailed and clearer explanations than that of either Grammarly or ProWritingAid. The free versions of Grammarly and ProWritingAid may be adequate for simple checks. With Grammarly, however, before having used the premium version, it was found that requiring login details led to receiving constant emails to upgrade. This led the author to be reluctant to recommend Grammarly to students. It was suggested to students to use an alternative email address where they would not mind receiving spam. There may have been a way to unsubscribe from such emails, but it was felt that it shouldn't have had to reach this level. This was far less of an issue with ProWritingAid, and such email solicitations were never received from OpenAI's ChatGPT.
In conclusion, grammar check software has the potential to help English language ELLs improve their writing skills and become more aware of English conventions. The results of a short questionnaire showed that most students found grammar checkers to be helpful and easy to use. Grammar checkers can be beneficial for both students and teachers. They can help ELLs become more autonomous language learners by allowing them to correct their own mistakes before submission. This allows students to engage in a deeper learning experience while teachers save time by not having to correct basic grammar and spelling mistakes. Grammar checkers can also help teachers provide more meaningful feedback by allowing them to focus more on higher-level elements of writing. In comparing Grammarly, ProWritingAid, and ChatGPT, ChatGPT would be the most highly recommended due to the detail of its analysis and easy-to-understand explanations, as well as the added benefit of translating explanations into Japanese. This would make it particularly ideal for lower-level Japanese ELLs. Based on the comparison results, ChatGPT was introduced to first-year Japanese university students as part of a paragraph writing assignment. Students found ChatGPT to be easy to use and highly effective in finding errors in their paragraph writing assignments.