Claude API

Si-Yuan Hu, Ming-Yu Ouyang, Di-Fei Gao, M. Shou

The recently released model, Claude 3.5 Computer Use, stands out as the first frontier AI model to offer computer use in public beta as a graphical user interface (GUI) agent. As an early beta, its capability in the real-world complex environment remains unknown. In this case study to explore Claude 3.5 Computer Use, we curate and organize a collection of carefully designed tasks spanning a variety of domains and software. Observations from these cases demonstrate Claude 3.5 Computer Use's unprecedented ability in end-to-end language to desktop actions. Along with this study, we provide an out-of-the-box agent framework for deploying API-based GUI automation models with easy implementation. Our case studies aim to showcase a groundwork of capabilities and limitations of Claude 3.5 Computer Use with detailed analyses and bring to the fore questions about planning, action, and critic, which must be considered for future improvement. We hope this preliminary exploration will inspire future research into the GUI agent community. All the test cases in the paper can be tried through the project: https://github.com/showlab/computer_use_ootb.

Nana Reinikainen, Mika V. Mäntylä, Yu-Qing Wang

Assessing the effectiveness of REST API tests in black-box settings can be challenging due to the lack of access to source code coverage metrics and polyglot tech stack. We propose three metrics for capturing average, minimum, and maximum log coverage to handle the diverse test generation results and runtime behaviors over multiple runs. Using log coverage, we empirically evaluate three REST API test generation strategies, Evolutionary computing (EvoMaster v5.0.2), LLMs (Claude Opus 4.6 and GPT-5.2-Codex), and human-written Locust load tests, on Light-OAuth2 authorization microservice system. On average, Claude Opus 4.6 tests uncover 28.4% more unique log templates than human-written tests, whereas EvoMaster and GPT-5.2-Codex find 26.1% and 38.6% fewer, respectively. Next, we analyze combined log coverage to assess complementarity between strategies. Combining human-written tests with Claude Opus 4.6 tests increases total observed log coverage by 78.4% and 38.9% in human-written and Claude tests respectively. When combining Locust tests with EvoMaster the same increases are 30.7% and 76.9% and when using GPT-5.2-Codex 26.1% and 105.6%. This means that the generation strategies exercise largely distinct runtime behaviors. Our future work includes extending our study to multiple systems.

Tio Haidar Hanif, Dana Susilo Kusumo, Nungki Selviandro

The use of AI has become widespread, and one area that has been significantly impacted is application development. GitHub Copilot is one of the AI-based tools that can simplify application development and improve efficiency in the development process. However, the efficiency of GitHub Copilot also depends on the AI model it uses. This study aims to quantitatively compare the efficiency of the GPT-4.1 and Claude 3.7 Sonnet AI models using GitHub Copilot in Agent Mode, with a case study focused on developing a backend REST API application. The study employs a comparative approach by conducting an experiment to develop several features in parallel using each model. The metrics measured are the number of prompts and the duration required by each prompt to complete each task. The conclusion of this study shows a correlation between generation speed and the model used for code generation. In this research, GPT 4.1 demonstrates higher efficiency, as seen in the shorter duration required by the model and the fewer prompts required. This model is appropriate for tasks that are relatively simple or for generating code incrementally rather than all at once. However, Claude 3.7 is more effective in handling larger code generation requests in a single prompt. These models are suited for complex tasks and for generating the complete code at once.

Sang-Min Park, Jiwon Park, Ho-Joong Kim, Jin S. Yeom

Purpose: This study evaluates the performance of Claude and GPT LLM Vision APIs for automated clinical questionnaire processing in spine surgery by comparing accuracy, efficiency, reproducibility, and cost-effectiveness.Methods: Clinical questionnaires from 56 patients (336 total pages) were processed using a Python 3.12-based system incorporating PDF preprocessing, image enhancement via OpenCV, and direct LLM Vision analysis. Both models were evaluated on 26 questionnaire items (1,456 data points) using accuracy comparison, processing time measurement, token utilization analysis, and intra-class correlation coefficient (ICC) assessment through three independent iterations.Results: GPT achieved 98.83% accuracy (1,439/1,456) compared to Claude's 97.94% (1,426/1,456). Both models processed questionnaires in 27 seconds per set, representing 68% time reduction versus manual entry (85 seconds). GPT demonstrated 59% cost advantage ($0.023 vs. $0.056 per questionnaire), while Claude showed superior reproducibility (ICC 0.98 vs. 0.96). GPT achieved 100% accuracy across 21 items versus Claude's 17 items. Error analysis identified predominantly handwriting recognition (52%) and image quality issues (28%), with 89% of errors successfully flagged for review.Conclusions: Both models achieve clinical-grade performance exceeding 90% accuracy. GPT demonstrates superior accuracy and cost-effectiveness, while Claude provides better reproducibility. Model selection should be guided by institutional priorities regarding accuracy, reproducibility, and operational scale.

Pratik Dinkar Rane

Enterprise and healthcare software systems demand rigorous, compliance-aware testing across both API and user interface layers. Conventional test automation toolchains treat API and UI testing as separate disciplines, forcing organizations to maintain parallel frameworks, duplicate test logic, and reconcile compliance evidence from disparate sources. This fragmentation imposes compounding overhead in regulated environments governed by HIPAA, HL7 FHIR R4, and FDA 21 CFR Part 11. The cost of maintaining disjoint toolchains is particularly acute in healthcare information technology, where test failures may carry patient safety implications and every audit trail gap constitutes a regulatory risk. This paper introduces ClinQA, a large language model orchestrated framework that unifies API and UI test generation through a single compliance-aware pipeline. ClinQA uses a structured prompt schema with five components, which it sends to Claude, an instruction-following language model. It then routes the generated test artifacts to platform-specific adapter modules for Playwright-based UI execution and consumer-driven contract testing for API validation. A four-layer architecture separates intent capture, test generation, execution, and compliance reporting into independently maintainable components. The framework embeds PHI masking, FHIR conformance evidence capture, and 21 CFR Part 11 audit trail emission at the infrastructure level rather than delegating compliance responsibility to individual test authors. Preliminary observations from 12 scenarios spanning Salesforce CRM workflows and FHIR R4 conformance testing indicate a Cross-Layer Coverage Index of 87%, a Test Authoring Effort Reduction of 68% compared to manual authoring, and a Compliance Coverage Score of 94% against active compliance profiles. ClinQA demonstrates that unified LLM-orchestrated test generation is a feasible approach for regulated enterprise and healthcare environments, and it establishes a foundation for controlled empirical validation at production scale.