AI agents that operate computers often work by analyzing screenshots, deciding where to click, and then controlling the mouse and keyboard. This approach has a cost: the screen must be captured frequently and fed into a multimodal model, which consumes a large number of tokens. For example, if a 1920×1080 screen is analyzed at every step, even changing a single cell in a spreadsheet can take many operations, such as selecting menus and moving between screens. Rendering delays and slight coordinate errors can also cause a task to fail.
Shell commands and scripts alone, on the other hand, struggle with adjusting on-screen layouts or editing dynamically changing drawing canvases. To combine GUI operation with code execution, earlier efforts therefore used external routers or multiple specialized agents. But these setups add calls and handoffs between models, which creates problems with latency and with carrying context across.
Mage-VL-Cowork, released by a Microsoft research team, takes a different approach to this problem. It trains a relatively small 4-billion-parameter (4B) vision-language model (VLM) on both screen operation and terminal operation, and lets the model itself decide which tool to use in each situation. The model weights are available on Hugging Face under the Apache-2.0 license. AI Tracker, which monitors the release of AI models, recorded that it detected the public weights at 8:20 a.m. US Eastern Daylight Time on October 30, 2026. This is the time an external service detected them, however, and does not indicate when Microsoft formally announced the release.
What matters in this research is not only that a small model was given two ways to operate a computer. The team reports that it significantly improved task success rates and efficiency through the way it taught the model which tool to use and when, and through the design of the environment that generates the training data.
A single model switches between GUI operation and shell execution
The defining feature of Mage-VL-Cowork is that the choice between the graphical user interface (GUI) and the command-line interface (CLI) is made by a single model rather than delegated to multiple external agents. Depending on the task, the model selects one of two channels: computer_use, which operates the screen directly, and bash, which executes shell commands. It uses the GUI when visual judgment is needed, and the terminal for work suited to direct file editing or precise verification.
| Channel | Main actions and processing | Information received as a result | Main uses |
|---|---|---|---|
computer_use |
Click, double-click, triple-click, mouse move, drag, text input, key press, scroll, wait, screen capture, terminate (11 types in total) | Screenshot after the action | Checking screen layout, canvas editing, dynamic UI operation |
bash |
Arbitrary shell commands, Python scripts, and execution of libraries such as openpyxl and pandas |
Text combining standard output (stdout) and standard error (stderr) | Editing documents and spreadsheet files, calculation, verifying results |
The GUI channel provides 11 basic actions. Coordinates used for clicks, drags, and similar actions are normalized to a range of 0 to 999 both horizontally and vertically, and the execution environment converts them to the actual screen resolution, such as 1920×1080. After an action finishes, the next screenshot is returned, and the model checks how the screen changed before deciding its next move.
The bash channel, by contrast, can run shell commands and Python scripts inside a working directory. Results come back as text combining standard output and standard error, so the entire screen does not need to be analyzed as an image. Checking file contents, bulk replacement, calculation with libraries, and process monitoring can all be handled without going through GUI operations.
Microsoft explains that the GUI is suited to checking visual layout, editing canvases in graphics software, and grasping the dynamic display state of applications. For structural changes to spreadsheets, documents, and presentation files, it says, direct editing with Python libraries is more efficient. Being able to verify programmatically that the modified file has the correct contents is another advantage of using the terminal.
Consider, for example, merging specific cells in a spreadsheet file and setting a heading. A GUI-only agent has to launch LibreOffice Calc, wait for the screen to appear, select the target cells, and change the formatting. If the toolbar layout changes along the way or an unexpected pop-up appears, the operation can fail. Mage-VL-Cowork, by contrast, can run a Python script that calls openpyxl from bash when it judges that the file can be edited directly, and save the change without touching the GUI. There is no need to wait for screen transitions, and the number of steps drops.
The aim of this design is not to reproduce every operation through the GUI, but to make active use of the programmatic means of control a computer already offers. Simply giving a model two kinds of tools, however, does not mean it will learn to use them appropriately. A small VLM may call the shell in the wrong situation, or go back and forth between the GUI and the terminal more than necessary. Microsoft tried to solve this through the model's training method and the design of its training environment.
A five-stage curriculum and improved teacher environments drive success rates
What Microsoft emphasized in developing Mage-VL-Cowork was not only the model architecture but also how the training data is generated and the order of learning. The base is "Mage-VL," an intermediate 4B-parameter checkpoint. Rather than teaching it both GUI and terminal use from the start, the team adopted a five-stage curriculum called "Specialize-then-Unify," in which each capability is acquired first and then integrated.
| Stage | Curriculum | Main learning content | Compute |
|---|---|---|---|
| Stage 1 | GUI-Oriented Midtraining | Visual grounding, reasoning process, adherence to tool formats | 1.18 ZFLOPs |
| Stage 2 | GUI Agent SFT | Multi-step screen operation with mouse and keyboard | 1.55 ZFLOPs |
| Stage 3 | Coding & Terminal SFT | Software development, file and system operations via scripts | 0.27 ZFLOPs |
| Stage 4 | Hybrid CUA SFT | Ability to switch between computer_use and bash on each turn |
2.17 ZFLOPs |
| Stage 5 | Online Hybrid RL | Optimizing success rate and operational efficiency with GRPO using verifiable rewards | 2.62 ZFLOPs |
In Stage 1, using datasets such as OSWorld-G and ScreenSpot-v2, the model learns the basic ability to match on-screen elements to text instructions correctly. In Stage 2, it acquires multi-step desktop and web operation, performed while looking at screenshots and controlling the mouse and keyboard, through supervised fine-tuning (SFT). Stage 3 teaches it to use the terminal, centered on software development and file operations through programs. Only in Stage 4 are the GUI and terminal integrated, with training on which channel to use on each turn. In the final Stage 5, online reinforcement learning (RL) with verifiable rewards raises the success rate while reducing the number of operations.
To generate the data needed for this training efficiently, the research team also developed an environment generation engine called "Mage-CUA-Forge." The conventional method, in which an AI agent is set loose to explore through trial and error each time a task is generated, requires large amounts of tokens and compute. Mage-CUA-Forge instead organizes individual operations into reusable basic skills and predefines task-generation scripts and environment setup procedures.
According to Microsoft, this mechanism cut token consumption per task by 93.3%, from 1,276,000 tokens to 85,000. The cost of repeatedly generating tasks was also reduced by 82.6%, from $0.906 to $0.158. In the end, 293,806 tasks were synthesized using 6,915 basic skills, and 318,837 verified hybrid operation trajectories were collected.
An even more notable result is that differences in the tool environment given to the teacher model strongly affected the performance of the small model after training. When distilling knowledge from the large teacher model "GPT-5.4," Microsoft compared two execution environments with different tool configurations.
In one environment (), the teacher model was given many sources of information and tools, including GUI operation, code execution, and an accessibility tree. In the other (), overlapping means of operation such as the accessibility tree and xdotool were removed, and practical libraries such as openpyxl and pandas were preinstalled instead. In a comparison on 672 composite tasks, the teacher model's own success rate rose from 89.6% to 92.0%, and the average number of operation steps fell 29.2%, from 35.33 to 25.00.
The difference in the teacher's own success rate was 2.4 points, but the small models trained on its operation histories showed a larger gap. The model trained on data generated in the tool-heavy, complex environment had a success rate of 25.1% and averaged 39.40 steps. The model trained on data generated in the streamlined tool environment recorded a 37.6% success rate and averaged 30.91 steps. That is a 12.5-point difference in success rate.
This result suggests that building an environment in which the teacher model can use tools efficiently and consistently may matter more than giving the small model many tools. These results were obtained in Microsoft's experimental environment, however. Whether streamlining the tool environment would bring a similar improvement with different teacher models or datasets awaits future verification.
66.0% success on OSWorld-Verified, ahead of some larger models
One reason Mage-VL-Cowork is drawing attention is that it recorded a 66.0% success rate on the desktop-operation benchmark OSWorld-Verified. Microsoft reports that, despite being a relatively small model with 4 billion parameters, it outperformed many larger models. When evaluating this figure, though, it is important to pay attention to the measurement conditions, such as the task composition and the cap on the number of operations.
The success rate Microsoft published is precisely 66.04%, measured on 361 tasks after excluding those related to Google Drive. Each task was limited to a maximum of 50 steps, and the reported figure is the average of 65.33% and 66.75% from two independent runs. Evaluation used a dedicated agent execution environment, "MAGEHybridAgent," which retains recent operation history and three screenshots. The settings were a 1920×1080 screen resolution, a sampling temperature of 0.7, top_p of 0.8, and a maximum of 4,096 generated tokens.
データを表で見る
| 成功率 (%) | |
|---|---|
| ScaleCUA-9B | 68.7 |
| Mage-VL-Cowork-4B | 66 |
| Muse Glimmer-30B | 65.9 |
| Qwen3.6-27B | 63.9 |
| EvoCUA-32B | 56.7 |
| ComputerRL-9B | 48 |
| OpenCUA-72B | 44.9 |
| Fara1.5-4B | 13.9 |
In the comparison Microsoft presented, Mage-VL-Cowork posts a high success rate among models whose weights are public and whose parameter counts are known. In particular, it far exceeded the 13.9% of "Fara1.5-4B," a GUI-specialized model of the same 4B size. It did not, however, take the top spot among all open-weight models. "ScaleCUA-Qwen3.5-9B" recorded 68.7% under the same 50-step limit.
Mage-VL-Cowork's 66.0% also exceeds Muse Glimmer-30B's 65.9% and Qwen3.6-27B's 63.9%, but Microsoft's comparison table does not list detailed evaluation conditions, such as step limits, for these larger models. Because it is not certain that all models were evaluated under exactly the same conditions, one cannot simply conclude that a model with fewer parameters is superior.
Comparing Mage-VL-Cowork's own training stages, meanwhile, shows the improvement brought by online reinforcement learning. Microsoft compares the success rate and inference cost of a GUI-specialized SFT model, an SFT model integrating GUI and terminal, and the final model that went through reinforcement learning.
| Development stage / reference model | OSWorld-Verified success rate (%) | Avg. model calls per task | Avg. output tokens per task | Avg. thinking tokens per task |
|---|---|---|---|---|
| Stage 2: GUI Agent SFT | 45.56 | 25.36 | 6,401 | 4,549 |
| Stage 4: Hybrid CUA SFT | 55.84 | 21.57 | 7,374 | 3,328 |
| Stage 5: Online Hybrid RL (Mage-VL-Cowork) | 66.04 | 18.77 | 5,499 | 2,482 |
| Reference: GPT-5.4-mini (xhigh) | 69.57 | 18.24 | 12,397 | 11,631 |
Stage 4, which integrates the GUI and terminal, had a success rate of 55.84%, which rose to 66.04% in Stage 5 after online reinforcement learning. That is an improvement of 10.20 points, and the average number of model calls also fell by about 13.0%, from 21.57 to 18.77. Average output tokens per task were cut by about 25.4%, from 7,374 to 5,499. Thinking tokens inside the <think> tags, where the model writes out its reasoning, also dropped from 3,328 to 2,482, showing that the model reduced the tokens needed for reasoning while raising its success rate.
Microsoft's project page also introduces a demo comparing execution time with Qwen3.6-27B. On a particular task using LibreOffice Calc, the GUI-specialized Qwen3.6-27B took 39 steps and 9 minutes 56 seconds, whereas Mage-VL-Cowork chose to edit the file directly with Python and finished in 6 steps and 39 seconds. That is a difference of about 15.3 times in processing time, a clear example of the advantage of switching between the GUI and the terminal. However, this is a demonstration of one specific task, and not every job will see a similar speedup.
Requirements and memory use for the released model
The weights of Mage-VL-Cowork are public, but running the model in practice requires meeting conditions related to its structure and inference environment. In the repository, the bfloat16 weights are split across two safetensors files, with a total file size of 9.48 GB. This size alone, however, is not enough to conclude that the model can easily run on a home PC, smartphone, or edge device.
At inference time, in addition to the memory needed to load the model weights, memory is required for the KV cache, the working space of the vision encoder that processes high-resolution images, and the execution environment and OS. Actual memory usage is therefore larger than the weight file size. To judge whether the model can run on small devices, one would need to measure the memory savings from quantization and the memory consumed by the full inference process.
The model architecture also deserves attention. Mage-VL-Cowork uses a custom multimodal architecture called llava_onevision2, with the model structure defined in Python code bundled with the repository. As a result, loading it with Hugging Face Transformers or similar tools requires specifying trust_remote_code=True. It is not a configuration where the weights can simply be swapped into a standard Qwen3-VL implementation. This setting also permits execution of the repository's Python code, so the trustworthiness of that code needs to be checked.
The context length used for training and inference is set to 32,768 tokens. The architecture is said to support up to 262,144 tokens, but this should be distinguished from the operating conditions evaluated in the released configuration. To limit context consumption on long tasks, Mage-VL-Cowork also uses a history compression method called "asymmetric working memory."
| History type | Target steps | Information retained | Purpose |
|---|---|---|---|
| Compressed history area | Past turns (steps 1 to $t-4$) | Aggregates the one-line action summaries (Action: lines) output by the model |
Limits token consumption from old history |
| Active working area | Recent turns (steps $t-3$ to $t$) | The three most recent screenshots and the full assistant responses from the last four turns | Checks screen changes and execution results right after an action |
In this scheme, the three most recent screenshots and four turns of assistant responses are kept as they are. This lets the model confirm whether the previous click succeeded and how the screen state changed. Earlier history, meanwhile, is compressed to only the Action: lines the model itself output, and placed together right after the system prompt. By not continuing to hold old screenshots and detailed responses, the design keeps context consumption down even during long operations.
For the inference environment, Microsoft recommends building an SGLang runtime using the official Docker image. The recommended setup is a GPU environment combining SGLang 0.5.2, PyTorch 2.8.0+cu128, Transformers 5.7.0, and FlashInfer 0.3.1. To recognize fine text and icons on the desktop screen correctly, the configuration shown sets the environment variable SGLANG_MAX_PIXELS=4000000 and processes at the full 1920×1080 resolution. The model card also lists an installation requirement of transformers>=4.57.3, but if benchmark reproducibility is a priority, it is appropriate to use the Docker environment Microsoft specifies as the baseline.
The repository's osworld_eval directory contains the minimal reference implementation needed for evaluation, including the agent definition and the SGLang launch script. To run the complete benchmark evaluation pipeline, however, it must be combined with the set of scripts in the mage_vl_cowork directory of the official GitHub repository.
Thus, the 66.0% success rate was not obtained from the model weights alone, but from a combination of a dedicated agent execution environment, the history retention method, the cap on operations, and the sampling settings. In evaluating the real capability of a small model, the design of the whole system that supports actual operation matters as well, not just the parameter count or weight file size. Whether a similar success rate can be maintained when the model is moved to another inference engine or agent framework will be a question for future verification.
Security challenges of arbitrary shell execution
When considering the practicality of Mage-VL-Cowork, security matters as much as performance. Combining GUI operation with shell execution broadens the range of work, but it also enlarges the authority of what the model can do. In the bash channel in particular, the AI can generate and run arbitrary shell commands, so a wrong judgment or a malicious instruction from outside could lead to destroyed files or leaked information.
Even a GUI-only agent can change important settings or data through applications. But when a shell is available, the file system, processes, network, and more can be manipulated directly without going through the GUI. If the execution environment is given excessive privileges, the effects of an unintended command could spread widely.
On its official GitHub repository, Microsoft positions this family of models as "for research purposes only." It says they are not intended for deployment in commercial products or production services. The safe-operation guidelines in the model card likewise assume research and evaluation in isolated environments, such as disposable Docker containers or virtual machines that can be restored from snapshots, rather than running the model without restriction on a PC used day to day.
Microsoft recommends that the evaluation environment have no access to the host file system, the local network, or personal environment variables. Credentials such as SSH private keys, cloud service access tokens, and browser cookies should also not be carelessly passed into the working environment. Even when credentials are needed for verification, they should be limited to temporary ones dedicated to the sandbox.
For operations with serious consequences, such as sending data externally, account authentication, moving money, or changing permissions, operation requires explicit human approval before execution. When loading external web pages or documents, one must also watch for indirect prompt injection, in which malicious instructions are embedded. To prevent unintended command execution or information transmission, restricting communication destinations to permitted addresses is also recommended.
Falls short of specialized models on web browser tasks
Mage-VL-Cowork is designed for general desktop operation, but it does not outperform specialized models on every task. On "WebVoyager-Static," an offline benchmark measuring web browser performance, Microsoft reports a success rate of 58.6% for "Mage-VL-Cowork-SFT," the checkpoint before reinforcement learning. The following summarizes the comparison Microsoft presented.
| Model | Parameters | Target domain | WebVoyager-Static success rate (%) | Live-environment success rate reported by each company (%) | Gap from live environment (points) |
|---|---|---|---|---|---|
| Fara1.5-4B | 4B | Web-browser specialized | 68.0 | 80.8 | −12.8 |
| Mage-VL-Cowork-SFT | 4B | General desktop / web browser | 58.6 | Not published | Not published |
| Qwen3-VL-8B-Thinking | 8B | General-purpose VLM | 53.7 | Not published | Not published |
| Qwen3.5-9B | 9B | General-purpose VLM | 53.3 | Not published | Not published |
| Qwen3-VL-4B-Thinking | 4B | General-purpose VLM | 49.6 | Not published | Not published |
| OpenWebRL-4B | 4B | Web-browser specialized | 47.1 | 74.1 | −27.0 |
| MolmoWeb-8B | 8B | Web-browser specialized | 46.8 | 78.2 | −31.4 |
Mage-VL-Cowork-SFT's 58.6% falls short of the 68.0% of Fara1.5-4B, which specializes in web browser operation. Meanwhile, Fara1.5-4B's success rate on OSWorld-Verified is only 13.9%, showing that models specialized in web browsers and models handling general desktop operation have different strengths. However, the 58.6% on WebVoyager-Static is a result for the pre-reinforcement-learning SFT model, and differs in both what is measured and the model's training stage from the 66.0% the final model recorded on OSWorld-Verified. The two cannot be directly compared to judge the performance gap between web and desktop operation.
The differences in operable scope and evaluation conditions among representative models can be summarized as follows.
| Model | Parameters | Operable scope | Main benchmark results | Evaluation conditions and notes | Source |
|---|---|---|---|---|---|
| Mage-VL-Cowork | 4B | Hybrid of 11 GUI actions and bash |
OSWorld-Verified: 66.0% | Max 50 steps, average of 2 runs, MAGEHybridAgent used, Google Drive-related tasks excluded | Microsoft model card |
| ScaleCUA-Qwen3.5-9B | 9B | GUI and various tools | OSWorld-Verified: 68.7% | Max 50 steps, top record among open-weight models | Microsoft comparison table |
| Qwen3.6-27B | 27B | GUI operation | OSWorld-Verified: 63.9% | Microsoft's comparison table does not state a step limit, so direct comparison requires caution | Microsoft comparison table |
| Fara1.5-4B | 4B | Web browser operation (DOM and screenshots) | OSWorld-Verified: 13.9%, WebVoyager-Static: 68.0% | Success rates differ greatly between general desktop and web browser operation | Microsoft comparison table |
Given these results, the distinguishing feature of Mage-VL-Cowork lies less in peak performance on any particular operation than in its ability to handle a broad range of work by combining the GUI with code execution. For tasks where direct programmatic operation is effective, such as spreadsheet aggregation, report generation, and bulk file format conversion, the advantages of the bash channel are especially easy to exploit. Given the safe-operation conditions Microsoft sets out, though, even for such uses it is appropriate to start by verifying in an evaluation environment isolated from production data and external networks. Using disposable containers or virtual machines and restricting file access permissions and communication destinations can reduce the risks that come with shell execution.
It is also necessary to distinguish between the weights being public under the Apache-2.0 license and Microsoft's intention that the model be used for research. Apache-2.0 is a relatively permissive license, but it does not guarantee that the model can be operated safely in a production environment. Whether outside researchers and developers can evaluate it in independent environments and reproduce high success rates on different tasks and execution platforms will be an important basis for judgment going forward.
What the Mage-VL-Cowork research shows is that the performance of computer-operating AI agents is not determined by model size alone. By combining a training method that appropriately switches between the GUI and terminal, high-quality teacher data, and efficient history management, it suggests that even a 4-billion-parameter model may be able to handle complex desktop operation. At the same time, a mechanism that can execute arbitrary shell commands requires strict permission management and sandbox isolation. Whether independent environments reproduce its performance and verify its safety will shape whether small hybrid agents become practical.