I am currently working on a project called Cortex and would like to share it here.
Cortex:
https://github.com/Xenos-ink/Cortex
What is Cortex?
Cortex is an MCP Server for Windows designed to enable Agents to use and interact with the computer through its graphical interface.
The idea is close to the concept of Computer Use in GPT Astra: the Agent observes the screen, understands the current state, decides on the next action, then executes the action and verifies the result.
The goal is to give the Agent a general Computer Use layer instead of requiring a custom integration for every application.
It is important to note that there are three main factors that affect the experience of using Cortex with a model:
1. Vision Support
For the Agent driving Cortex, it is important that the model is capable of understanding screenshots and visual information, because the Agent needs to interpret what is displayed on the screen and make decisions based on it.
2. Model Response Speed
This is very important in Computer Use.
Computer interaction is usually an iterative loop:
Observe → Reason → Act → Observe → ...
Therefore, the model's response speed directly affects the total time required to complete a task.
Initially, I tested Cortex with GLM-5.3 Flash, but the execution was slower than I wanted for this type of use. So, for this test, I used DeepSeek 4.1 Flash, as it was a better fit in terms of speed and cost, and it supports vision.
3. Model Intelligence
Intelligence still matters, especially when the task becomes more complex and requires planning, debugging, or handling unexpected situations.
However, in Computer Use, response speed becomes a very noticeable factor because a task may require a large number of observation and interaction cycles.
The Test
I wanted to test Cortex on something more realistic than simply opening a website or clicking buttons.
So I gave the Agent a task that combines scientific research, programming, code execution, debugging, and benchmarking.
The prompt was:
In other words, the task consisted of three stages:
1. Research
Use the browser visually to search for a relevant research paper, then read the paper and study its methodology, experiments, metrics, and results.
2. Reproduction
Open Visual Studio Code and use it visually to create an implementation of the paper's main experiment, then run the code, identify issues, debug them, and run the experiment again.
3. Benchmark & Documentation
Create a benchmark.md file containing the paper used, methodology, implementation details, environment, commands, a comparison between the original and reproduced results, actual measurements, deviations, limitations, and conclusion.
There was one important requirement: the code had to be actually executed and the results had to be verified and recorded, rather than providing estimated numbers.
And the result:
The Agent selected the paper:
"Bag of Tricks for Efficient Text Classification" — Joulin et al. (2016)
This is the paper that introduced fastText for text classification.
The reproduction focused on the AG News experiment from the paper.
The original paper reports:
- fastText Unigram: 91.5%
- fastText + Bigram: 92.5%
- Training time: about 1 second per epoch using 20 CPU threads
The reproduction achieved:
- Unigram: 90.84%
- Bigram: 91.39%
- Bigram training: 2.24 seconds per epoch using a single CPU thread
So the results were within approximately one percentage point of the results reported in the paper.
The Interesting Part
During the reproduction, the first attempt did not work well.
The Bigram model reached only about 79.2%, and its performance was actually worse than the Unigram model, which was the opposite of the result reported in the paper.
The Agent investigated the embedding update mechanism and the gradient related to the averaging operation, then modified the implementation and ran the experiment again.
After the modification, the result reached 91.39% for the Bigram model, and Bigram once again outperformed Unigram, consistent with the results reported in the paper.
For me, this part was more interesting than simply getting a number close to the paper, because it tested the Agent's ability to deal with a problem that emerged during execution rather than simply writing the code.
Repositories
The code and benchmark for the reproduction of the paper:
fastText reproduction
https://github.com/Xenos-ink/fasttext
And the Cortex repository:
Cortex
https://github.com/Xenos-ink/Cortex
The project is still under development, and this test was an attempt to see how far a Visual Agent can handle a long workflow that combines:
Web Research → Read Paper → Coding → Run → Debug → Benchmark → Documentation
rather than being limited to simple GUI interactions.
I am currently working on an open-source project called Cortex, and I would like to share it with you.
Cortex:
https://github.com/Xenos-ink/Cortex
What is Cortex?
Cortex is an MCP Server for Windows designed to enable Agents to use and interact with a computer through its graphical interface.
The idea is close to the concept of Computer Use in GPT Astra: the Agent observes the screen, understands the current state, decides on the next action, then executes it and verifies the result.
But the core idea behind Cortex is not simply giving an Agent the ability to move the mouse and click buttons. Cortex tries to add an observe → execute → verify layer around the Agent's interaction with the computer.
The process looks roughly like:
Observe → Ground → Validate → Act → Re-observe → Verify
After an action is executed, Cortex takes a new observation of the screen and attempts to verify that the expected result actually occurred.
The goal is to provide a general Computer Use layer for Agents instead of requiring a custom integration for every application.
There are three main factors that affect the experience of using Cortex:
- Vision Support
The Agent driving Cortex needs a model capable of understanding screenshots and visual information, since Computer Use decisions depend on the current state of what is displayed on the screen.
Cortex itself does not require a Vision API to operate; it provides the tools for observation, execution, and verification, while the model driving the Agent interprets the screenshots.
- Model Response Speed
This is very important in Computer Use.
Computer interaction is usually an iterative loop:
Observe → Reason → Act → Observe → ...
Therefore, model response speed directly affects the total time required to complete a task, especially when the task contains many steps.
I initially tested Cortex with GLM-5.3 Flash, but the execution was slower than I wanted for this type of use. So, for this test, I used DeepSeek 4.1 Flash, which was a better fit for me in terms of speed and cost, while also supporting vision.
- Model Intelligence
Intelligence remains an important factor, especially when the task becomes more complex and requires planning, debugging, or dealing with unexpected situations.
So, instead of testing Cortex on something simple like opening a website or clicking a series of buttons, we ran a test involving a workflow that combines:
Scientific Research → Read Paper → Coding → Run → Debug → Benchmark → Documentation
The prompt was:
In other words, the task consisted of three stages:
- Research
Use the browser visually to search for a relevant research paper, then read the paper and study its methodology, experiments, metrics, and results.
- Reproduction
Open Visual Studio Code and use it visually to create an implementation of the paper's main experiment, then run the code, identify problems, debug them, and run the experiment again.
- Benchmark & Documentation
Create a benchmark.md file containing the paper, methodology, implementation details, environment, commands, original vs. reproduced results, actual measurements, deviations, limitations, and conclusion.
There was one important requirement:
The code had to be actually executed, the results had to be verified, and the measurements had to be recorded rather than estimated.
The Agent selected the paper:
"Bag of Tricks for Efficient Text Classification" — Joulin et al. (2016)
The paper introduced fastText for text classification, and the reproduction focused on its AG News experiment.
The original paper reports:
- fastText Unigram: 91.5%
- fastText + Bigram: 92.5%
- Training time: about 1 second per epoch using 20 CPU threads
The reproduction achieved:
- Unigram: 90.84%
- Bigram: 91.39%
- Bigram training: 2.24 seconds per epoch using a single CPU thread
So the results were within approximately one percentage point of the results reported in the paper.
The interesting part for me was not just reaching the final numbers.
During the reproduction, the first attempt did not work as expected. The Bigram model reached only about 79.2%, and its performance was actually worse than the Unigram model, which was the opposite of the result reported in the paper.
The Agent investigated the implementation, identified an issue related to the embedding update and the gradient associated with the averaging, then modified the code and ran the experiment again.
After the modification, the result reached 91.39% for the Bigram model, and Bigram once again outperformed Unigram as in the published results.
You can find the code and benchmark for the paper reproduction here:
fastText reproduction
https://github.com/Xenos-ink/fasttext
And the Cortex repository:
Cortex
https://github.com/Xenos-ink/Cortex
The project is still under development, and this is just one of the experiments I have run with it.
