"what's on screen", "show me"?
screenshot
← Skill registry
Developer workflowsPublic skills
Computer Use
Desktop automation for AI agents — screenshot, click, type, scroll, accessibility trees, form filling, menu navigation, scripts (AppleScript/JXA/PowerShell), app/window management. 49 tools.
What this skill teaches
A reusable playbook for a specific kind of work.
An MCP server tells an agent which actions are available. A skill adds the judgment around those actions: how to recognize the job, which sequence to follow, what to avoid, and how to decide that the result is complete.
Orchestrate desktop computer control — take screenshots, click/type/scroll, manage windows and apps, read accessibility trees, fill forms, run scripts (AppleScript/JXA/PowerShell), manage clipboard, and navigate UI elements. Use when automating desktop tasks, clicking buttons, filling forms, reading screen content,…
Architecture
How Computer Use guides an ADK-Rust agent.
The skill stays readable and portable because it contains instructions rather than service credentials or business data. ADK-Rust supplies it to the agent, the agent chooses from its reviewed tool boundary, and the connected MCP server performs the authenticated operation.
01
User request
The agent receives a goal expressed in ordinary language.
02
Computer Use skill
Matches intent, supplies the decision guide, and narrows the tool boundary.
03
ADK-Rust agent
Plans the workflow and streams each meaningful step through the runtime.
04
See compatibility notes
Executes authenticated operations against the system that owns the capability.
05
Verified result
The skill's completion rules shape the evidence returned to the user.
Decision guide
How the agent turns a request into the right action.
These routes come directly from the skill instructions. They help the model recognize intent and select a focused tool or workflow instead of improvising across the entire capability surface.
"click", "press button"?
click_element / left_click
"type", "enter text"?
type / set_value
"fill form"?
fill_form
"open app"?
open_application
"menu", "File Save"?
list_menu_bar
"copy", "paste", "clipboard"?
read_clipboard / write_clipboard / key("command+c")
"scroll"?
scroll
Proven workflows
Repeatable sequences for useful outcomes.
A workflow joins several tool calls into a task the user actually recognizes. The skill explains the sequence and the intended result while ADK-Rust streams the agent's progress through the shared runtime.
013 calls
See → Click → Verify
Screenshot, interact, confirm
022 calls
Fill Form
Find fields → fill all
032 calls
Menu Action
Find shortcut → press key
041 calls
Script Automation
Complex AppleScript/PowerShell
052 calls
App Management
List → activate
Tool boundary
The skill may use 50 documented tools.
This allowlist is declared by the skill. It keeps the agent focused on the actions needed for this job while See compatibility notes retains responsibility for authentication, validation, and the connected system.
screenshot
zoom
doctor
policy_status
agent_pointer
openai_computer
left_click
right_click
middle_click
double_click
triple_click
mouse_move
left_click_drag
left_mouse_down
left_mouse_up
scroll
cursor_position
type
key
hold_key
read_clipboard
write_clipboard
open_application
list_running_apps
hide_app
unhide_app
get_window
get_cursor_window
list_windows
get_frontmost_app
activate_app
activate_window
get_display_size
list_displays
wait
get_ui_tree
get_focused_element
find_element
click_element
set_value
press_button
list_menu_bar
select_menu_item
fill_form
run_script
get_app_dictionary
get_tool_guide
get_app_capabilities
list_spaces
get_active_space
Working rules
What the agent should do.
- Always screenshot before clicking (verify target is visible)
- Prefer click_element/press_button over coordinate clicks (more reliable)
- Use get_tool_guide before complex multi-step tasks
- Use list_menu_bar to find keyboard shortcuts (faster than menu navigation)
- Verify actions with follow-up screenshot
Operating boundaries
What the agent should avoid.
- Don't click without seeing the screen first
- Don't use coordinate clicks when accessibility elements are available
- Don't type into wrong field (check get_focused_element first)
- Don't run destructive scripts without user confirmation
- Don't interact with apps outside the allow list (check policy_status)
Install and connect
Add the skill beside the capability it expects.
Install the repository where your ADK-Rust skill loader can discover it, connect See compatibility notes, and confirm the declared tools are available before asking the agent to use the workflow.
Install the skill
git clone https://github.com/zavora-ai/skill-computer-use.git \
~/.skills/skills/computer-useADK-Rust loading shape
let skills = SkillLoader::from_dir("~/.skills/skills").await?;
let skill = skills.load("computer-use").await?;
let agent = LlmAgentBuilder::new("agent")
.instruction(skill.instructions())
.tools(skill.allowed_tools(toolset)?)
.build()?;The repository's compatibility statement: Requires computer-use-mcp server. macOS (AppleScript/JXA) and Windows (PowerShell). Accessibility permissions required.
Official documentation
Read the complete skill package.
The repository remains authoritative for its exact instructions, examples, helper scripts, assets, MCP requirements, license, and later updates.
Keep exploring
More in developer workflows.
Browser AutomationWeb operations for AI agents — read pages, extract data, take screenshots, interact with forms, and scrape structured content via headless Chromium.17 allowed toolsCI/CD Pipeline ManagementPipeline operations for AI agents — monitor builds, debug failures from logs, trigger safe deployments, and manage release gates via mcp-cicd.8 allowed toolsCode Search & NavigationCode intelligence for AI agents — semantic search, symbol lookup, call graphs, dependency mapping, blast radius analysis, and ownership resolution via mcp-code-search.8 allowed tools
Source record
Repository metadata for this skill entry.
- License
- Apache-2.0
- Allowed tools
- 50
- References
- 3
- Revision
- d8fb7beeee4c
- Updated
- May 31, 2026