BenchmarksExplore on Hugging Face

MECHANICAL CAD / PHYSICS

PhyCAD-Bench

Evaluating model physics reasoning through mechanical CAD edits.

Models edit a starting CadQuery solid. The grader exports the result to STEP, runs the task simulation, and checks the goal and constraints.

Leaderboard Evaluation protocol
50runnable tasks
12evaluated models
1attempt per task
72%highest observed pass rate

RECORDED RESULTS / PHYCAD-BENCH-V0.2

PhyCAD-Bench leaderboard

Each model has one recorded attempt per task. A pass requires a valid STEP export and successful task-specific goal and constraint checks.

RANKMODELPASS RATEAPI COST

AGGREGATE RESULTS

Results by task family and cost

6 tasks had no recorded pass across the evaluated models.

FIGURE 1

Pass rate by task family

Motion and contact60%
Structural response43%
Thermal response56%
Electric fields58%
Terrain and vehicles43%
Robotics28%
Pass rate is the share of recorded model-task pairs that passed in each family.
FIGURE 2

Pass rate against API cost

Scroll chart sideways to see all models.

0%20%40%60%80%$0$5$10$15$20Claude Opus 5.5: 72%, $6.03GPT-6.1 Sol: 70%, $2.00GPT-6 Astra: 68%, $9.89Claude Opus 5: 62%, $9.38Muse Wonder: 62%, $7.03Gemini 4 Argon: 60%, $2.24Grok 4.7: 60%, $6.12Claude Fable 5.1: 56%, $17.59Kimi K3.1 Pro: 48%, $2.21GLM-5.3: 22%, $0.75Qwen3.8 Max: 20%, $4.12DeepSeek V4.1 Flash: 16%, $0.36Claude Opus 5.5Reported API cost (USD)
Reported API cost excludes local CAD and simulation compute. Select a point to inspect its runs.

TASK-LEVEL RESULTS

Recorded task outcomes

Select a model and task to inspect the recorded stage, failed checks, token use, and API cost.

SELECTED MODEL

Claude Opus 5.5

72%36 / 50 passed
1,306,759 total tokens$6.0257 reported API cost49 reached grading

PASS RATE BY TASK FAMILY

Motion and contact19/26
Structural response5/7
Thermal response3/3
Electric fields2/2
Terrain and vehicles4/6
Robotics3/6

TASK OUTCOMES

Passed Did not pass

TASK 001 / RECORDED RUN

PASSED

free rotation about the centered Z axis

Physics engine
Project Chrono / PyChrono
Last stage
graded
Failed checks
None reported
Prompt tokens
7,094
Completion tokens
408
Total tokens
7,502
Reported API cost
$0.0365

Recorded request: 8192 output tokens, low reasoning effort.

TASK SET

Task examples

TASK 001Motion

Flywheel

Gain at least 30% angular speed after 2 seconds while keeping mass, material, and outer diameter fixed.

TASK 018Contact

Gravity chute

Reshape a jammed throat so three loose balls reach the collection pocket in time.

TASK 039Terrain

Terrain wheel

Reduce wheel sinkage on fixed soil under an 80 kg load while preserving the wheel’s core dimensions.

TASK 045Robotics

Robot gripper

Retain a cylindrical payload through a fixed lift-and-shake path without excessive jaw force.

Motion and contact 26Structural response 7Thermal response 3Electric fields 2Terrain and vehicles 6Robotics 6

EVALUATION PROTOCOL

CAD export, simulation, fixed checks

  1. 01

    Task inputs

    The model receives an instruction, experiment setup, starting CadQuery script, and starting STEP solid.

  2. 02

    Model response

    The response must define one valid CadQuery solid in millimeters. The grader exports it to STEP.

  3. 03

    Physics experiment

    The task's simulation evaluates the submitted geometry using Project Chrono and any task-specific mapping.

  4. 04

    Pass criteria

    Task checks test the physical objective and fixed constraints. The task passes only when all required checks succeed.