tanshuai.pages.dev AI research

Tan Shuai

谭帅 · 譚帥 · Tán Shuài

Independent AI researcher and developer. I test AI models blind, on real work, and publish what wins.

Portrait of Tan Shuai

I spent twenty years building the chips, machines and code that AI runs on. Now I work on AI itself, on my own. This page collects my tests of AI models. Each one is run blind on a real task, scored on stated criteria and priced at the vendors' own rates, with the full write-up on tanshuai.com.

Benchmark 1 · October 2026

Which AI model captions and tags photos best? GPT-6 vs Claude 5.5

Ten configurations covered Claude Haiku, Sonnet and Opus 5.5, and OpenAI's GPT-6.1 Sol, GPT-6 Astra and GPT-6 Luna at several effort levels. Each one gave the same 50 real photos a title, a caption, keywords and an album, and judged whether to keep or delete each photo.

Answers
500, shuffled and relabelled A to J for every photo, so no grader knew which model wrote which answer.
Graders
Claude Fable 5.1 and GPT-6 Astra at xhigh effort, scoring independently while looking at each photo.
Criteria
Five, worth 0 to 2 points each: facts, no invented details, the delete call, album fit and usefulness.
Cost
What captioning the whole 142,521-photo library would cost at each vendor's official API price.
Average score out of 10, and the cost of the full library
Model and effortScoreBatch API
GPT-6.1 Sol, max9.72$563
GPT-6 Astra, max9.61$2,987
GPT-6.1 Sol, ultra9.59$468
GPT-6 Astra, ultra9.58$2,156
GPT-6.1 Sol, medium9.56$240
Claude Opus 5.58.57$661
GPT-6 Luna, max8.41$20
GPT-6 Luna, medium8.11$11
Claude Haiku 5.57.81$18
Claude Sonnet 5.57.68$334

Cost is for all 142,521 photos through each vendor's Batch API. The standard API costs twice as much.

  • Best score: GPT-6.1 Sol at max effort. 9.72 out of 10.
  • Best value: GPT-6.1 Sol at medium. 98% of the top score for $240, about a third of what Opus would cost.
  • Tight budget: GPT-6 Luna at max. 8.41 for $20, ahead of Haiku and Sonnet.
  • Skip: Claude Sonnet 5.5. Mid-priced, and the lowest score.

Limits: 50 recent photos from one city, the Claude models at their default settings only, and graders that are AI models rather than people.

Read the full write-up, with charts and examples, on tanshuai.com

What I work on

  • Evaluating models on real tasksBlind grading, stated criteria and real prices, so the result is one you can act on
  • AI agents and the devices they run onWhere the model meets the hardware
  • The silicon underneathTwenty years of chips and systems, from mining ASICs to mobile SoCs

Before AI

Co-founder of Chipchain Technologies, and a founding member and director of MicroBT, the maker of the WhatsMiner Bitcoin mining ASIC. Earlier I was R&D manager at Vimicro, which made mobile multimedia SoCs. I hold 20 granted patents in the United States and China. See my work and patents, or the engineering notebook at tanshuai.github.io.

Follow the research

My official pages: tanshuai.com, tanshuai.github.io, tanshuai.vercel.app and tanshuai.netlify.app. The full list is at tanshuai.com/official.