Jadi saya menguji GPT-6.1 Sol pada pekerjaan nyata: 2 repo, 105 bug y…

@dazuoforyou Balasan ♥ 0

@PawelHuryn claude sonnet better than opus?

Asli

@HyCe Balasan ♥ 0

@PawelHuryn Sonnet 5.5 in first, what?

Asli

@4misceldah Balasan ♥ 0

@PawelHuryn @grok explain

Asli

@cyberpeterg Balasan ♥ 0

@PawelHuryn GPT 6.1 sol has been absolutely stupid for me. https://x.com/cyberpeterg/status/2105313537543303596?s=20

Asli

@TheHasnainK Balasan ♥ 3

@PawelHuryn what was the 1 bug Astra caught that Sol missed?

Asli

@PawelHuryn Berantai ♥ 8

@TheHasnainK The answer key and specific bugs aren't public, but you can see a bug coverage here: https://bughunt.productcompass.pm/
Asli
- @tf3y Berantai ♥ 1
  
  @PawelHuryn @TheHasnainK Why is Fable 5.1 not included?
  Asli
  - @PawelHuryn Berantai ♥ 1
    
    @tf3y @TheHasnainK You can include it on the website. It's barely better than Opus 5.5. No point in using it right now, IMO.
    Asli
    - @PawelHuryn Berantai ♥ 1
      
      @tf3y @TheHasnainK @tf3y, it's a single run, so the coverage looks less advanced. This single score is 1.3 points higher than the average for Opus 5.5 (n=3)
      Asli

@Vaeshkar Balasan ♥ 0

@PawelHuryn Nuts

Asli

@pelfiv Balasan ♥ 0

@PawelHuryn 44 bugs, $6.56. Cost-per-fix is the ladder.

Asli

@nukey_the Balasan ♥ 0

@PawelHuryn WOWIE! amazing results :)

Asli

@borgerntendies Balasan ♥ 0

@PawelHuryn @grok sol 6.1 is clearly cutting into OAI's margins. Is this "race to the bottom" reminiscent of early 2000's internet prices? When did this race start? When did it end? Outcomes?

Asli

@magdake2876 Balasan ♥ 0

@PawelHuryn Просто на подписке за 200$ можно было бы бесконечно пользоваться 6.1 sol ultra и не покупать кредиты, openai это непонравилось😂

Asli

@heysrihari Balasan ♥ 0

@PawelHuryn do the fixes get checked for new bugs too, or just whether the planted ones are gone?

Asli

@thierrybijou Balasan ♥ 0

@PawelHuryn Opus 5.5 at 41.7 for $58.53 against 6.1 Sol at 44 for $6.56 is the line that matters here. Once you log a few more runs, I'd love to see whether the misses overlap or whether Opus catches bugs Sol never touches, because that decides routing versus just picking one.

Asli

@turkelitopcusu Balasan ♥ 0

@PawelHuryn how much you got from openai? "With a model like this, the 50% cut to the $200 plan doesn't matter." all these are just for this sentences to plant the idea it's ok. it's not ok bro.

Asli

@Sercanvwty Balasan ♥ 0

@PawelHuryn Short: 6.1 Sol finally sits on the right shelf. Not the bug king. The king of cost per bug found.

Asli

@1st_knight Balasan ♥ 0

@PawelHuryn Interesting

Asli

@icauro_ Balasan ♥ 0

@PawelHuryn There is alternative 3D visualization of the results, can make 3 axis comparison easier. https://icauro.com/toabundance?benchmark=bug-hunt

Asli

@AlessandroScola Balasan ♥ 0

@PawelHuryn The only problem? "remove the xx button from the page" ... 'valutating bla bla bla' ..... 10 minutes later 'button removed, i'll now verify if the button is not visible' ..... . 15 minutes... 'the button is removed and not present on the page' (but still in temp page) 30m tot

Asli

@Blum_OG Balasan ♥ 0

@PawelHuryn thanks for running this and sharing the test results!

Asli

@travisfont Balasan ♥ 0

@PawelHuryn A bit useless to show the highest level of AI. The real challenge comes when you test on medium.

Asli

@sturzstrom Balasan ♥ 2

@PawelHuryn is it just me or is gpt 6.1 sol slow? what tps are you getting?

Asli

@IngIndexado Balasan ♥ 0

@PawelHuryn In terms of criticality (identifying more impactful bugs), was GPT-6.1 Sol on par with GPT-6 Astra and Fable 5.1? Btw, thanks for this, this is gold! 🤝

Asli

@Mai8ja4z Balasan ♥ 0

@PawelHuryn https://x.com/SlovenskiGlas1/status/2105265982948180341

Asli

@OM56271680 Balasan ♥ 0

@PawelHuryn GPT 6.1 Sol is basically a telenovela: the cast is great, young, and gorgeous, the costumes are stunning, but almost nothing actually happens. You could easily skip four episodes and still not miss a beat.

Asli

@PhilForrence Balasan ♥ 1

@PawelHuryn TY

Asli

@jakestone2306 Balasan ♥ 1

@PawelHuryn Glad you're adding n=3. I'd put a bug-by-bug stability view next to the totals: fixing 44 different bugs each run creates a different review workload from reliably fixing the same 44.

Asli

@PawelHuryn Berantai ♥ 2

@jakestone2306 It's not 100% deterministic. Bugs differ a lot between runs. This view: https://bughunt.productcompass.pm/?preset=featured&view=coverage
Asli
- @SvenMeyer Berantai ♥ 0
  
  @PawelHuryn @jakestone2306 GPT-5.6 Sol + GPT-6.1 Sol looks like an interesting combination or is it just a coincidence that one finds bugs at the end the other one missed ?
  Asli
  - @SvenMeyer Berantai ♥ 0
    
    @PawelHuryn @jakestone2306 Uh, just saw that GPT-5.6 Sol is horrible expensive, I would get away cheaper with Astra ! ... but combination of multiple models might be a cost effective approach to try.
    Asli
    - @jakestone2306 Berantai ♥ 0
      
      @SvenMeyer It could be complementary coverage, or just run-to-run variation. I'd compare two passes of one model with one pass each of two models, at the same total budget. Count newly fixed bugs on the second pass, not just the combined score.
      Asli

@lau65536 Balasan ♥ 0

@PawelHuryn For finding bugs, does it make sense to first run Sol 6.1. Then run Opus 5.5 with the knowledge of known bugs from Sol. Can it lower the total cost while catching more bugs?

Asli

@WirMachenAuf_de Balasan ♥ 1

@PawelHuryn i used 6.1 for bug findings. it was much better than opus 5.5. but opus is much better in performance, design and smart code

Asli

@laugh2MuchSoMB Balasan ♥ 0

@PawelHuryn 跟我在實際工作中使用6.1 sol xhigh 的體感一樣，有好又省！好像有點慢？

Asli

@sapientwilight Balasan ♥ 0

@PawelHuryn I would advise against testing every model on Max reasoning. Most tend perform better on Medium or High, like my test here where Terra and Luna both suffer on Max:

Asli

@taoyetouzi Balasan ♥ 0

@PawelHuryn The real story is the price ratio, not the score — 44 bugs for $6.56 vs 45 for $33. At one-fifth the cost you can just run five passes and brute-force the gap. Makes the whole "Astra got shelved for safety" line look more like a pricing problem.

Asli

@iae2499 Balasan ♥ 0

@PawelHuryn 6.1 sol’s performance for the price is honestly wild.

Asli

@Jorne2039616 Balasan ♥ 0

@PawelHuryn Great overview! Are you planning to also check gpt-6.1-luna?

Asli

@shamanic_arts Balasan ♥ 1

@PawelHuryn it matters. this model ships far more slop than 6 sol did

Asli

@JDisclosur94877 Balasan ♥ 0

@PawelHuryn Would be nice to see amount of slop and regressions introduced. Gemini for me was terrible for fixing things, it wasn't using good architecture, patterns and introduced a lot of regression. But the results may actually vary by how good your agentic context is in the workspace.

Asli

임베드 보드

Komentar 42