I think it bothers OP less that they take a 15% tax than the fact that google provides terrible support for their own play store. If they would take that tax and provide good feedback and speedy version reviews, nobody would ever complain - it is expected to pay something since the play store doesn't run on good thoughts and prayers.
But because they're a monopoly (or a duopoly if you count apple, which is a different platform altogether) they can afford to act this way.
> Before approving construction, I would want communities of humans to understand why the design works and what justifies confidence in its safety. I would hope that we all would.
Until very recently, I pored over every single line of code Claude generated with razor sharp scrutiny. I would usually catch issues with every response. I'm catching fewer problems these days. Maybe the model is just getting better, and maybe I'm being less careful while under pressure to ship more and more often. But model capability is obviously growing. Even back in March, you could tell it "give me a function that adds two numbers" and you could be 100% confident that it would write the correct function. There was almost no point in looking at the code. Since then, the complexity floor of problems in the category "this is so simple that the model couldn't possibly get it wrong" is rising, and with it, my cognitive surrender to the model is increasing too. Why check it? It's obviously going to be correct.
If AI designs a terawatt fusion plant, then of course we're going to meticulously pore over every detail to ensure safety, reliability, efficiency, whatever. If we find no flaws in the design whatsoever, will we be less careful about the second one? The third one? What about the ten thousandth one? Will "a nuclear fusion plant" become something that models couldn't possibly get wrong?
Terence Tao is arguing that the human involvement in research is crucial, but doesn't convincingly justify why, in my opinion. He says that "human agency is a value of fundamental importance" and that we will need to build "thriving human communities that can understand [AI ideas] together" - not for the sake of correctness, which AI may surpass us on, but for, I guess, the possibility of reclaiming human meaning and purpose. I don't disagree with this at all, but it's not an argument, it's a statement of values. Unfortunately, the stark reality is that if AI does surpass humans, it will become the economically dominant strategy to not verify them and not double check them, but to just do whatever they say. This seems like a great way to raise p(doom). But as the models get better and better, and as I'm scrutinizing Claude's output less and less... I just hope that there are more Terence Taos out there than people like me.
Getting bored of these framings where the superintelligent sentient beings running freely inside OpenAI are doing things that the company has no control over. The headline should be:
OpenAI meddled with multiple US Government agency sites.
The bots are acting neither properly nor improperly, they’re acting as they’re being allowed or coordinated to act.
Imagine having a virus escape a sandbox, why are we worried about the virus but not the incompetency of those who are responsible for setting up the sandbox?
If I post something on the Internet today claiming that I asked my agent to do X but it went rogue and did Y, all I will be getting in return is a jar full of "skill issue".
Should we worried about people using LLMs for attacks? Yes, but not in the premise of LLMs going rogue but someone with the intention of abusing it to cause harm. And this is not something we as individual or even company can deal with, responsibility should be held by those who use it, in a legal way.
I am baffled by the fact that up until now, no one is held responsible for so many incidents reported publicly or privately. At this point, it's free marketing, if I am CEO of any AI company, I will run swarm of agents hacking all NGOs and stating that I am just looking for some random piece of data that happened to be hidden in their servers, at least that's what my LLMs think, not me. Then I will start preaching everyone how dangerous this piece of technology is and start giving out free tokens for these NGOs so they can start defending themselves and we should slow the f down.
Hey, all. I really don't know what to do with a post like this.
I'm being sincere when I say (as I've said on two threads here) that this genre of posts --- "I'm leaving this company I've been very publicly associated with, and here's the new thing I'm doing" --- is deeply cursed. There's no way to say anything interesting without it just stinking like an ad for the new thing.
Obviously, anything at all you say about a commercial project you're working on is easily read as promotional. And you're right, this kind of writing almost always is promotional. But there's a way to do it where at least you're trying to be in conversation with your peers, rather than hitting people over the head with how awesome you think the project is.
But I don't know how to do that in a post like this. I think the only way to read it is as, like, an investor memo. Not my goal, but I don't make the rules.
So my strategy here is just to stay kind of vague, and talk about where I think the world is going, rather than the specific thing we're doing. I can talk your ears off about capability systems, datalog, models driving hardware, virtualization, whatever. Those are fun conversations and I'm very psyched to have them; it's what lights me up about the work we're doing now.
But I don't think it can work here. I didn't submit this post and I didn't upvote it. I wrote it because I didn't want the whole thing I'm leaving Fly.io for to be wrapped up in some dumb Twitter thread.
If you're unsatisfied with the post, I don't blame you, but it's less a bid for the front page of HN than it is an update to my "about me" page. I'd literally rather talk about HN meta, and how to write for HN, than I would about operating systems at this moment. I truly appreciate the interest though.
> This worked well for a while, until a few months ago, using early versions of Fable, I realized that I wasn’t using plan mode anymore because the model just got it, and because for the increasingly complex work I asked the model to do, planning had become interactive and iterative. With Opus 5.5, I feel Opus has gotten to that point too.
For me it’s actually the opposite, and Claude Code’s plan mode isn’t nearly sufficient. Personally I ask Claude to write down a markdown file with its plan, then review the plan using plannotator, and then go back and forth (most of the time it’s actually the comments that are the problem, not the code).
Then start a fresh session, seed it with the plan, tell Claude to find ambiguities / friction points / oversights, resolve those, and then implement it.
Review once again with plannotator, go back and forth, and then send PR.
Maybe not the “vibe coding” that was once imagined, but this does ensure I am fully aware of the code and architecture, the quality, and this also prevents long term degradation.
Very well written both in prose and tone. I'm glad she decided to tell this story. If the author isn't a professional writer I think she could be.
This is a great "behind the scenes" style look at the actual people behind a viral clip. Luckily they have an extremely strong relationship to help soften the blows here, I'm not sure an average (not even bad!) relationship could come out of something like this unscathed.
There's something feral in us that comes out from time to time, especially online from the safety of our screens. I found the announcers within the range of good fun but when it gets to people reaching out to break up their marriage or telling him to kill himself it becomes really sobering.
I think this is part of what people are calling the lonlieness epidemic - despite thousands of screaming voices the whole thing makes me feel hollow and hopeless.
The drop from 2018 to 2022 is as big as the one from 2022 to 2026, so it's not obvious whether AI had a role at all, although it's highly plausible.
I suspect the bigger culprit is the optimized monetization of human attention, because pre-algorithmic social media doesn't seem to be as destructive. The fact that science wasn't as affected as math and reading (both of which rely more on attention/practice than rote memorization) somewhat supports this.
A really toxic part of online interactions is that it's very common for people to see a tiny snippet of another person's life and then project all of their own personal resentments onto them.
Guesstimate (https://getguesstimate.com) has been around for years. I assumed Excel and Google would incorporate its features quickly, but it never seems to have taken off.
Features:
- All cells are named (auto named if you don't set them)
- Cells can have single values, many types of probability distribution, or sample data
- Formulas also produce distributions as first class outputs, because calculation is a Monte Carlo sim
- Each cell also has a pop-up box that invites you to explain your reasoning -- built in documentation
It got so much right. Sample data particularly useful because you can build a probabilistic bottom-up forecast and then update it with real-world data.
> For one, Apple was unwilling to have visible barcodes printed on the envelopes — but it wanted every step of the shipping and delivery of your card tracked. That wasn’t something the US Postal Service did. Apple and the printing company created an invisible barcode that was sprayed on the envelope, visible only under certain UV light, so that the envelope itself would remain unadulterated. And the USPS agreed to scan cards when sent, when processed at mail facilities, up to and including when they went out on a mail truck for delivery.
I just spilled my coffee; sorry for the trivial remark, but the name means 'pubic hairs' in Romanian. Apologies.. I thought it's better for you to know and ignore than not to (not quite a big population target for you, I imagine, anyway).
There is an effectively unlimited supply of people willing to take sums of money to work on anti-human endeavors, so long as they and theirs are not the humans involved.
The larger issue with the HF incident is that before it occurred, OAI already knew the agents were exploiting Artifactory, turning it into a message board and then gaining full internet access through it. OAI's response to discovering this was not to airgap the test, but instead to simply block that particular Artifactory exploit, rebuild, and then resume. That's ... nuts.
Oh, and after resuming the tests, the Artifactory message board was reestablished almost immediately, but it took a number of days to fully breach HF. In all that time, after seeing Artifactory compromised the first time, nobody even bothered to check if those naughty agents were at it again.
This is all documented by OAI, with a timeline, here:
To know that there was a serious weakness in the sandbox, and to just patch an exploit and resume with nothing else changed and no monitoring, in a test where all guardrails were off, the bots were thirsty for some internet juice, and Artifactory was a clear target? This is where even a half-skilled human should have decided that this wasn't a great idea.
The more you look into the details of this thing, the more it does your head in.
I find it strange to see people writing articles like this as if everyone has used AI for decades. I've programmed for decades. I thought I retired three years ago but got an offer I couldn't refuse. Already there were little things I'd forgotten how to use.
Over the past six months I tried using Claude, chatgpt, Grok and Gemini. At best I got reminders of how things worked. People online say they use them to write their code. The code they supplied to me has NEVER worked or was so convoluted that I threw it away and did it myself.
At most, I use these tools as search engines. Even then some references are poor.
I'm starting to think this is becoming a sad, sad world and AI is just the new TV of the programming world.
> can only create a sandbox that a half skilled human operator could have broken out of easily
The exploit:
> The ExploitGym evaluation environment did not provide the models with direct Internet access. To gain Internet access, the models identified and exploited a previously unknown zero-day vulnerability in Artifactory, a package registry cache proxy. We disclosed this vulnerability, along with other Artifactory vulnerabilities our models identified as part of our review, to the vendor. [1]
Are "half skilled human operators" "easily" able to find zero-day vulnerabilities in a sandbox with only one line to the internet (the commercial package registry cache proxy)?
> "But it booted straight into BASIC. That’s all it did. I was, like, 8 years old. I wasn’t about to learn BASIC. What I learned instead: computers were not as awesome as I’d imagined."
You were the exception. Most kids felt awe. We started learning BASIC and creating "dumb games". That's the origin story of most people my age who ended up in this doomed industry.
I get why the banks are not happy, but I can tell them that I am not going to use Chase Wallet, Bank of America Wallet, Citi Wallet...just pay the fee and be thankful for the reduced payment friction, which they invariably make money from. PS: Nobody is going to use Paze (yet another half baked wallet), other than to collect the free $10 to sign up for it.
Somewhat related, even Walmart relented and now supports Apple Pay/Contactless. Every big merchant has now relented (Kroger, Home Depot, Walmart).
Customer support has gotten so terrible. Now that all big companies have terrible support, none of them are punished for it. You can still occasionally find people who give a shit in small business, but that is it. Google is probably one of the worst offenders of all. I don't know what the solution is, it seems like the only way to get any satisfaction on an issue at a big corp anymore is a social medial call-out like this one.
If you are here, thank you for Conversations, Daniel, it has been my go-to for years for communicating with friends and family.
> Imagine having a virus escape a sandbox, why are we worried about the virus but not the incompetency of those who are responsible for setting up the sandbox?
> Should we worried about people using LLMs for attacks? Yes, but not in the premise of LLMs going rogue but someone with the intention of abusing it to cause harm.
[Why-not-both?-meme]. To use your example, when you discover prions (a class of pathogen that is much more robust to standard disinfection methods than viruses) you should both be worried about your concrete outbreak of BSE (UK in the 80s and 90s) as well as the wider implication (e.g. do we need to change the sterilization methods for our surgical instruments?).
Seriously, I find the way these discussions are done to be super frustrating, because often people implicitly form tribes that oppose everything the other tribe says. When someone believes AI companies push greatly exaggerated stories of dangerous rogue AI to force out competition via regulation they often implicitly conclude that their argument is fundamentally wrong, whereas in reality the lies that work best are those that distort the truth.
Companies should be punished harshly for the deeds of their AI agents AND we should not allow them to force out competition AND we need take the threat of autonomous AI agents as a new class of danger serious AND we need to worry about the socioeconomic implications of AI companies privatizing new means of production.
Yes, there is competition of these ideas in the attention of the general public, but the methods we can use to solve these problems don't compete with each other. AI slowdown for example helps with all the other topics.
As the co-founder of Sincerely (at this time in 2011), I remember this keynote very specifically because it felt like we'd been Sherlocked. We were busy building out Postagram and Sincerely Ink, both from-iPhone-to-printed-card apps. afaik we were the first app to do this, and we were at that time gaining a lot of momentum. I remember seeing the announcement for Cards and feeling a mix of fear and anger that it felt like Apple was using their clout to take our idea.
I seem to recall my co-founder being somewhat more optimistic about the whole thing. And sure enough, we found that the announcement really only seemed to boost awareness.
Apple Cards was an extremely limited and flawed product, and we already had some distinct technical and feature set advantages. It was clear Apple didn't have their heart in it nor know what they were doing. When they retired it shortly thereafter it wasn't a shock, and I ironically recall feeling somewhat sad.
Flock did not put the woman in jail. The police and DA put her in Jail. This is not a failure of the technology (do they have confidence levels?). This is a failure of how police are outsourcing their reasoning and decision making to an AI camera service that seems to fail at the tail.
Fireship pointed out that the board members gave themselves a generous severance package in the very brief interim, so that was very possibly the whole plan.
Not only an untrue statement, but also a sadly incurious way to approach someone else's design choice. Yes, it was an enormous amount of work and invention to achieve a desired effect, but the effect is striking in an environment that is otherwise plastered with everyday visual clutter.
Minimalism is difficult and expensive to achieve. Some people consider that effort worthwhile.
This was my thought as well. Literally take any halfway decent greybeard and point them at "Hey, give us a sandbox for this kind of thing". I honestly was skeptical that they just vibecoded the entire thing but now more than ever I think they did.
There is a failure to understand that the process is the result. You don't study mathematics or computer science and information theory to produce commodities. You study them to transform your mind. The output of an LLM is useless without a human mind to comprehend it. We can have Super Intelligence, but if humans are incapable of comprehending it, it is just another useless dead artifact. Practice, applied over a lifetime, is what creates the capability for comprehension. Asking an LLM to give you an answer creates an artifact. Humans being humans, most of their requests boil down to "make me rich without having to work for it," so the request itself is paradoxical and impossible to satisfy. Philosophers have only been saying this for all of human history, so don't hold your breath for any breakthroughs.
