Looks Right, Teaches Nothing
The Setup
It’s been almost two months since my last Field Note. Honestly? This AI transformation stuff has been hard for me too.
My AI as knowledge layer approach makes sense in theory. I’m trying things, figuring them out, slowly making sense of them, and results are promising where teams adopt a new way of approaching the task/project/work. But at scale for everyone else, currently it’s “just using a tool”. Underneath it all, we’re slowly changing how we work, all while the pressure for more output keeps growing.
Meanwhile, my life outside work got taken over by hockey season. Both my boys are playing, and this year I signed up to volunteer as a coach. I’ve never coached before. But I loved playing BACK IN THE DAY, and I still have dreams about playing hockey at least once a year, so… might as well!
The league gives new coaches a lot of material: practice plans, a development model we need to get certified on, guidance on what matters at each age. And I had real questions. What do I emphasize with 11 and 12 year olds? What about my middle son’s group of 9 and 10 year olds? Is it mostly strategy and tactics or skills? What do I do when a kid keeps doing a certain thing on the ice?
So of course (classic me) I went and built a hockey brain.
My Approach + AI Role
I started by pulling the league’s core principles and materials into a Claude project. What came out was simpler than I expected. It’s about player development and kids learning to play as a team. Less about the outcome you see on the ice, because they’re still learning.
One story from our certification course stuck with me. The goalie who carried the US Olympic team to gold this year played high school hockey in Michigan. No elite travel team, no national development program, and no junior team picked him up after his senior year. Then look at who he became. A great player can come from anywhere, and early on they might not look like anything we’re trained to spot.
So I want to pay attention to each kid on their own terms. That turned into two tools.
The first is a shift tracker on my phone. Line changes come fast, and it’s easy to keep sending out the kids who are making plays without meaning to. Each shift, it marks whoever has had the least skating time, and I decide who goes. After the game, I can see patterns I didn’t notice I was making. (The first night, two kids argued they’d sat back-to-back. No app can work that one out live. Building trust with the kids does though.)
The second is a drill animator. Practice plans are a few arrows, some squiggly arrows, and some text, and I’d stare at them in the parking lot trying to work out what the drill was about. So the plan PDF goes in, and out comes a short looping animation I can glance at on my phone at the rink. Claude draws each one from a small file describing positions, paths, and timing, with design guardrails for what each frame should look like.
What Actually Happened
The first drill I ran through the animator was a 1v1. One player carries the puck down the wall, the defender mirrors them and then angles them into the boards to take it away. Competitive contact, no checking yet.
The animation looked right. Every line was drawn correctly. And the defender just… followed the puck carrier around, instead of cutting them off.
It felt lifeless and robotic following a path.
That’s when it clicked for me. My guardrails described what a frame looks like with placements and directions. They never said why players move. Who’s reacting to whom. Where your chest points when you’re defending. When a skater is actually going backward. I had encoded a clever animation, but the whole point of it is communicating what the drill/play is about.
So I asked Claude to research and learn how the game works from coaching sources, so it could judge an animation instead of just following coordinates. There’s a relationship between the players being animated as circles, and it needs to figure out what the natural reactions would be for these circles. That helped a lot. Then AI got excited and started proposing measurements: the defender finishes three feet too far ahead, the facing is off by so many degrees, every skater moves at the same speed… etc.
Over engineering flag. I denied that rabbit hole since I’ve been there before.
I understood the drill from the animation even with the defender drawn a little ahead. The goal was never to explain 100% of the drill via perfect pixel animations. It was to get a new coach to go “ohhh, now I get it.” So the point became: would a new coach get this at a glance?
There was also a moment where I caught myself. Watching the drill’s ending, I wanted the defender to win the puck. Then I realized that if I’m helping the puck carrier on ice, I want the opposite. Which ending I wanted depended on who I was picturing in my head. So the drill animations now end neutral, both players shoulder to shoulder on the puck, and the coach finishes the coaching in real life.
Yes, real life and people, doing real explanation and demonstration on the ice. AI helps us understand better, shoulders repeatability. Not to replace the entire explanation and coaching part entirely.
The next two drills I animated were small-area games, and the rules I’d built for the 1v1 made it a mess.
Every movement line was drawn on screen to animate a 3v3 game, correct and cluttered. And I learned: In a 1v1 skills-based drill, how you skate is the lesson. In a game-centric drill, what matters is illustrating where all the players and the puck are, not to replicate movement for movement for all 6 players on the ice. I didn’t know that until I saw the cluttered animation using the wrong intended use I developed first.
The corrections kept shrinking. The first drill took five rounds. The second took about two. The third didn’t need any, because everything I’d learned was already written down.
The Real Insight
I couldn’t have described what good looked like before I started. I only knew it when I saw it drawn wrong.
So the useful part wasn’t the animation. It was the judgment I wrote down each time something looked off, which made the next drill easier. Make it, see what’s wrong, name it, write it down, build the next one.
Try This If…
If you’re building an assistant for something you’re familiar with but perhaps not broken all the rules down (for me, a volunteer coach brain): build the smallest version quickly and let it be wrong. Each mistake tells you a rule you didn’t know you knew. Trust your instinct and describe what’s wrong, and then ask AI to document the learnings as a principle so it knows how to navigate it next time.
A watch out is understanding what “good enough” means before the AI decides for you. It will happily keep outputting in logical ways. You’re the one who knows what the point of the work is meant to serve and how it helps a person on the other end.
Systems Lens
The league kept telling us the same thing: judge development, not what it looks like on the ice. My first drill animation looked right and taught nothing. It only got useful once I paid attention to what a coach actually needed to understand.
I think that’s what’s been hard at work too. The pressure is on output that looks efficient. The real change is slower and harder to see… people learning a new way to approach the work. It’s messy. Some of them won’t look like they’re getting it at first. Neither did a high school goalie from Michigan.
And if you’re a volunteer coach who loves the game but feels lost with a practice plan, or with handling situations in developmentally appropriate ways, I’d love to know if something like this would help.
The Drill in Motion
Here’s the 1v1 drill as it plays now.
I’d love to show you the robotic first version next to it, but I deleted it. I’ve been trying to get better at digital hoarding… turns out it works.