<h3 id="user-content-intro--tldr">Intro / TLDR</h3>
<p>There have been a lot of developments since I wrote about my first stab at <a href="/blog/the-app-that-ai-made">creating an app via LLM</a>, and by leveraging them I've since speedrun an AI generated concept from blank slate to passing the App Store acceptance gauntlet in ~20 minutes (less than 5% the time it took on my first attempt).  My take on the best way to capitalize on the monumental shift in software development brought to bear by frontier LLMs has only grown stronger and the tools that empower it have (predictably) become even better.</p>
<p>In this post I walk through what's new and how best to apply it given my hands-on experience, through which I created a tool to help with this sort of workflow called <a href="https://www.arcodeai.com/">arcode</a> (which I'll also compare and contrast with other broadly available similar solutions).</p>
<h3 id="user-content-my-workflow-broke">My Workflow Broke</h3>
<p>When I first started experimenting with creating an app in a language I didn't know using modern generative AI, I had an intentionall primitive approach.  Given the still relatively nascent nature of capable LLMs, I wanted to ensure there were as few external dependencies and abstractions as possible.  The more variables I could control for, the better.  This lead me to initially interfacing largely with various vendors' consumer facing UIs directly - an arrangement which came with its own set of pros and cons.</p>
<p>Most notably, this workflow incurred significant friction from shuttling code back and forth via copy/paste - but I generally got more bang for my buck given the flat monthly vs by-token billing model.  Since <a href="http://localhost:3000/blog/the-app-that-ai-made#rubber-meet-road">my process</a> involved dumping as many complete files that were relevant in my codebase into the prompt as possible and requesting complete modified files without truncation as output, I was generating a lot of throughput - and this input/output commonly triggered Anthropic and OpenAI's rate limits (Gemini had already constrained input message length in their UI to be insufficient for my purposes at the time of my initial experiments).</p>
<p>Still, I was getting a great deal.  Looking back at the billing incurred when I ultimately switched to using API calls rather than monthly-billed consumer UIs, I'd easily run through a month subscription's ($20) worth of usage in just a few days on average.</p>
<p>Given this value proposition, I was more than happy to keep playing proxy for the robots - but the providers got wise and tightened the screws.  Both OpenAI and Anthropic began limiting my message size arbitrarily below the max token length of the underlying models when using their UIs - a huge blow given that the <a href="https://crayeye.com">CrayEye</a> codebase had grown beyond the size of these new limits.</p>
<p>It's worth noting that at the time of this writing, Claude has since increased its UI message length constraints at least enough to accommodate the codebase it previously limited - alongside adding some other cool (and relevant) features like <a href="https://support.anthropic.com/en/articles/9487310-what-are-artifacts-and-how-do-i-use-them">artifacts</a> and <a href="https://www.anthropic.com/news/projects">projects</a>.  The direction of their UI is, in my opinion, the current gold standard for a consumer facing LLM product - and continues to improve.  That said, when it initially broke it left me without a way to continue building the way I had been and served as the perfect impetus to embark on the next stage of my workflow optimization.</p>
<h3 id="user-content-theres-an-app-for-that">There's an App for That</h3>
<p>I was aware of <a href="https://www.cognition.ai/blog/introducing-devin">Devin</a> and <a href="https://github.blog/2024-04-29-github-copilot-workspace/">Copilot Workspace</a>, but wasn't hip enough to get off the waitlist for either.  Of the myriad freely available solutions available, <a href="https://aider.chat/">Aider</a> looked the most interesting to me - it was a CLI tool, supported sending and receiving complete files, and was open-source.  I fired it up and tried to make some minor changes to CrayEye, but due to an error in the way its default setting tried to generate and interpolate snippets of code rather than dealing with whole files, the change broke the app.  I decided that, given my previous experience expediting development via holistic AI code generation, it was worth taking a quick pass at recreating a workflow that I already knew worked and using tools and UX I was well familiar with.  This is all to say that yes, there are other (frankly more sophisticated) tools available for AI driven development today (more on that later) - but we're already at the point where, at least for certain simple workflows, it's just as quick to build something bespoke as it is to come up to speed on something new that you aren't acquainted with.</p>
<h3 id="user-content-frankensteins-automaton">Frankenstein's Automaton</h3>
<p>In order to unblock the ability to work on CrayEye as quickly as possible, I prioritized re-enabling the process exactly as I'd performed it before (i.e. prompting for complete file changesets with a large input context) with further optimizations and enhancements to follow.  Fairly quickly, I had a scrappy CLI that would let me copy the modified files one by one, and as I started using it to build itself it fast grew the ability to write the full changeset, support image input and followup prompts, and include context from remote URLs.  It was almost surreal watching the tool build itself, and while it was trivial to move to using a stable version to build changes in the very beginning it was effectively performing surgery on itself (breaking itself in the process more than once).</p>
<h3 id="user-content-videos">Videos</h3>
<h3 id="user-content-uninhibited-development">Uninhibited Development</h3>
<p>One highly underrated aspect of not writing the code to make a project, particularly one that is open source, is the implicit detachment of shame for imperfections or incompleteness.  When it's the robots writing the thing, you can let yourself work less inhibitted by the need to be obsessive and write unimpeachably perfect code - even if you're going to show the world that code.</p>
<h2 id="user-content-post-photo">Post Photo</h2>
<p><a href="/img/blog/upshifting-the-sdlc/bg.jpg"><img src="/img/blog/upshifting-the-sdlc/bg.jpg" alt="alt text" title="Minolta XG1 / CineStill 800T / Onondaga Lake, NY during the 2024 total solar eclipse"></a></p>
<p><em>Minolta X700 / CineStill 800T / Cyclist in Manhattan, 2024</em></p>


Lifting off for less by having bolder expectations

Upshifting the SDLC

<h3 id="user-content-intro--tldr">Intro / TLDR</h3>
<p>I made the multimodal multitool mobile application <a href="https://www.crayeye.com">CrayEye</a> in a language &#x26; framework with which I'm unfamiliar by relying on modern large language models to write not just snippets but the entirety of the code.  While I later on made minimal tweaks by hand (e.g. changing element colors or swapping their positions), LLMs did all the early &#x26; heavy lifting.</p>
<p>For the purpose of this exercise, I used the latest state-of-the-art models available to me via web UI:</p>
<ul>
<li>OpenAI's <a href="https://cdn.openai.com/papers/GPTV_System_Card.pdf">GPT-4</a></li>
<li>Anthropic's <a href="https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf">Claude 3 Opus</a></li>
<li>Google's <a href="https://support.google.com/gemini/answer/14517446?hl=en&#x26;co=GENIE.Platform%3DAndroid">Gemini Advanced</a></li>
</ul>
<h3 id="user-content-how-it-started">How it Started</h3>
<p>Ever since the early pre-release demos of <a href="https://openai.com/research/gpt-4v-system-card">GPT-4V</a>, imaginations have run wild with what sorts of magical new things we'll be able to build with it.  The implications of the technology for vision based user experience can't be overstated - what once took a dedicated team of specialists a substantial amount of time and resources can now be done with even greater proficiency and detail with a trivial call to a multimodal LLM.  I wanted to explore what they were capable of without spinning up <code>create-react-app</code> or <code>create-next-app</code> frontends for every idea and truly explore what <a href="https://www.oneusefulthing.org/">Ethan Mollick</a> calls the <a href="https://www.oneusefulthing.org/p/centaurs-and-cyborgs-on-the-jagged">jagged frontier</a> (i.e. the line between what this tech is and isn't capable of which can shift rapidly even among seemingly adjacent or comparable domains and tasks).</p>
<h3 id="user-content-the-multitool">The Multitool</h3>
<p>My requirements were largely:</p>
<ul>
<li>Quick interface for capturing input</li>
<li>Ability to use all cameras with minimal friction</li>
<li>Configurable prompts that could be edited and shared</li>
<li>Incorporation of on-board sensor data (e.g. location) into prompts</li>
</ul>
<p>I decided to create an app.  It had been a while since I'd created a native app and I'd been wanting to kick the tires on the landscape again, and this use case of a multimodal multitool offered the perfect opportunity.</p>
<p>Since my last run at making native apps, <a href="https://flutter.dev/">Flutter</a> had grown in popularity, so I decided to give it a go even though I hadn't worked with <a href="https://dart.dev/">Dart</a> before.  My unfamiliarity with the language was actually going to be useful here, because another thing I wanted to get my feet wet with was testing today's LLMs' capabilities for holistic development.</p>
<h3 id="user-content-rubber-meet-road">Rubber, Meet Road</h3>
<h4 id="user-content-setup">Setup</h4>
<p>I kicked things off by following <a href="https://flutter.dev/learn">Flutter documentation</a> to set up the <a href="https://docs.flutter.dev/get-started/install/macos/mobile-ios?tab=download">Flutter dev tools for iOS</a> and firing off a <code>flutter create</code>.</p>
<p>At this point, the boilerplate app's core logic was contained entirely in <code>lib/main.dart</code> - this made it particularly easy to start working with immediately.  I began prompting to add simple functionality - a camera preview, a remote HTTP request to analyze the image via GPT, and the app's functionality (and lines of code) quickly began growing.  My prompts leaned heavily on requesting:</p>
<blockquote>
<p>"the complete modified files after these changes with no truncation"</p>
</blockquote>
<p>This was critical because I wanted to both reduce the friction of shuttling responses between the LLM and disk as well as ensure it was fully explicitly accounting for areas of change in context with the rest of the code as it generated its responses.  My evidence is purely anecdotal, but it generally seemed to produce higher quality results with fewer regression issues than letting it pass along patch updates to partial sections of files.</p>
<p><img src="/img/blog/the-app-that-ai-made/claude-1.jpg" alt="alt text" title="Claude Chat Demo"></p>
<p>Shortly after my <a href="https://github.com/alexdredmon/crayeye/commit/76841ef2a105817d7062baf5055147bd8842a27c">initial commit (76841ef)</a> of a minimally functional POC, I had the LLM perform a <a href="https://github.com/alexdredmon/crayeye/commit/624797532388cd24e46041f97914378fcc15bcf2">modular refactor (6247975)</a> to split things into separate files.  This was helpful both from a best practices and workflow performance standpoint, given that I had to wait less for it to output smaller chunks of more modularly split files.</p>
<p>Now that things were in separate modules, I needed to distinguish the different files/modules from each other as I passed the codebase along to the LLM.  At this point, I added comments to the beginning of each file containing its name and an <code>// eof</code> comment at the end.  My prompt looked something like this:</p>
<blockquote>
<p>You are a software development team.  Ask for the codebase if not provided, and base all your changes around this codebase.  When provided with the codebase, reply only with "I'm in" unless a new feature has been requested.</p>
<p>The user will ask for features - respond first with the files that need to be changed followed by a brief summary of the changes. Only output files that require changes to implement the current feature. When you output the files, begin with the filename of the file being updated and then output the entire un-truncated file after modifications. Do not remove any comments. Ensure that the first line of each file is "// FILENAME" where "FILENAME" is the file's name, and that the last line of each file is "// eof" - and most importantly, make sure to surround each file with triple backticks "```" so that they will be entirely formatted as code.</p>
<p>Here's my codebase:</p>
</blockquote>
<p>By saving the prompt in the lib directory prefixed by an <code>_</code> (specifically <code>lib/_autodev_prompt.txt</code>) to ensure it floated to the top of sorted lists of files, I could easily <code>cat lib/* | pbcopy</code> and have my prompt alongside my entire codebase, delimited by filename identifiers, ready to paste in my LLM of choice.  In the running were:</p>
<ul>
<li>OpenAI's <a href="https://cdn.openai.com/papers/GPTV_System_Card.pdf">GPT-4</a></li>
<li>Anthropic's <a href="https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf">Claude Opus</a></li>
<li>Google's <a href="https://blog.google/technology/ai/google-gemini-next-generation-model-february-2024/#gemini-15">Gemini Advanced</a></li>
</ul>
<h4 id="user-content-bake-off">Bake Off</h4>
<p>I found that the various providers excelled at different things.  Anthropic's Claude was perfectly suited to the copy-pasta workflow, automatically collapsing a large paste into an attachment style inteface.  ChatGPT and Gemini neither collapsed nor auto-formatted the code input, leading to a bit messier of a UX out the gate:</p>
<p><img src="/img/blog/the-app-that-ai-made/gpt4-1.jpg" alt="alt text" title="GPT-4 Demo"></p>
<p>Gemini rendered pretty much exactly the same, although it also ultimately ran into a character count constraint at ~31,000 characters.  This was pretty limiting once the app was off the ground.</p>
<p><img src="/img/blog/the-app-that-ai-made/gemini-1.jpg" alt="alt text" title="Gemini Advanced Demo"></p>
<p>Gemini also seemed consistently keen to dive into suggesting changes before any features had been requested, although with some adjustment to the prompt this could be somewhat avoided.</p>
<p><img src="/img/blog/the-app-that-ai-made/gemini-2.jpg" alt="alt text" title="Gemini Advanced Demo"></p>
<p>Claude generally seemed to do the best with the prompt I had at producing complete changes without introducing regression issues, as well as properly responding "I'm in" at the start rather than diving into un-requested changes.  The larger the code base grew, the less this seemed to be the case - I ultimately wound up adding another reminder at the end of my prompt in later requests:</p>
<p><img src="/img/blog/the-app-that-ai-made/claude-2.jpg" alt="alt text" title="Claude Demo"></p>
<p>I frequently began hitting the Claude message throttle, which would reset every ~8hrs - this became my primary bottleneck as the features acculuated and codebase grew.  Claude 3 Opus showed itself to be hands-down the champ at consistently producing complete untruncated files and changes with few to no bugs or regressions.</p>
<p>Eventually, the codebase grew large enough that even Claude Opus would start to suggest changes before any features were described, just as Gemini had done.  It seemed the context window or at least the size of the prompt was the cause here, as it consistently happened over a certain line/character count.</p>
<p>In <a href="https://github.com/alexdredmon/crayeye/commit/5539cfb428770785779f3ba78599ed5972cb1e93">autodev prompt tweaks (5539cfb)</a>, I finally broke the prompt out into two parts - sandwiching the codebase with a <a href="https://github.com/alexdredmon/crayeye/blob/5539cfb428770785779f3ba78599ed5972cb1e93/flutter/lib/_autodev_prompt.txt">before</a> and <a href="https://github.com/alexdredmon/crayeye/blob/5539cfb428770785779f3ba78599ed5972cb1e93/flutter/lib/z_autodev_prompt_end.txt">after</a> prompt.  This seemed to resolve the issues with suggesting changes before features were requested, as well as ensuring more consistent adherence to the "complete files after these changes with no truncations" rule.</p>
<p>With the sandwich prompt in place, it was off to the races again - iterating quickly was again a breeze and feature requests readily turned into code.</p>
<h3 id="user-content-minimum-viable-product">Minimum Viable Product</h3>
<h4 id="user-content-the-good">The Good</h4>
<p>The MVP, which allowed me to add/edit prompts with location data interpolated in, came out quite usable and useful:</p>
<p><img src="/img/blog/the-app-that-ai-made/iphone-triple.jpg?v=3" alt="alt text" title="iPhone CrayEye Demo 1"></p>
<p>It supported interpolating the user's location values using the tokens <code>{location.lat}</code>, <code>{location.long}</code>, and <code>{location.orientation}</code> to represent their current latitude, longitude, and north/south/east/west orientation at the time of executing the prompt.</p>
<p>I initially assumed I migh need to use an API call like I do for <a href="https://whatsmyhood.com/">WhatsMyHood</a> to interpret the user's neighborhood from their latitude/longitude, but it turned out just providing the raw values to the LLM was sufficient - it was able to work out which neighborhood you were in just as well as Google Maps' API.</p>
<p>There were areas for improvement, such as improving the cramped "Add/Edit Prompt" dialog, but I could easily manage and share my prompts and test them in the field - and even save my favorite responses.</p>
<h4 id="user-content-the-bad">The Bad</h4>
<p>I was ready to share my app.  I prepared to test on Android and submit to the Google Play Store and Apple App Store.  That's when I first encountered my first serious setback - after setting up the <a href="https://docs.flutter.dev/get-started/install/macos/mobile-android?tab=download">Android dev tools</a> I fired up <code>flutter emulators</code> and tried to run my app on an Android emulator.  It was a no go - it turned out several packages I was using weren't compatible with my target Android SDK version, and after several attempts of getting an LLM to properly address I finally landed on a solution involving <a href="https://github.com/alexdredmon/crayeye/commit/f18c8b2e276f12b21f7e586f46d7581e34ceb1ed">dropping a dependency (f18c8b2)</a> (and in doing so, support for the <code>{location.orientation}</code> interpolation values in prompts).</p>
<h3 id="user-content-takeaways">Takeaways</h3>
<h4 id="user-content-advantages">Advantages</h4>
<ul>
<li>I was able to quickly make a feature-complete cross-platform MVP with minimal effort/input thanks to the force multiplier of modern LLMs - the initial MVP took ~10hrs of human work/input.</li>
<li>For my purposes, Claude Opus 3 did far and away the best job of consistently producing functional code without regressions</li>
</ul>
<h4 id="user-content-limitations">Limitations</h4>
<ul>
<li>Gemini was limited to building an app beyond ~31k characters (including all prompts)</li>
<li>ChatGPT had the largest tendency to introduce regression errors or ignore instructions to output complete untruncated files</li>
<li>Business limitations outpaced technical ones (i.e. Anthropic message throttle, initial App Store rejection)</li>
</ul>
<h4 id="user-content-areas-for-improvement">Areas for Improvement</h4>
<ul>
<li>The workflow utilized could obviously lend to further automation, particularly using autonomous agents (e.g. Autogen) to write and test changes with a human in the loop purely for requirements input and validation of acceptance criteria.</li>
<li>Code search and ability to map / use code maps or documentation would be ideal for larger projects</li>
<li>While the MVP took ~10hrs of hands-on input/work, this was spread across a number of days/weekends given the message cap for Claude 3 Opus.  By using the API rather than the web UI or otherwise circumventing message caps, the timeline to delivery could have been accelerated.</li>
</ul>
<h4 id="user-content-higher-level-language">Higher Level Language</h4>
<p>Large language models as they're used for code generation can be conceptualized similarly to the latest higher level language for development - just as the existence of Python has not supplanted all development in C, LLMs don't necessarily obviate 100% of lower level language development - even if it undeniably accelerates the ability to execute on said lower level development.</p>
<p>They do however take a big bite - a strikingly large and growing percentage of use cases can be handled exclusively by a modern foundational LLM <em>today</em> and that number is only going to go up.</p>
<h4 id="user-content-get-the-app">Get the App</h4>
<p><a href="https://crayeye.com/"><img src="/img/blog/the-app-that-ai-made/crayeye.jpg?v=3" alt="alt text" title="CrayEye"></a></p>
<ul>
<li><a href="https://www.crayeye.com/">CrayEye.com</a></li>
<li><a href="https://apps.apple.com/us/app/crayeye/id6480090992">Download for iOS</a></li>
<li><a href="https://play.google.com/store/apps/details?id=com.crayeye.app">Download for Android</a></li>
<li><a href="http://github.com/alexdredmon/crayeye">View Source on GitHub</a></li>
</ul>
<h3 id="user-content-post-photo">Post Photo</h3>
<p><a href="/img/blog/the-app-that-ai-made/bg.jpg"><img src="/img/blog/the-app-that-ai-made/bg.jpg" alt="alt text" title="Minolta XG1 / CineStill 800T / Onondaga Lake, NY during the 2024 total solar eclipse"></a></p>
<p><em>Minolta XG1 / CineStill 800T / Onondaga Lake, NY during the 2024 total solar eclipse</em></p>


Holistic engineering using large language models

alexandria redmon

home > blog