StackOverflow vs ChatGPT as the Tortoise and the Hare
Some People Smarter that us have argued that surviving the test of time marks the greatest validation of relevance.
Still being told today, Aesop’s Fables date to 564 BCE, which implies the story of the Tortoise and the Hare contain knowledge of great relevance.
Flush with some intense funding (11BN+) and some the fastest user growth of any modern product/company, ChatGPT (OpenAI) embodies the Hare.
Recently sold for not quite 2BN and experiencing a decline in traffic/engagement StackOverflow represents the Tortoise.
We’ve spent months asking ChatGPT reasonably general questions about ‘how to run x’ ‘what is the the code to do y’..
Time and time again, ChatGPT has shiny answers that seem amazing but are generally incomplete, cue the iterations on debugging the incompleteness.
Ironically, if you aren’t already an experienced programmer who has spent their formative years on stack overflow, you likely won’t know how to spot or debug ChatGPT’s incomplete answers.
We might categorize usage of ChatGPT as;
- amazingly fast input and feedback
- something out of scifi
- a bit misleading
- terrible for beginners attempting to solve advanced (non educational) problems
- terrible for any level learners seeking consensus around why an answer is the answer
Continuing to use Google/StackOverflow we find ourselves having to do a bit more work sifting through results. When we find an answer, it tends be an exact answer with code that is 99% reflective of the problem we started with and usually directly runnable in a console/program. More importantly we think that the process of looking at many answers including tangential answers that are the 60% quality answer are incredible at triangulate what we thought, what other people think and what is right. This process is paramount for long term learning.
We might categorize StackOverflow as;
- Slow and steady that wins the battle
- clunky to google for answers but downright dangerous to depend on google search
- lacking in immediate feedback and “ease of use” that is chatgpt
In Aesop’s Fable, the Hare loses the race because it got a bit malicous, wanting the Hare to feel ridiculous, so it took a nap, awaiting the Tortoise, but low the Tortoise finished the race before the Hare could spring back into action.
In our modern day allegory, OpenAi/Chatgpt can avoid becoming the hare, in our opinion, by working in a more transparent manner, by not assuming that riches, growth, funding and copyright impunity will always be so common. As written in the Fable, the Tortoise wins the race, but we think there are many reasons Stackoverflow could stil possibly lose.
Without knowing exactly what goes on inside the c-suite of Stackoverflow, speaking only as regular everyday users, StackOverflow needs to innovate at the intersection of a tool like Dash, ChatGPT and the traditional QA forum. We’d like to see something that works with us, next to the command line and is both as fast and easy as ChatGPT but produces the rock solid clarity that is the intersection of community feedback, official technical documentation and some LLM magic. StackOverflow’s current internal company QA forum tool is probably intersting, but when ChatGPT is converting every single person on the planet to be a some kind of user, StackOverflow needs to be thinking and executing on a much larger scale.
For a small look at our experience using ChatGPT as a coding support resource, checkout this link https://chat.openai.com/share/aebba213-cc8d-478b-95f0-e833dce36076
You’ll notice us asking a pretty straightforward question, if you dig into ChatGPTs answer it is not correct but reads like it is, we had to googl-overflow and iterate in the console about this one.
Be Fast, Be Slow, Just Be Deliberate.
references;
- https://blog.pragmaticengineer.com/are-reports-of-stackoverflows-fall-exaggerated/
- https://tracxn.com/d/companies/openai/__kElhSG7uVGeFk1i71Co9-nwFtmtyMVT7f-YHMn4TFBg/funding-and-investors
- https://en.wikipedia.org/wiki/Aesop%27s_Fables
- https://explodingtopics.com/blog/chatgpt-users
- https://en.wikipedia.org/wiki/Nassim_Nicholas_Taleb
