Or if you prefer to not sign up to our Slack, here's a demo video showing the same thing: https://youtu.be/fPhP1325RkI
We’re in the process of making everything open source (there are some contractual issues we’re working through), but our client side code and basic infra is here: https://github.com/ai-sidekick/sidekick.
Providing technical support to developers has been expensive for companies because they need to hire skilled engineers to do it. We’ve seen community support channels with a 2000:1 ratio of developers to support engineers - there’s no way every question will get answered. We built Sidekick to make this much easier. It’s particularly helpful for open-source companies/projects because many OSS communities have a lot of people asking questions, but hardly anyone helping troubleshoot.
We integrate with Slack and Discord, since that’s where developer support is already happening. On the backend, we use Weaviate to index the data and OpenAI’s text-davinci-3 model to generate responses.
In addition to answering questions, Sidekick can also update .md files automatically with new information. When someone reacts to a message in Slack with the emoji, Sidekick will use Weaviate to find the part in your documentation that’s most related to the message, then use GPT3 to merge the new info into the documentation. Finally, it will submit a pull request on Github with the changes. We saw that devrel teams are already making product announcements and helping users troubleshoot common issues in the community, so we built this feature to save them even more time.
We use GPT for generating the responses and new documentation, but are relying less and less on it after learning that you hit a ceiling on answer quality very quickly by using only GPT and prompt engineering techniques. Here’s some of what we learned trying to prevent hallucinations in our answers: https://medium.com/@jfan001/how-we-cut-the-rate-of-gpt-hallu...
What we found makes a much bigger difference is the breadth and quality of the content you can search through, which is why we now rely a lot more on cleaning and annotating data, which yields far better results when combined with prompt chaining. For example, instead of naively chunking data into 1000 token blocks, we parse the markdown into semantically meaningful sections (e.g. paragraphs, lists, code blocks) and tag the content with the header name and document name so it’s more likely to surface for searches that are match for the section it’s from, even if it doesn’t exactly match the content in that chunk.
One fun thing we also learned is that when Sidekick gets added to a #help channel, people who otherwise wouldn’t ask questions start using it. It turns out, there are a lot of “lurkers” who come to these channels to find answers, but don’t want to bother anyone with their issue. Adding a tool that they can get answers from instantly brings these people out into the community, giving founders and community managers an opportunity to reach out to them.
To summarize, Sidekick 1) saves support engineers time, 2) keeps the docs up to date and 3) helps engage developers in the community. Long-term we want to provide an analytics product on top of Sidekick so companies can understand how their product is being used and where there are opportunities to add more value to their customers (and charge more money for it).
We’d love to hear from the HN community about this product! Do you think using a tool to search through and update developer docs from Slack would save you time?
by Andrey
Azimov