If you’re a Technical Writer migrating to a new Component Content Management System, you know it’s a massive undertaking. Our team is currently preparing for such a migration, which involves:

• Moving all content into a new system.
• Converting unstructured content into structured content.
• Shifting from features-based documentation to topic-based user documentation.

This is a simplified view, focusing on my team’s role in the process. There are broader impacts on other teams, like Learning, but for now, we’re prioritizing content migration, as it’s foundational to the entire project. I confirmed that starting with pilot content will allow others to begin the process sooner. But, I digress.

Last week, I was tasked with breaking down two of our longest and shortest guides into topic-based documentation. After completing this, I was asked to run the guides through an in-house GPT model and compare the results to my manual work. Here’s what I observed after running the analysis and fine-tuning the GPT model.

Pros: For one guide, ChatGPT created two concept topics that weren’t in the original guide but were useful and worth considering.

Cons:

• Information retention: ChatGPT often drops important details, leaving only the bare minimum. Examples:
  • Short descriptions lacked context.
  • Procedural steps were missing key explanations.
  • Important scenarios were overlooked.
• Inability to “think” critically: ChatGPT struggles to determine when information should be excluded. For example, in one guide, a table had two redundant columns that I manually removed. ChatGPT couldn’t make this judgment.

Final thoughts

ChatGPT is a promising tool, but it’s in its early stages. While it can accelerate tasks like writing short descriptions, generating concepts, and validating style guide compliance, it falls short in more complex tasks like breaking down product guides, where critical thinking is essential. In many cases, you’ll end up with skeleton information that lacks depth.

After presenting my findings, my team requested that I explore how we can create a repeatable process using both technology and manual review to improve efficiency. I’m drafting a solution, but I’ll wait to share more once we’ve tested its effectiveness.

In a recent webinar, The Future of AI: Structure Content is Key, a timely quote resonated with my findings. When asked, “How can organizations quickly convert unstructured content into structured content at scale?” the answer was clear: “It probably can’t be done quickly. You have technical debt and structured content is more interesting or enriched than unstructured. Quickly and at scale don’t go together in this realm.”