In Part I of my two-part series on avoiding conflicts and delays in AI development, I introduced the first three steps of Intuit’s AI playbook. Today, I’ll walk through the last three steps. When put together, all six steps will help you integrate AI into your product, thus delivering improved customer experiences and customer benefits, improving your efficiency, accelerating your customer support, ensuring the security of your data and products, and more.
在關於避免人工智慧開發中的衝突和延遲的兩部分系列的第一部分中，我介紹了 Intuit 人工智慧手冊的前三個步驟。今天，我將逐步完成最後三個步驟。綜合起來，所有六個步驟將幫助您將人工智慧整合到您的產品中，從而改善客戶體驗和客戶利益、提高效率、加速客戶支援、確保數據和產品的安全等等。
Last three steps to avoiding conflicts and delays in AI development
避免人工智慧開發中的衝突和延誤的最後三個步驟
We were able to create Intuit’s comprehensive AI playbook with help from Intuit’s fraud prevention mission team and Intuit® AI team members. It provides us with a clear, step-by-step process (integrated into the wider planning and development cycle) for avoiding AI development conflicts and delays. The playbook allows us to align our teams and plan efficiently, and it can help your team do the same.
在 Intuit 詐欺防制任務團隊和 Intuit® AI 團隊成員的幫助下，我們得以創建 Intuit 全面的 AI 手冊。它為我們提供了一個清晰的逐步流程（整合到更廣泛的規劃和開發週期中），以避免人工智慧開發衝突和延誤。這本行動手冊使我們能夠協調我們的團隊並有效地進行計劃，它也可以幫助您的團隊做同樣的事情。
Before we look at the last three steps in
the playbook, let’s do a quick, high-level review of the first three steps.
在查看手冊中的最後三個步驟之前，讓我們先對前三個步驟進行快速、進階的回顧。
Step one follows an established intake process (led by the product manager (PM) of the mission team) in which the internal customer defines the problem using the Intuit Design for Delight (D4D) template. The intake requires the input of key performance indicators (KPIs) and the impact of the problem using numbers, success metrics, and how to measure them. The impact estimation is used to prioritize the AI initiative against other initiatives.
第一步遵循既定的接收流程（由任務團隊的產品經理 (PM) 領導），其中內部客戶使用 Intuit Design for Delight (D4D) 範本定義問題。攝取量需要輸入關鍵績效指標 (KPI) 以及使用數字、成功指標以及如何衡量問題的影響。影響評估用於確定人工智慧計劃相對於其他計劃的優先順序。
Steps
two and three
involve setting up the specific mission team for the prioritized AI initiative.
The team works on the solution design, design review/sign off, and quarterly
commitment planning, where the wider mission team completes a capacity review
(through a defined quarterly planning process) of all the prioritized requests
and determines if the AI initiative makes the cut for the quarter. If it does,
then it’s time to move on to step four.
第二步和第三步涉及為優先人工智慧計畫建立特定的任務團隊。該團隊致力於解決方案設計、設計審查/簽署和季度承諾規劃，其中更廣泛的任務團隊完成對所有優先請求的能力審查（通過定義的季度規劃流程），並確定人工智能計劃是否成功對於本季度。如果是，那麼是時候進入第四步了。
4. Quarterly project planning
4. 季度專案計劃
Once an AI initiative makes the cut and becomes a commitment for the quarter, a kickoff meeting will need to be set up with all the relevant stakeholders, including PD (product developer), data scientist (DS), program manager/technical program manager (PM/TPM), and partner teams. Using the design, each PD team will break the requirements into tasks (Epic level) and documents these tasks into Jira stories
一旦人工智慧計畫獲得批准並成為本季的承諾，就需要與所有相關利益相關者召開啟動會議，包括 PD（產品開發人員）、資料科學家（DS）、專案經理/技術專案經理（ PM/TPM）和合作夥伴團隊。使用該設計，每個 PD 團隊將把需求分解為任務（史詩級別），並將這些任務記錄到 Jira 故事中
During this stage, which is estimated to take 1-2 weeks, it’s up to the project driver (usually the PM/TPM) to communicate the timelines and the definition of “done” to all relevant stakeholders.
在此階段（預計需要 1-2 週），由專案推動者（通常是 PM/TPM）向所有相關利害關係人傳達時間表和「完成」的定義。
5. Project execution 5. 項目執行
It’s now time to execute the project. Project execution is made up of five stages:
現在是執行該專案的時候了。專案執行分為五個階段：
5a: DS research and development (4-16 weeks). This stage involves data collection, exploration, experimentation, feature generation, and model development. The mission-based team would meet weekly to monitor progress and ensure that every change in design, scope, or timelines is communicated to all stakeholders. The DS communicates model results and progress, and reiterates the model/data based on policy feedback until achieving required performance. During this period, the DS team will experiment with different types of solutions and will work with PM/policy/analytics to evaluate their effectiveness. Keep in mind, this might change the research and development estimated time period. At the end of this stage, the DS team will share the model and feature details and performance evaluation with the mission team.
5a：DS 研究和開發（4-16 週）。此階段涉及資料收集、探索、實驗、特徵生成和模型開發。基於任務的團隊將每週召開一次會議，以監控進度情況，並確保將設計、範圍或時間表的每項變更傳達給所有利害關係人。 DS 傳達模型結果和進度，並根據政策回饋重申模型/數據，直到達到所需的效能。在此期間，DS 團隊將嘗試不同類型的解決方案，並與 PM/策略/分析部門合作評估其有效性。請記住，這可能會改變研發的預計時間段。在此階段結束時，DS 團隊將與任務團隊分享模型和功能細節以及效能評估。
5b: Model implementation to silent release (2-7 weeks). In this stage, the MLE (machine learning engineer) collaborates with DS to develop production features, backfill the features with historical data, and finalize model production code. The PD integrates the call to the model from the product at the required checkpoint. Finally, required dashboards are built to fulfill model monitoring requirements.
5b：模型實施到靜默發布（2-7 週）。在這個階段，MLE（機器學習工程師）與DS合作開發生產特徵，以歷史資料回填特徵，最後確定模型生產程式碼。 PD 在所需的檢查點整合從產品到模型的呼叫。最後，建立所需的儀表板來滿足模型監控要求。
5c: Model monitoring (ongoing). This is where we start monitoring model performance in production. It is important for the DS team to define standard monitoring requirements to make sure we are alerted in case of model performance deterioration or any malfunction. Main items we monitor include: score distribution, feature health, flag rate, performance [model defined business KPIs (analytics)], service level agreement, and timeout rate.
5c：模型監控（正在進行中）。這是我們開始監控生產中模型效能的地方。對於 DS 團隊來說，定義標準監控要求非常重要，以確保我們在模型效能惡化或出現任何故障時收到警報。我們監控的主要項目包括：分數分佈、功能運作狀況、標記率、效能[模型定義的業務 KPI（分析）]、服務等級協定和逾時率。
5d: Sign off process (at the end of the silent period, right before project release). During the silent period, before the action mode release, the PM initiates and completes predefined signoff process with all the relevant stakeholders based on the model performance analysis during the signoff period.
5d：簽署過程（在靜默期結束時，專案發布之前）。在靜默期，在行動模式發布之前，PM 根據簽核期間的模型效能分析，與所有相關利害關係人啟動並完成預先定義的簽核流程。
5e: Silent period, policy development, model release, and retrospective (4-12 weeks).
5e：靜默期、政策制定、模型發布和回顧（4-12 週）。
Policy
development—After a silent period of 4-12 weeks
(depends on label maturation period, policy decision and sample size),
policy/PM/DS makes the analysis on the model scores and defines the strategy
under which the model would go to action mode.
政策制定－經過 4-12 週的靜默期（取決於標籤成熟期、政策決策和樣本量），政策/PM/DS 對模型分數進行分析，並定義模型將採取行動的策略模式。
Model
release—PD transitions the model to action mode, with
DS, MLE, policy and analytics monitoring the process in the first few hours. A
dedicated technical product manager would manage the release stages and provide
a single point of contact to the release.
模型發布－PD 將模型轉換為行動模式，DS、MLE、策略和分析在最初的幾個小時內監控該過程。專門的技術產品經理將管理發布階段並為發布提供單點聯繫。
Retrospective—1-2 weeks after release, the driver of the mission team sets a
retrospective to learn what worked well and what didn’t work well and gain
insights for next projects.
回顧－發布後 1-2 週，任務團隊的負責人會進行回顧，以了解哪些內容效果良好，哪些內容效果不佳，並為下一個專案獲得見解。
6. RTB – Model upkeep and incident management
6. RTB – 模型維護與事件管理
We’ve finally reached the final step.
我們終於到了最後一步。
- 6a: Model retraining (Ongoing according to model production performance). When the model deteriorates beyond a certain threshold (defined by the policy/PM/DS team at model release), the DS team should work to retrain the model in production, and PM/policy/DS should analyze the model results and update the policy/strategy as necessary. Phases 3-5 would repeat as necessary in the model retrain process for the new model version.
6a：模型再訓練（根據模型生產性能進行中）。當模型惡化超過某個閾值（由策略/PM/DS團隊在模型發佈時定義）時，DS團隊應努力在生產中重新訓練模型，PM/策略/DS應分析模型結果並更新策略/ 必要時的策略。在新模型版本的模型重新訓練過程中，將根據需要重複第 3-5 階段。
- 6b: Incident management (approximately one week). Occasionally, we experience incidents including feature anomaly and/or platform timeout. RCA should be written in the case of an incident. If the problem is in the PD code, then PD should be the RCA drivers. If the problem is in the DS code, then DS are the RCA drivers. The analytical team needs to quantify the business impact.
6b：事件管理（約一週）。有時，我們會遇到包括功能異常和/或平台逾時在內的事件。發生事故時應寫 RCA。如果問題出在 PD 程式碼中，則 PD 應該是 RCA 驅動程式。如果問題出在 DS 程式碼中，則 DS 是 RCA 驅動程式。分析團隊需要量化業務影響。
Six steps to AI-development confidence
增強人工智慧發展信心的六步曲
Now that you know Intuit’s six steps to
avoiding conflicts and delays in AI development, I hope you’re feeling
confident that you and your team can efficiently integrate AI solutions into
your product. It is by no means an easy or quick process, but it is a very
doable one if you define your team’s roles and responsibilities, follow the six
steps, and, as you learn what works and doesn’t work for you, make
improvements.
現在您已經了解了 Intuit 避免人工智慧開發中的衝突和延遲的六個步驟，我希望您對您和您的團隊能夠有效地將人工智慧解決方案整合到您的產品中充滿信心。這絕對不是一個簡單或快速的過程，但如果您定義團隊的角色和職責，遵循六個步驟，並且當您了解什麼對您有用、什麼不適合您時進行改進，那麼它是一個非常可行的過程。
I’ve learned many lessons leading AI teams, and I expect that you’ll learn quite a few yourself as you start your own journey. With AI redefining apps, you’re well on your way to being ahead of other businesses that have yet to consider utilizing AI.
我在領導人工智慧團隊方面學到了很多經驗教訓，我希望您在開始自己的旅程時也能學到很多東西。透過人工智慧重新定義應用程序，您將遠遠領先其他尚未考慮利用人工智慧的企業。