Online Transaction Processing Versus Online Analytical Processing
線上事務處理與線上分析處理

Online transaction processing (OLTP) is a type of informational system or application that processes online create, read, update, and delete (CRUD) transactions (see Chapter 2) in a real-time environment. OLTP systems are designed to support high levels of concurrency, which means they can handle a large number of transactions at the same time. They typically use a relational model (see Chapter 8) and are optimized for low latency, which means they can process transactions very quickly. Examples include point-of-sale applications, ecommerce websites, and online banking solutions.
線上事務處理（OLTP）是一種在即時環境中處理線上建立、讀取、更新和刪除（CRUD）交易（請參閱第 2 章）的資訊系統或應用程式。 OLTP 系統旨在支援高並發性，這意味著它們可以同時處理大量事務。它們通常使用關係模型（請參閱第 8 章），並針對低延遲進行了最佳化，這意味著它們可以非常快速地處理交易。例子包括銷售點應用程式、電子商務網站和網路銀行解決方案。

In an OLTP system, transactions are typically processed quickly and in a very specific way. For example, a customer’s purchase at a store would be considered a transaction. The transaction would involve the customer’s account being debited for the purchase amount, the store’s inventory count being reduced, and the store’s financial records being updated to reflect the sale. OLTP systems use a variety of DBMSs to store and manage the data, such as Microsoft SQL Server and Oracle.
在 OLTP 系統中，事務通常以非常特定的方式快速處理。例如，顧客在商店購物就屬於事務處理。該事務包括從顧客的帳戶中扣除購買金額，減少商店的庫存數量，以及更新商店的財務記錄以反映銷售情況。 OLTP 系統使用各種 DBMS 來儲存和管理數據，如 Microsoft SQL Server 和 Oracle。

OLTP systems are often contrasted with online analytical processing (OLAP) systems, which are used for data analysis and reporting to support business intelligence and decision making. OLAP systems are optimized for fast query performance, allowing end users to easily and quickly analyze data from multiple perspectives by slicing and dicing the data in reports and dashboards much faster than with an OLTP system. Think of an OLAP system as “write-once, read-many” (as opposed to CRUD). Often, multiple OLTP databases are used as the sources that feed into a data warehouse, which then feeds into an OLAP database, as shown in Figure 7-1. (However, sometimes you can also build an OLAP database directly from OLTP sources.)
OLTP 系統通常與線上分析處理（OLAP）系統形成對比，後者用於資料分析和報告，以支援商業智慧和決策制定。 OLAP 系統針對快速查詢效能進行了最佳化，與OLTP 系統相比，OLAP 系統能以更快的速度在報表和儀表板中對資料進行切分，從而使最終用戶能從多個角度輕鬆、快速地分析數據。將 OLAP 系統視為 "一次寫入，多次讀取"（與 CRUD 相反）。通常，如圖 7-1 所示，多個 OLTP 資料庫被用作資料倉儲的資料來源，然後資料倉儲再將資料輸入 OLAP 資料庫。 (不過，有時也可以直接從 OLTP 資料來源建立 OLAP 資料庫）。

Figure 7-1. OLAP database architecture
圖 7-1.OLAP 資料庫架構

An OLAP database is typically composed of one or more OLAP cubes. An OLAP cube is where data is pre-aggregated by the cube, meaning that it has already been summarized and grouped by certain dimensions; end users can quickly access it from multiple dimensions and levels of detail without having to wait for long-running queries to complete. Creating an OLAP cube generally involves using a multidimensional model, which uses a star or snowflake schema to represent the data. (Chapter 8 will discuss these schemas in more detail.)
OLAP 資料庫通常由一個或多個 OLAP 立方體組成。 OLAP 立方體是指資料已由立方體預先聚合，這表示資料已按特定維度匯總和分組；終端使用者可從多個維度和詳細程度快速存取數據，而無需等待長時間運行的查詢完成。建立 OLAP 立方體通常需要使用多維模型，該模型使用星形或雪花模式來表示資料。 (第 8 章將詳細討論這些模式）。

For example, a retail corporation might use an OLAP database with OLAP cubes to quickly analyze sales data. This pre-aggregated data is organized by time, allowing for the analysis of sales trends, whether yearly or daily or anywhere in between. The data is also sorted by location, which facilitates geographic comparisons to look at, for example, product sales in different cities. Furthermore, the data is categorized by product, making it easier to track the performance of individual items. If a regional manager needs to quickly review how a particular product category performed in their region during the last holiday season, the OLAP cube has already prepared the data. Instead of waiting for a lengthy query to sort through each individual sale, the manager finds the relevant data—sorted by product, region, and time—is readily available, accelerating the decision-making process.
例如，一家零售公司可能會使用具有 OLAP 立方體的 OLAP 資料庫來快速分析銷售資料。這種預先匯總的數據是按時間組織，可以分析銷售趨勢，無論是按年、按日或介於兩者之間的任何時間。數據也按地點分類，便於進行地理比較，例如，查看不同城市的產品銷售情況。此外，數據還按產品分類，以便於追蹤單一產品的業績。如果區域經理需要快速查看特定產品類別在其所在區域上一個假期季節的表現，OLAP 立方體已經準備好了資料。經理無需等待冗長的查詢來對每筆銷售進行分類，而是可以隨時獲得按產品、地區和時間分類的相關數據，從而加快決策過程。

You can use another, more recent model, called a tabular data model, that gives similarly fast query performance as an OLAP cube. Instead of a multidimensional model, a tabular data model uses a tabular structure, similar to a relational database table, to represent the data. This makes it more flexible and simpler than a multidimensional model. This book refers to the multidimensional model as an “OLAP cube” and the tabular model as a “tabular cube,” but please note that the term cube is used interchangeably in the industry to denote either of these models.
您可以使用另一種更新穎的模型，即表格資料模型，它能提供與 OLAP 立方體類似的快速查詢效能。與多維模型相比，表格資料模型使用類似關聯式資料庫表的表格結構來表示資料。這使得它比多維模型更靈活、更簡單。本書將多維模型稱為 "OLAP 立方體"，將表格模型稱為 "表格立方體"，但請注意，在行業中，立方體一詞可交替使用，以表示這兩種模型中的任何一種。

Table 7-1 compares the OLTP and OLAP design styles.
表 7-1 比較了 OLTP 和 OLAP 的設計風格。

OLAP databases and data warehouses (see Chapter 4) are related but distinct concepts and are often used in conjunction. Through a multidimensional or tabular model, OLAP databases provide a way to analyze the data stored in the DW in a way that is more flexible and interactive than executing traditional SQL-based queries on a DW that contains large amounts of data. Multidimensional models and tabular models are often considered semantic layers or models, which means that they provide a level of abstraction over the schemas in the DW.
OLAP 資料庫和資料倉儲（見第 4 章）是相互關聯但又截然不同的概念，兩者經常結合使用。透過多維模型或表格模型，OLAP 資料庫提供了一種分析儲存在資料倉儲中的資料的方法，這種方法比在包含大量資料的資料倉儲上執行傳統的基於SQL 的查詢更靈活，互動性更強。多維模型和表格模型通常被視為語義層或模型，這意味著它們為 DW 中的模式提供了一個抽象層。

Operational and Analytical Data
運行和分析數據

Operational data is real-time data used to manage day-to-day operations and processes. It is captured, stored, and processed by OLTP systems. You can use it to take a “snapshot” of the current state of the business to ensure that operations are running smoothly and efficiently. Operational data tends to be high volume and helps in making decisions quickly.
運行數據是用於管理日常運行和流程的即時數據。它由 OLTP 系統採集、儲存和處理。您可以用它來 "快照 "業務的當前狀態，以確保業務平穩且有效率地運作。營運數據往往是大容量數據，有助於快速做出決策。

Analytical data comes from collecting and transforming operational data. It is a historical view of the data maintained and used by OLAP/Tabular systems and DWs. With data analytics tools, you can perform analysis to understand trends, patterns, and relationships over time. Analytical data is optimized for creating reports and visualizations and for training machine learning models. It usually provides a view of data over a longer period than you get with operational data, often has a lower volume, and is generally consolidated and aggregated. Data is usually ingested in batches and requires more processing time than operational data.
分析資料來自業務資料的收集和轉換。它是 OLAP/Tabular 系統和 DW 所維護和使用的資料的歷史視圖。利用數據分析工具，您可以進行分析，了解一段時間內的趨勢、模式和關係。分析資料經過最佳化，可用於建立報表和視覺化以及訓練機器學習模型。與操作資料相比，分析資料通常能提供更長的資料視圖，資料量通常較小，而且一般都經過合併和匯總。資料通常是成批攝取的，需要比業務資料更多的處理時間。

In summary, operational data is used to monitor and control business processes in real time, while analytical data is used to gain insights and inform decision making over a longer period. Both types of data are essential for effective business management, and they complement each other to provide a complete view of an organization’s operations and performance.
總之，營運數據用於即時監控和控制業務流程，而分析數據則用於在較長時間內獲得洞察力並為決策提供資訊。這兩類數據對於有效的業務管理都是必不可少的，它們相輔相成，為組織的營運和績效提供了一個完整的視角。

Think of OLTP as the technology used to implement operational data, and of OLAP/Tabular and DWs as the technology used to implement analytical data.
將 OLTP 視為用於執行操作資料的技術，而將 OLAP/Tabular 和 DW 視為用於執行分析資料的技術。

Symmetric Multiprocessing and Massively Parallel Processing
對稱多處理與大規模平行處理

Some of the first relational databases used a symmetric multiprocessing (SMP) design, where computer processing is done by multiple processors that share disk and memory, all in one server—think SQL Server and Oracle. (This is pictured on the left side of Figure 7-2.) To get more processing power for these systems, you “scale up” by increasing the processors and memory in the server. This works well for OLTP databases but not so well for the write-once, read-many environment of a DW.
一些最早的關聯式資料庫使用對稱多處理（SMP）設計，即電腦處理由多個處理器完成，它們共享磁碟和內存，所有這些處理器都在一台伺服器中，如 SQL Server 和 Oracle。 (要提高這些系統的處理能力，可以透過增加伺服器中的處理器和記憶體來"擴展"。這對OLTP 資料庫很有效，但對DW 的"一次寫入，多次讀取"環境就不太適用了。

As data warehouses grew in popularity in the 1990s and started to ingest huge amounts of data, performance became a big problem. To help with that, along came a new kind of database design. In a massively parallel processing (MPP) design, the database has multiple servers, each with multiple processors, and (unlike in SMP) each processor has its own memory and its own disk. This allows you to “scale out” (rather than up) by adding more servers.
隨著資料倉儲在 20 世紀 90 年代的普及，並開始攝取大量數據，效能成為一個大問題。為了解決這個問題，一種新型資料庫設計應運而生。在大規模並行處理（MPP）設計中，資料庫有多個伺服器，每個伺服器有多個處理器，而且（與 SMP 不同）每個處理器都有自己的記憶體和磁碟。這樣，您就可以透過增加伺服器來 "向外擴展"（而不是向上擴展）。

MPP servers distribute a portion of the data from the database to the disk on each server (whereas SMP databases keep all the data on one disk). Queries are then sent to a control node (also called a name node) that splits each query into multiple subqueries that are sent to each server (called a compute node or worker node), as shown on the right in Figure 7-2. There, the subquery is executed and the results from each compute node are sent back to the control node, mashed together, and sent back to the user. This is how solutions such as Teradata and Netezza work.
MPP 伺服器將資料庫中的部分資料分發到每台伺服器的磁碟上（而 SMP 資料庫將所有資料保存在一個磁碟上）。然後，查詢被傳送到控制節點（也稱為名稱節點），控制節點會將每個查詢拆分成多個子查詢，然後傳送到每台伺服器（稱為計算節點或工作節點），如圖7- 2 右側所示。在那裡，子查詢被執行，來自每個計算節點的結果被傳送回控制節點，合併後再傳送給使用者。這就是 Teradata 和 Netezza 等解決方案的工作原理。

Figure 7-2. SMP and MPP database designs
圖 7-2.SMP 和 MPP 資料庫設計

By way of analogy, imagine your friend Fiona has a deck of 52 cards and is looking for the ace of hearts. It takes Fiona about 15 seconds, on average, to find the card. You can “scale up” by replacing Fiona with another friend, Max, who is faster. Using Max brings the average time down to 10 seconds—a limited improvement. This is how SMP databases work.
打個比方，假設你的朋友菲奧娜有一副 52 張的撲克牌，她正在找紅桃 A。菲奧娜平均需要 15 秒才能找到這張牌。您可以 "放大 "菲奧娜，換成另一位速度更快的朋友麥克斯。使用 Max 可以將平均時間縮短到 10 秒--這只是有限的改進。這就是 SMP 資料庫的工作原理。

Now imagine you “scale out” instead of up, by replacing Fiona with 26 people, each of whom has only two cards. Now the average time to find the card is just 1 second. That’s how MPP databases work.
現在想像一下，把菲奧娜換成 26 個人，每個人只有兩張牌，"擴大 "而不是 "增加"。現在，找到卡片的平均時間僅為 1 秒。這就是 MPP 資料庫的工作原理。

SMP and MPP databases started as on-prem solutions, and these are still prevalent today, but there are now many equivalent solutions in the cloud.
SMP 和 MPP 資料庫最初是內部部署的解決方案，如今仍很普遍，但現在雲端中也有許多類似的解決方案。

Kappa Architecture 卡帕建築事務所

As opposed to the Lambda architecture, which is designed to handle both real-time and batch data, Kappa is designed to handle just real-time data. And like the Lambda architecture, Kappa architecture is also designed to handle high levels of concurrency and high volumes of data. Figure 7-4 provides a high-level overview of the Kappa architecture.
Lambda 架構可同時處理即時數據和批次數據，而 Kappa 架構則只處理即時數據。與 Lambda 架構一樣，Kappa 架構也是為處理高並發和大量資料而設計的。圖 7-4 提供了 Kappa 架構的高層概覽。

The three key principles of the Kappa architecture are:
Kappa 架構的三大原則是

Real-time processing 即時處理

The Kappa architecture is designed for real-time processing, which means that events are processed as soon as they are received rather than being batch processed later. This decreases latency and enables the system to respond quickly to changing conditions.
Kappa 架構專為即時處理而設計，這意味著一旦收到事件，就會立即對其進行處理，而不是稍後再批量處理。這減少了延遲，使系統能夠對不斷變化的情況做出快速反應。

Single event stream 單一事件流

The Kappa architecture uses a single event stream to store all data that flows through the system. This allows for easy scalability and fault tolerance, since the data can be distributed easily across multiple nodes.
Kappa 架構使用單一事件流來儲存流經系統的所有資料。由於資料可以輕鬆分佈在多個節點上，因此可以輕鬆實現可擴展性和容錯性。

Stateless processing 無狀態處理

In the Kappa architecture, all processing is stateless. This means that each event is processed independently, without relying on the state of previous events. This makes it easier to scale the system, because there is no need to maintain state across multiple nodes.
在 Kappa 架構中，所有處理都是無狀態的。這意味著每個事件都是獨立處理的，無需依賴先前事件的狀態。這使得系統更容易擴展，因為無需在多個節點上維護狀態。

Figure 7-4. Overview of Kappa architecture
圖 7-4.Kappa 架構概覽

The layers in the Kappa architecture are exactly the same as in the Lamba architecture, except that the Kappa architecture does not have a batch layer.
Kappa 架構的層與 Lamba 架構的層完全相同，只是 Kappa 架構沒有批次層。

Some potential drawbacks of the Kappa architecture include:
Kappa 架構的一些潛在缺點包括

Complexity 複雜性

The Kappa architecture involves a single event stream and stateless processing, which can be more complex to implement and maintain than other architectures.
Kappa 架構涉及單一事件流和無狀態處理，與其他架構相比，其實作和維護更為複雜。

Limited batch processing
有限的批量處理

The Kappa architecture is designed for real-time processing and does not easily support batch processing of historical data. If you need to perform batch processing, you may want to consider the Lambda architecture instead.
Kappa 架構專為即時處理而設計，不易支援歷史資料的批次。如果需要執行批次處理，可以考慮使用 Lambda 架構。

Limited support for ad-hoc queries
對臨時查詢的支援有限

Because the Kappa architecture is designed for real-time processing, it may not be well suited for ad hoc queries that need to process large amounts of historical data.
由於 Kappa 架構是為即時處理而設計的，因此可能不太適合需要處理大量歷史資料的臨時查詢。

Overall, the Kappa architecture is an excellent choice for building distributed systems that need to handle large amounts of data in real time and that need to be scalable, fault tolerant, and have low latency. Examples include streaming platforms and financial trading systems. However, if you need to perform batch processing or support ad hoc queries, then the Lambda architecture may be a better choice.
總體而言，Kappa 架構是建立分散式系統的絕佳選擇，這些系統需要即時處理大量數據，並具有可擴展性、容錯性和低延遲性。例如串流媒體平台和金融交易系統。但是，如果您需要執行批次處理或支援臨時查詢，那麼 Lambda 架構可能是更好的選擇。

Note that the Lambda and Kappa architectures are high-level design patterns that can be implemented within any of the data architectures described in Part III of this book. If you use one of those architectures to build a solution that supports both batch and real-time data, then that architecture supports the Lambda architecture; if you use one of those architectures to build a solution that supports only real-time data, that architecture supports the Kappa architecture.
請注意，Lambda 和 Kappa 架構是高階設計模式，可在本書第三部分所述的任何資料架構中實現。如果使用其中一種架構建立同時支援批次和即時資料的解決方案，則該架構支援 Lambda 架構；如果使用其中一種架構建立僅支援即時資料的解決方案，則該架構支援 Kappa 架構。

Polyglot Persistence and Polyglot Data Stores
多語言持久性和多語言資料存儲

Polyglot persistence is a fancy term that means using multiple data storage technologies to store different types of data within a single application or system, based upon how the data will be used. Different kinds of data are best kept in different data stores. In short, polyglot persistence means picking the right tool for the right use case. It’s the same idea as the one behind polyglot programming, in which applications are written in a mix of languages to take advantage of different languages’ strengths in tackling different problems.
多語言持久性是一個花哨的術語，意思是根據資料的使用方式，在單一應用程式或系統中使用多種資料儲存技術來儲存不同類型的資料。不同類型的資料最好保存在不同的資料儲存中。簡而言之，多語言持久性意味著為正確的用例選擇正確的工具。這與多語言程式設計背後的理念相同，即混合使用多種語言編寫應用程序，以發揮不同語言在解決不同問題方面的優勢。

By contrast, a polyglot data store means using multiple data stores across an organization or enterprise. Each data store is optimized for a specific type of data or use case. This approach allows organizations to use different data stores for different projects or business units, rather than a one-size-fits-all approach for the entire organization.
相較之下，多語言資料儲存意味著在整個組織或企業中使用多個資料儲存。每個資料儲存都針對特定類型的資料或用例進行了最佳化。這種方法允許企業針對不同的專案或業務部門使用不同的資料存儲，而不是針對整個企業使用一刀切的方法。

For example, say you’re building an ecommerce platform that will deal with many types of data (shopping carts, inventory, completed orders, and so forth). Instead of trying to store all the different types of data in one database, which would require a lot of conversion, you could take a polyglot persistence approach and store each kind of data in the database best suited for it. So, an ecommerce platform might look like the diagram in Figure 7-5.
例如，您正在建立一個電子商務平台，該平台將處理多種類型的資料（購物車、庫存、已完成訂單等）。與其在一個資料庫中儲存所有不同類型的資料（這將需要大量轉換），不如採用多點持久性方法，將每種資料儲存在最適合它的資料庫中。因此，電子商務平台的外觀可能如圖 7-5 所示。

Figure 7-5. An ecommerce platform with a polyglot persistence design
圖 7-5.採用多重持久性設計的電子商務平台

This results in the best tool being used for each type of data. In Figure 7-5, you can see that the database uses a key-value store for shopping cart and session data (giving very fast retrieval), a document store for completed orders (making storing and retrieving order data fast and easy), an RDBMS for inventory and item prices (since those are best stored in a relational database due to the structured nature of the data), and a graph store for customer social graphs (since it’s very difficult to store graph data in a non-graph store).
這樣，每種類型的資料都能使用最好的工具。在圖7-5 中，您可以看到資料庫使用鍵值儲存來儲存購物車和會話資料（提供非常快速的檢索），使用文件儲存來儲存已完成的訂單（使儲存和檢索訂單資料變得快速而簡單），使用RDBMS 來儲存庫存和商品價格（因為這些資料的結構化性質最好儲存在關聯式資料庫中），使用圖儲存來儲存客戶社交圖（因為在非圖儲存中很難儲存圖資料） 。

This will come at a cost in complexity, since each data storage solution means learning a new technology. But the benefits will be worth it. For instance, if you try to use relational databases for non-relational data, the design can significantly slow application development and performance; using the appropriate storage type pays off in speed.
這將以複雜性為代價，因為每種資料儲存解決方案都意味著要學習一種新技術。但這樣做的好處是值得的。例如，如果試圖將關聯式資料庫用於非關聯式數據，這種設計會大大降低應用程式開發速度和效能；而使用適當的儲存類型則可以提高速度。

Summary 摘要

This chapter explored the architectural concepts and design philosophies that form the basis of effective data systems.
本章探討了構成有效資料系統基礎的架構概念和設計理念。

First, you learned about the two primary types of data processing systems: online transaction processing (OLTP) and online analytical processing (OLAP). OLTP systems are designed for fast, reliable, short transactions, typically in the operational databases that power daily business operations. In contrast, OLAP systems support complex analytical queries, aggregations, and computations used for strategic decision making, typically in a data warehouse. You then learned about the differences between operational and analytical data.
首先，您了解了兩種主要類型的資料處理系統：線上事務處理 (OLTP) 和線上分析處理 (OLAP)。 OLTP 系統設計用於快速、可靠、簡短的事務處理，通常用於支援日常業務運作的作業資料庫。相較之下，OLAP 系統支援用於策略決策的複雜分析查詢、聚合和計算，通常在資料倉儲中使用。然後，您了解了操作資料和分析資料之間的差異。

You also learned the differences between symmetric multiprocessing (SMP) and massively parallel processing (MPP) architectures. We then delved into two modern big data–processing architectures: Lambda and Kappa. Last, we explored the concepts of polyglot persistence and polyglot data stores, which promote using the best-suited database technology for the specific needs and workload characteristics of the given data.
您也了解了對稱多處理（SMP）和大規模平行處理（MPP）架構之間的差異。然後，我們深入研究了兩種現代大數據處理架構：Lambda 和 Kappa。最後，我們探討了多點持久性和多點資料儲存的概念，這些概念提倡針對給定資料的特定需求和工作負載特徵使用最適合的資料庫技術。

Starting with the next chapter, our focus will shift from data storage and processing to the principles and practices of data modeling: the crucial bridge between raw data and meaningful insights. The approaches to data modeling, such as relational and dimensional approaches and the common data model, serve as an underpinning structure that allows you to use and interpret data efficiently across diverse applications.
從下一章開始，我們的重點將從資料儲存和處理轉向資料建模的原則和實踐：原始資料和有意義的見解之間的重要橋樑。資料建模的方法，如關係和維度方法以及通用資料模型，是一種基礎結構，可以讓你在各種應用中有效地使用和解釋資料。

As we delve into these topics, you’ll see how data modeling lets you use the storage and processing solutions we’ve studied efficiently, serving as a blueprint for transforming raw data into actionable insights.
隨著我們對這些主題的深入探討，您將看到資料建模如何讓您有效率地使用我們所研究的儲存和處理解決方案，並將其作為將原始資料轉化為可操作見解的藍圖。