Exploring Query Understanding for Amazon Product Search
探索亚马逊产品搜索中的查询理解

A search is triggered exclusively in response to queries identified with a Product Search intent. We further leverage the task dependencies as a chain-of-thought ( CoT ) and structure the intent classification as the generation of class tokens together with the other two tasks, thereby streamlining the identification and handling of conversational queries.
仅在识别到具有产品搜索 意图的查询时，才会触发搜索。我们进一步利用任务依赖关系作为思维链（CoT），并将意图分类结构化为生成类 tokens，同时结合其他两个任务，从而简化对话查询的识别和处理。

Context switch prediction The Session and Conversation Understanding (SCU) model is designed to manage multi-turn conversational queries within a shopping session. The model triggers a search page refresh when a new topic is introduced, particularly for general knowledge questions. Therefore, we developed the context switch signal to identify whether the current question maintains the ongoing topic or shifts context. This context switch signal is
上下文切换预测 会话和对话理解（SCU）模型旨在管理购物会话中的多轮对话查询。当引入新主题时，模型会触发搜索页面刷新，特别是对于一般知识问题。因此，我们开发了上下文切换信号，以识别当前问题是否保持正在进行的主题或切换上下文。这个上下文切换信号是
Table 3: Search irrelevance rate (IRR) and sparse result rate (SRR) using original conversational queries and rewritten keywords.
表 3：使用原始对话查询和重写关键词的搜索无关率（IRR）和稀疏结果率（SRR）。

integrated into a unified response template, serving as a precursor for generating subsequent signals.
集成到一个统一的响应模板中，作为生成后续信号的前驱。

Question-to-keywords rewrite To seamlessly incorporate the conversational query understanding with current search experience, the CSU model finally generates the question-to-keywords (Q2K) signal. Specifically, the Q2K identifies the shopping intent and rewrites the conversational query, i.e., the question, into search keywords that align with the capabilities of the current search frameworks. The Q2K task is unique in its approach, diverging from conventional text summarization or named entity recognition (NER) techniques, as it adeptly captures shopping intents that may be implicit or indirectly expressed. In scenarios involving multi-turn sessions, the Q2K model plays a crucial role in addressing anaphoric questions, where queries in the latest turn refer back to entities mentioned in earlier interactions. The search is then triggered using the generated keywords, rather than the original conversational query. On real search traffic, we evaluate the search quality improvement on conversational queries brought by the Q2K signal. We quantify the search quality by computing the irrelevance rate and sparse result rate of search results derived from the original query (question) and rewritten keywords, individually. The scores shown in Table 3 indicates a significant improvement on the search quality introduced by the Q2K intervention.
问题到关键词重写 为了将对话查询理解无缝地融入当前的搜索体验，CSU 模型最终生成了问题到关键词（Q2K）信号。具体而言，Q2K 识别购物意图，并将对话查询（即问题）重写为与当前搜索框架能力相匹配的搜索关键词。Q2K 任务在其方法上是独特的，区别于传统的文本摘要或命名实体识别（NER）技术，因为它巧妙地捕捉到可能隐含或间接表达的购物意图。在涉及多轮会话的场景中，Q2K 模型在处理指代问题时发挥着关键作用，此时最新轮次的查询指向早期交互中提到的实体。然后，使用生成的关键词触发搜索，而不是原始的对话查询。在真实的搜索流量中，我们评估了 Q2K 信号对对话查询带来的搜索质量提升。我们通过计算源自原始查询（问题）和重写关键词的搜索结果的无关率 和稀疏结果率 来量化搜索质量。表 3 中显示的分数表明，Q2K 干预显著改善了搜索质量。

Product ranking models play a critical role in optimizing search results by utilizing behavioral signals such as clicks and purchases. However, these models tend to heavily rely on behavioral features, overlooking the importance of understanding the specific attributes within queries and products. This limitation often leads to challenges in achieving precise matches and ultimately hinders overall performance. For instance, a customer might search for "iPhone" but end up purchasing an "iPhone accessory". Such behavioral signals add certain bias to the ranking model and thus hurt the performance.

We acknowledge the significance of incorporating a query understanding system into the search process. By harnessing advanced named-entity recognition (NER) techniques [33], we can effectively extract and link queries to specific entities, thereby gaining a deeper understanding of customer intent. This approach empowers us to leverage the insights derived from the query understanding system for constructing ranking features and integrating them into the

Figure 2: Query Understanding based Ranking Features
ranking model. The ranking features based on query understanding align query attributes with product attributes, enabling more precise and contextually relevant matches. In this way, our goal is to bridge the gap between customer intent and product offerings, ensuring that users find exactly what they are looking for, thereby enhancing their overall shopping experience on our platform.

In this work, we introduce a framework for generating query understanding-based ranking features and assess the impact of these query understanding features on actual ranking performance. At a high level, the construction of ranking features from Query Understanding involves three main components.

We evaluate the impact of QU-based ranking features through product ranking (e.g., ) at Amazon Search, aiming to reorder the top-k products based on their relevance to the query intent. In this work, we choose as top 16 products are usually in the first page in Amazon Search. We first use query understanding to extract attributes from the product search queries [19, 20]. Next, we generate boolean features, such as "is product type a match" and "is brand a match," grounded in the attribute values of both queries

Table 4: Online A/B Testing Results on QU Ranking Feature: Relative Improvement in Amazon Ranking Performance.
and products. These boolean features are referred to as QU ranking features. There are also some attributes that are not product-related, such as query specificity (whether the query is broad, exploring, or looking for specific products) or query statistics, such as purchase rate or click rate. Subsequently, we train two learning-to-rank (LTR) models that are used at Amazon Search: one employs QU ranking features, while the other does not. All other features, configurations, and hyperparameters of these two models remain identical. To compare the two models, we utilize NDCG@16, representing the normalized discounted cumulative gain (NDCG) score for the top 16 products in the search results. We conducted these experiments on different ranking models and different data across six countries: United States, United Kingdom, India, Canada, Japan, and Germany. The relative improvement in NDCG@16 due to QU ranking features is presented in Table 4. On average, we observe an enhancement of in NDCG@16, affirming the effectiveness of QU ranking features. This demonstrated that the use of query understandingbased ranking features enables us to enhance ranking performance in the real production system like Amazon Search.

Today, when evaluating the product ranking models, we do so in an aggregated manner. For instance, we compute metrics like NDCG, HERO, etc [32] on the entire test dataset. However, this aggregated approach limits our insights into understanding the model's performance. We lack knowledge of how well the model performs on different types of queries, such as broad queries or spearfishing queries. Additionally, we are unaware of the specific categories where the model may underperform.

To address this limitation, we propose a solution using query understanding (QU) features to gain a deeper understanding of the search models. By incorporating segments over the data, we can analyze the model's performance on various query types and categories, providing valuable insights into its strengths and weaknesses. This approach will enable us to make more informed decisions in optimizing and refining the product ranking models based on their performance in specific scenarios (Fig. 3).

By utilizing this query segment framework, we can acquire the performance of product ranking models for various query segments. Consequently, we can engage in multi-task training for different query segments. This involves training the model on distinct objectives tailored to each query segment, with the goal of enhancing the overall performance of the ranking model.

Figure 3: The framework of using Query Understanding signals to segment the evaluation data and get the detailed evaluation result

Table 5: Product Ranking Model Tasks

Further more, we expand this framework to ranking model training progress. The usual next step after the query segment analysis is to create a separate loss function specifically for that query segments or increase a weight specifically for these low-performing query segments groups. For example, we can set a separate purchase task on the tail queries, i.e queries with low frequency to force the ranking model focus more on the purchase task when the query is a tail query. Similar action can be performed on other query segments for every task but it is exponentially time-consuming to do so on the combinations of all the tasks and all the query segments. Additionally, product ranking model is a multi-task learning model and how to tune the multi-task weights is usually a pain point to find the preferred the pareto front optimal point. More weights on one task, say purchase, means to aim for the optimal point with higher purchase score. Ideally we would want to maximize the revenue while maintaining relevance score by not dropping too much. So it becomes tricky to tune the task weights. Grid Search is a usual way of tuning the task weights, but with a finite number of tasks we have infinite choices of the task weights and that would take time and computation resources. Therefore, it would alleviate this pain point by incorporating the query segment analysis into the model training process and have the model dynamically tune the task weights based the performance of query segments at each step.

The overarching goal of this algorithm is to incorporate query segment analysis into the model training process by identifying low-performing query segments for each task and subsequently increasing the task weight for those specific segments. In cases of conflicting purchase tasks and relevance tasks, the model is designed to prioritize one task over the other to align with the company's objectives. The determination of low-performing query segments is accomplished through the measurement of task loss for that particular query segment within a support set, enabling the estimation of validation loss. This concept is inspired by the approach introduced in 'Learning to Weight' [23]. There are two primary methods for identifying low-performing query segments: comparing a segment's performance against its own historical data vertically, or evaluating its performance relative to its peers horizontally: 1) Worse performance compared within its history. If the support set loss for that query segments is more than last checkpoint. It means this task is defeated by other tasks and the model sacrifice this task in the favor of other tasks. 2) Low performing compared with other query segments. If the support set loss for that query segments is more than average task loss of other segments, then we should increase the task weight for that query segments because it is falling behind other query segments.

In this work, we design a strategy to update tasks after identifying the low-performing query segments. For each task , this strategy is executed at every steps:

In above equation, Loss denotes the loss function of the ranking model. controls the frequency of updating task weights per query segments. From our experience, we recommended to have large N , say 500 steps, to show the effect of task weights tuned in the previous checkpoint. controls how much more penalty we want to have on primary task weights if the support set loss of primary tasks get higher. Here .

In the previous published works, the multi-task weights are tuned together for the whole training dataset. They ignore the possibility that different groups of training data has different learning patterns. For example, high frequency queries is very easy to be learned with high purchase score whereas low frequency queries are struggling with purchase metrics. Our proposed algorithm tunes the task weight in a fine grained fashion that it would focus more weight on purchase metric for low frequency queries than high frequency queries.

The term inside Relu is to identify low-performing query segments by comparing itself within its own history. In our product ranking model training at Amazon Search, a common situation is that purchase task validation losses increases during the training while the relevance-related validation losses decreases which shows the competition between purchase task and relevance task and purchase task loses. Adding this term into the algorithm help alleviate such issue by focusing more purchase task. To solve this problem, we compare the support set losses during the training process, if the any of the task support set loss gets higher, we assume that it is losing to other tasks and thus increases its task weight to maintain the balance. For primary tasks, we increases it more than auxiliary tasks. controls how much more here.

The term inside max is the way of identifying low-performing query segments that is doing worse on primary task. Because the

Table 6: Evaluation Results

primary task is the main goal of this ranking model, those query segments perform worse on the primary task should focus more weight on primary task and sacrifice the auxiliary tasks. This term simulates the idea that when students in universities need to switch more time from elective courses to required courses when they don't perform as good as others in the required courses.

We want the to be not too small to see the effect of weight changes but also don't want it to be too high which would make the task weights fluctuates too much and make training performance unstable. Depending on the average change of loss value per N step. A good selection of would be

5.2.1 Results Discussion. We compare our methods with the fixed weight strategy for multi-task ranking model training, referred to as the 'Manual Grid Search' model. A comprehensive comparison of results between the 'Manual Grid Search' model and the query segment-based algorithm is provided in Table 6. Overall, our proposed algorithm achieves superior results compared to the 'Manual Grid Search' model while consuming fewer computational resources and less time. This efficiency is attributed to our algorithm's utilization of approximately choices, in contrast to the 'Manual Grid Search' model's evaluation of over 100 different task weight parameter selections. Additionally, our algorithm yields greater improvements in revenue-related tasks. Notably, the NDCG@16 model demonstrates a 150 bps improvement in the revenue task when compared to the 'Manual Grid Search' model. In terms of relevance tasks, the query segment-based ranking model showcases comparable performance to the 'Manual Grid Search' model, with an approximately 40 bps improvement.
5.2.2 Effect of . As discussed in section 5.1, controls the weight incremental for primary tasks when the support set loss is decreasing. If is too small, then primary tasks would need more steps to get enough attention. Additionally, setting to 0 would turn the ranking model into static weight. If is too large, the model could fluctuates over the optimal task weights. In our experiment, we observe that with non zero selection, we sees the primary tasks gets more weight to be learned and the validation loss starts dropping instead of increasing. With larger value, we see a steeper drop with the first 2000 steps, a better primary task performance but worse relevance task performance.To select an appropriate , this depends on the launching criteria of the ranking model, i.e. the lowest threshold of relevance loss a ranking model needs to have. In the experiments we run, we chose the equal to 10 which meets the bar of relevance score and a very good primary tasks performance.
Table 7: Query Segments Weight Analysis

5.2.3 Query Segments Weight Analysis. The overall query segment weights, using purchase task as an example, increases quickly in the beginning stage and then slows down, reaching the optimal weight in small steps. Other tasks follow the same pattern except that the auxiliary task weights would be decreasing to find the optimal weight since the ranking model would like to focus more on the purchase tasks. See Table 7 for the final query segment weights. Use query specificity as an example. The low query specificity has lower revenue task weight than the revenue task weight of high speficity and mid specificity. Because if the query is low specificity, it means the customers are very aware of the products they would like to buy and make orders once they find the desired products. Usually, the low specificity queries would bring the most revenue compared to mid specificity and high specificity.

The evolution of online shopping platforms, exemplified by giants like Amazon, has revolutionized the global marketplace, offering unparalleled services to billions worldwide. Unlike traditional web searches, product search engines possess distinctive attributes, predominantly characterized by succinct queries composed of product attributes within a structured search space. The significance of the query understanding component in product search cannot be overstated, yet empirical investigations into its impact within real-world product search engines remain limited.

In this study, we endeavored to address this gap by embarking on a comprehensive exploration of the influence of query understanding on Amazon Product Search. Over the course of a year, our journey led us to dissect the intricate interplay between query understanding and product ranking. Our findings underscore the indispensable role of query understanding in optimizing product search engines, paving the way for enhanced search experiences and heightened user satisfaction in the ever-expanding landscape of online commerce.

[1] Leif Azzopardi and Guido Zuccon. 2016. Advances in formal models of search and search behaviour. In Proceedings of the 2016 ACM International Conference on the Theory of Information Retrieval. 1-4.
[2] Keping Bi, Choon Hui Teo, Yesh Dattatreya, Vijai Mohan, and W Bruce Croft. 2019. Leverage implicit feedback for context-aware product search. arXiv preprint arXiv:1909.02065 (2019).
[3] Olivier Chapelle and Yi Chang. 2011. Yahoo! learning to rank challenge overview. In Proceedings of the learning to rank challenge. PMLR, 1-24.
[4] Nurendra Choudhary, Nikhil Rao, Sumeet Katariya, Karthik Subbian, and Chandan K Reddy. 2022. ANTHEM: Attentive hyperbolic entity model for product search. In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining. 161-171.
[5] Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2021. Domain-specific language model pretraining for biomedical natural language processing. Transactions on Computing for Healthcare (HEALTH) 3, 1(2021), 1-23.
[6] Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. 2020. Don't Stop Pretraining: Adapt Language Models to Domains and Tasks. arXiv preprint arXiv:2004.10964 (2020).
[7] Peter V Henstock, Daniel J Pack, Young-Suk Lee, and Clifford J Weinstein. 2001. Toward an improved concept-based information retrieval system. In Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval. 384-385.
[8] Sharon Hirsch, Ido Guy, Alexander Nus, Arnon Dagan, and Oren Kurland. 2020. Query reformulation in E-commerce search. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. 1319-1328.
[9] Haoming Jiang, Danqing Zhang, Tianyu Cao, Bing Yin, and Tuo Zhao. 2021. Named Entity Recognition with Small Strongly Labeled and Large Weakly Labeled Data. arXiv preprint arXiv:2106.08977 (2021).
[10] Weize Kong, Rui Li, Jie Luo, Aston Zhang, Yi Chang, and James Allan. 2015. Predicting search intent based on pre-search context. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval. .
[11] Mukul Kumar, Youna Hu, Will Headden, Rahul Goutam, Heran Lin, and Bing Yin. 2019. Shareable Representations for Search Query Understanding. arXiv preprint arXiv:2001.04345 (2019).
[12] Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics 36,4 (2020), .
[13] Minghan Li and Eric Gaussier. 2022. Bert-based dense intra-ranking and contextualized late interaction via multi-task learning for long document retrieval. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2347-2352.
[14] Zheng Li, Danqing Zhang, Tianyu Cao, Ying Wei, Yiwei Song, and Bing Yin. 2021. Metats: Meta teacher-student network for multilingual sequence labeling with minimal supervision. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 3183-3196.
[15] Heran Lin, Pengcheng Xiong, Danqing Zhang, Fan Yang, Ryoichi Kato, Mukul Kumar, William Headden, and Bing Yin. 2020. Light Feed-Forward Networks for Shard Selection in Large-scale Product Search. (2020).
[16] Tie-Yan Liu et al. 2009. Learning to rank for information retrieval. Foundations and Trends® in Information Retrieval 3, 3 (2009), 225-331.
[17] Bo Long, Jiang Bian, Anlei Dong, and Yi Chang. 2012. Enhancing product search by best-selling prediction in e-commerce. In Proceedings of the 21st ACM international conference on Information and knowledge management. 2479-2482.
[18] Yichao Lu, Ruihai Dong, and Barry Smyth. 2018. Why I like it: multi-task learning for recommendation and explanation. In Proceedings of the 12th ACM Conference on Recommender Systems. 4-12.
[19] Chen Luo, Rahul Goutam, Haiyang Zhang, Chao Zhang, Yangqiu Song, and Bing Yin. 2023. Implicit Query Parsing at Amazon Product Search. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval. .
[20] Chen Luo, William Headden, Neela Avudaiappan, Haoming Jiang, Tianyu Cao, Qingyu Yin, Yifan Gao, Zheng Li, Rahul Goutam, Haiyang Zhang, et al. 2022. Query attribute recommendation at Amazon Search. In Proceedings of the 16th ACM Conference on Recommender Systems. 506-508.
[21] Chen Luo, Vihan Lakshman, Anshumali Shrivastava, Tianyu Cao, Sreyashi Nag, Rahul Goutam, Hanqing Lu, Yiwei Song, and Bing Yin. 2022. Rose: Robust caches for amazon product search. In Companion Proceedings of the Web Conference 2022. 89-93.
[22] Yu A Malkov and Dmitry A Yashunin. 2018. Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs. IEEE transactions on pattern analysis and machine intelligence 42, 4 (2018), 824-836.
[23] Yuren Mao, Zekai Wang, Weiwei Liu, Xuemin Lin, and Pengtao Xie. 2022. MetaWeighting: Learning to Weight Tasks in Multi-Task Learning. In Findings of the Association for Computational Linguistics: ACL 2022. Association for Computational Linguistics, Dublin, Ireland, 3436-3448. https://doi.org/10.18653/v1/ 2022.findings-acl. 271
[24] Priyanka Nigam, Yiwei Song, Vijai Mohan, Vihan Lakshman, Weitian Ding, Ankit Shingavi, Choon Hui Teo, Hao Gu, and Bing Yin. 2019. Semantic product search. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2876-2885.
[25] Nish Parikh and Neel Sundaresan. 2011. Beyond relevance in marketplace search. In Proceedings of the 20th ACM international conference on Information and knowledge management. 2109-2112.
[26] Nita Patil, Ajay Patil, and BV Pawar. 2020. Named entity recognition using conditional random fields. Procedia Computer Science 167 (2020), 1181-1188.
[27] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. In NeurIPS EMC Workshop. arXiv:1910.01108 http://arxiv.org/abs/1910.01108
[28] Daria Sorokina and Erick Cantu-Paz. 2016. Amazon search: The joy of ranking products. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval. 459-460.
[29] Niek Tax, Sander Bockting, and Djoerd Hiemstra. 2015. A cross-benchmark comparison of 87 learning to rank methods. Information processing & management 51, 6 (2015), 757-772.
[30] Andrew Trotman, Jon Degenhardt, and Surya Kallumadi. 2017. The Architecture of eBay Search.. In eCOM@ SIGIR.
[31] Xuyang Wu, Alessandro Magnani, Suthee Chaidaroon, Ajit Puthenputhussery, Ciya Liao, and Yi Fang. 2022. A Multi-task Learning Framework for Product Ranking with BERT. In Proceedings of the ACM Web Conference 2022. 493-501.
[32] Tao Yang, Chen Luo, Hanqing Lu, Parth Gupta, Bing Yin, and Qingyao Ai. 2022. Can clicks be both labels and features? Unbiased Behavior Feature Collection and Uncertainty-aware Learning to Rank. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. .
[33] Danqing Zhang, Zheng Li, Tianyu Cao, Chen Luo, Tony Wu, Hanqing Lu, Yiwei Song, Bing Yin, Tuo Zhao, and Qiang Yang. 2021. Queaco: Borrowing treasures from weakly-labeled behavior data for query attribute value extraction. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management. 4362-4372.
[34] Yu Zhang and Qiang Yang. 2018. An overview of multi-task learning. National Science Review 5, 1 (2018), 30-43.