Microsoft Patents Big Brother AI To Monitor Everything You Do In Windows And Feed It To Bing For Search Results

It looks as though Microsoft stands to garner more criticism if its recent patent filing comes to life in a production software product. The title of the company's patent filing is “Query Formulation Via Task Continuum” and it aims to make it easier for apps to share data between them, in real-time, so that the user can make more informed decisions when performing searches.

Microsoft feels that the current software model, in which applications are self-contained within their own silos, is detrimental to productivity and potentially slows the user down. “The first application does not provide the browser implicit hints as to what the user might be seeking when there is a switch from the first application to the second application,” writes Microsoft in the patent filing. “The user perceives tasks in the totality. However, since applications are typically disconnected, and not mediated in any way by the operating system, the computing system has no idea as to the overall goal of the user.”

The Redmond, Washington-based giant highlights the example of a user researching the topic of dancing by taking notes in OneNote, and then using Internet Explorer to search for different styles of dance. These two apps typically wouldn’t communicate with one another, but Microsoft sees a better way forward.

To combat this disconnect, Microsoft has devised a way to facilitate better communications between apps through the use of what it calls a “mediation component”. This is Microsoft’s all-seeing-eye that monitors all textual input within apps (including text embedded in photos and digital fingerprints from songs currently playing) to intelligently decipher what the user is trying to accomplish. All of this information could be gathered from apps like Word, Skype, or even Notepad by the Mediator and processed. So when the user goes to, for example, the Edge web browser to further research a topic, those contextual concepts are automatically fed into a search query.

The patent filing describes:

The search engine (e.g., Bing and Cortana) uses contextual rankers to adjust the default ranking of the default suggested queries to produce more relevant suggested queries for the point in time. The operating system, comprising the function of mediation component, tracks all textual data displayed to the user by any application, and then performs clustering to determine the user intent (contextually).

Microsoft says that this in effect provides faster and more relevant searchers to users. However, to some people this likely sounds rather creepy and big brother-esque. The Mediator would in essence keep track of everything you type and interact with in the OS and stockpile it in real-time to data-dump into Bing. Microsoft does provide the disclaimer, however, that personally identifiable information is not transferred to Bing (only high-level contextual concepts are transmitted).

Microsoft also describes how this would work with a full suite of Microsoft-backed products:

For example, in a Windows OS ecosystem of devices, which can include two or more Windows devices such as a Windows Phone, a Surface Pro tablet, and Xbox, etc., the OS of a given device gathers and monitors the information from the non-OS applications engaged on the given device, and communicates with the operating systems of the other devices to enable the OS of each device to have a “global awareness” of the contexts of the other ecosystem devices. These separate device contexts can then be computed to derive a global user context of the ecosystem.

The company says that its Mediator can be introduced as an optional module that can be installed on an operating system like Windows 10, or it can be directly built-in to the operating system. If it’s the former, most folks likely won't have any problems with this new Mediator. It could also potentially be the foundation of a powerful platform for contextually aware computing. Then again, if it's built in and not optional feature, we're sure there will be plenty of people looking for a kill switch for this functionality.