Because data potentially reflect the design intent of the data obtainers, there is an inherent risk of false hypothesis testing in the analysis of third-party data that does not account for design intent or bias.

There is, therefore, a need for a mechanism to support data design that does not rely solely on experience and intuition, but understands the differences and diversity brought about by the human cognitive model of observing and dataing a dynamic world.

Unexplored data refers to potential data in areas not directly connected to our dynamically changing perceptions and knowledge. For example, the classification is shown in the figure on the left. Unexplored data refers specifically to potential data in areas where we have data but do not know what events can be observed, and where there is no data and the events we wish to observe are unknown or known.

Data origination is an approach to data design, acquisition, and utilization that goes back to the human act of observation, the origin of data, for such unexplored data. Variable Quest and TEEDA are tools that support the design and origination for such unexplored data.

The methodology of how to observe events that have not yet been converted into data is highly individualistic, and no systematic knowledge exists. Therefore, it is important to have a precise data design and its design support methodologies to determine the purpose of data collection and how to utilize the data.

In this study, we focused on the variables (the data attributes that make up the data), which are the main components of the data. We implemented Variable Quest (VQ), a system to estimate a set of variables from an summary of data in which the variables are unknown. By considering the two models of "data similarity" and "variable co-occurrence," VQ makes it possible to supplement variables that are missing in one set of data with variables that appear in other sets of data.

Data catalogs and portal sites are increasing the opportunities for users to learn about the third-party data. However, information on what kind of data users want and for what purpose (Call for Data) is not sufficiently shared.

TEEDA is a web application that matches data providers and users based on the data information they want (data requests) and the data they can provide (providable data). TEEDA supports encounters between data providers and users by exchanging information on what kind of data users want, in addition to one-way information provision by data holders.

TEEDA is an acronym for "Treasuring Every Encounter of Data Affairs.

This is one of the cases where data design and origination were practiced to address the problem of how unknown events should be observed and converted into data, which became apparent in the COVID-19 disaster.

Infection countermeasures due to human-object contact (people touching something) have been limited to measures such as encouraging hand washing, sterilization of hands with chemicals, and disinfection of objects, with little data leading to effective infection countermeasures. The workshop using TEEDA revealed that there is a high need to "examine the locations and objects that are frequently contacted and come up with guidelines for disinfection locations, priority disinfection items, and contact behavior in locations and vehicles. Furthermore, using Variable Quest, we designed "contact behavior data for people and objects" in the COVID-19 Disaster, and cooperated with the Cabinet Secretariat's "COVID-19 AI and Simulation Project" to actually acquire and analyze the data. As a result, we obtained guidelines for disinfectant locations, priority disinfected objects, and contact behavior at locations and in vehicles.

With the rapid advancement of data distribution and the growing demand for buying and selling data in the marketplace, there is great anticipation for new knowledge discovery and value creation by combining heterogeneous datasets. However, it is important to note that existing data was rarely designed or collected with interoperability. While linking disparate data can enhance our understanding of target phenomena and reveal novel insights, it also increases complexity and the risk of diminishing the explainability and persuasive power of analytical results.

Therefore, a pressing challenge is to simultaneously search for and recommend data that are well suited for integration with one’s own assets, while also establishing data integration and analysis methods that account for the characteristics of each combined dataset. Moreover, there is a growing need for techniques that assess the reliability of analyses based on heterogeneous data linkages and make their results explainable to stakeholders.

Data platforms for exchanging heterogeneous datasets have emerged, but most services still rely on general information retrieval methods applied to metadata written in natural language, which leads to challenges such as incomplete, inconsistent or unreliable metadata and insufficient information volume. To address these issues, our study proposes a metadata-less dataset embedding model based on tabular language models (TaLMs) that enhances search and recommendation functions by using the actual data. Specifically, we apply metric learning to HyTrel, one of the TaLMs, to train a dataset embedding model, and we then perform domain adaptation through additional training on GitTables and Kaggle datasets together with their associated metadata. Experimental results demonstrate that our metadata-less approach achieves search and recommendation performance in topic estimation and human evaluation tasks that is equivalent to or better than methods using metadata.

Previous research has depended heavily on metadata, which is prone to variable‐naming inconsistencies and misalignment with the actual data, and the cost of creating metadata for data publication or sale has been high. Our method is expected to support the development of systems capable of accurately exploring both internal and external datasets without relying on metadata.

Not only are organizations increasingly designing, generating, and using data internally, but they are also more often leveraging datasets created by other companies and combining them with their own. At the same time, challenges have emerged around integrating and jointly using multiple, heterogeneous datasets.

In this study, we propose a novel embedding method that harnesses graph structures encoding each dataset’s contextual information, and we apply it to the tasks of schema matching and entity resolution across datasets with diverse characteristics. Building on graph-based dataset embedding techniques, we construct a four-part graph for each table—consisting of rows, cells, variables, and variable semantics—and generate sequences from this graph to learn embedding representations. In experiments using both benchmark datasets and real operational datasets, our approach outperformed existing methods on both schema matching and entity resolution tasks.