Unstructured data

Unstructured data (or unstructured information) refers to information that either does not have a pre-defined data model or is not organized in a pre-defined manner. Unstructured information is typically text-heavy, but may contain data such as dates, numbers, and facts as well. This results in irregularities and ambiguities that make it difficult to understand using traditional programs as compared to data stored in fielded form in databases or annotated (semantically tagged) in documents.

In 1998, Merrill Lynch cited a rule of thumb that somewhere around 80-90% of all potentially usable business information may originate in unstructured form. This rule of thumb is not based on primary or any quantitative research, but nonetheless is accepted by some.

IDC and EMC project that data will grow to 40 zettabytes by 2020, resulting in a 50-fold growth from the beginning of 2010. Computer World states that unstructured information might account for more than 70%–80% of all data in organizations.

Background

The earliest research into business intelligence focused in on unstructured textual data, rather than numerical data.[1] As early as 1958, CS researchers like H.P. Luhn were particularly concerned with the extraction and classification of unstructured text.[1] However, only since the turn of the century has the technology caught up with the research interest. In 2004, the SAS Institute developed the SAS Text Miner, which uses SVD to reduce a hyper-dimensional textual space into smaller dimensions for significantly more efficient machine-analysis.[2] The mathematical and technological advances sparked by machine textual analysis prompted a number of business to research applications, leading to the development of fields like sentiment analysis, voice of the customer mining, and call center optimization.[3] The emergence of Big Data in the late 2000s led to a heightened interest in the applications of unstructured data analytics in contemporary fields such as predictive analytics and root cause analysis.[4]

Issues with terminology

The term is imprecise for several reasons:

  1. Structure, while not formally defined, can still be implied.
  2. Data with some form of structure may still be characterized as unstructured if its structure is not helpful for the processing task at hand.
  3. Unstructured information might have some structure (semi-structured) or even be highly structured but in ways that are unanticipated or unannounced.

Dealing with unstructured data

Techniques such as data mining, natural language processing (NLP), and text analytics provide different methods to find patterns in, or otherwise interpret, this information. Common techniques for structuring text usually involve manual tagging with metadata or part-of-speech tagging for further text mining-based structuring. The Unstructured Information Management Architecture (UIMA) standard provided a common framework for processing this information to extract meaning and create structured data about the information.

Software that creates machine-processable structure can utilize the linguistic, auditory, and visual structure that exist in all forms of human communication. Algorithms can infer this inherent structure from text, for instance, by examining word morphology, sentence syntax, and other small- and large-scale patterns. Unstructured information can then be enriched and tagged to address ambiguities and relevancy-based techniques then used to facilitate search and discovery. Examples of "unstructured data" may include books, journals, documents, metadata, health records, audio, video, analog data, images, files, and unstructured text such as the body of an e-mail message, Web page, or word-processor document. While the main content being conveyed does not have a defined structure, it generally comes packaged in objects (e.g. in files or documents, …) that themselves have structure and are thus a mix of structured and unstructured data, but collectively this is still referred to as "unstructured data". For example, an HTML web page is tagged, but HTML mark-up typically serves solely for rendering. It does not capture the meaning or function of tagged elements in ways that support automated processing of the information content of the page. XHTML tagging does allow machine processing of elements, although it typically does not capture or convey the semantic meaning of tagged terms.

Since unstructured data commonly occurs in electronic documents, the use of a content or document management system which can categorize entire documents is often preferred over data transfer and manipulation from within the documents. Document management thus provides the means to convey structure onto document collections.

Search engines have become popular tools for indexing and searching through such data, especially text.

Commercial solutions

Several commercial solutions are available for analyzing and understanding unstructured data for business applications. This includes products from companies like NetOwl, LogRhythm, ZL Technologies, SAS, Provalis Research, Inxight, Datagrav,[5] ORKASH and IBM's SPSS or Watson, as well as more specialized offerings such as People Pattern, Attensity, Megaputer Intelligence, Clarabridge, Graphext, Stratifyd, Medallia, General Sentiment, and Sysomos, which focus on analyzing unstructured social media data. Other vendors such as Smartlogic or IRI (CoSort) can find and structure data in unstructured sources, then integrate and transform it along with structured data for business intelligence and analytic purposes.

See also

Notes

  1. ^ Structure, Models and Meaning: Is "unstructured" data merely unmodeled?, Intelligent Enterprise, March 1, 2005.
  2. ^ Structuring Unstructured Data, Forbes, April 5, 2007.
  3. ^ Christopher C. Shilakes and Julie Tylman, "Enterprise Information Portals", Merrill Lynch, 16 November 1998.
  4. ^ Holzinger, Andreas; Stocker, Christof; Ofner, Bernhard; Prohaska, Gottfried; Brabenetz, Alberto; Hofmann-Wellenhof, Rainer (2013). "Combining HCI, Natural Language Processing, and Knowledge Discovery – Potential of IBM Content Analytics as an Assistive Technology in the Biomedical Field". In Holzinger, Andreas; Pasi, Gabriella. Human-Computer Interaction and Knowledge Discovery in Complex, Unstructured, Big Data. Lecture Notes in Computer Science. Springer. pp. 13–24. doi:10.1007/978-3-642-39146-0_2. ISBN 978-3-642-39146-0. 
  5. ^ Unstructured Data and the 80 Percent Rule, Seth Grimes, Clarabridge Bridgepoints, 2008 Q3.
  6. ^ Today’s Challenge in Government: What to do with Unstructured Information and Why Doing Nothing Isn’t An Option, Noel Yuhanna, Principal Analyst, Forrester Research, Nov 2010
  7. ^ New Digital Universe Study Reveals Big Data Gap: Less Than 1% of World’s Data is Analyzed; Less Than 20% is Protected, EMC Press Release, December 2012.
  8. ^ Semi- and unstructured data processing / preparation in IRI CoSort, May 2014.

References

  1. 1 2 Grimes, Seth. "A Brief History of Text Analytics". B Eye Network. Retrieved June 24, 2016.
  2. ↑ Albright, Russ. "Taming Text with the SVD" (PDF). SAS. Retrieved June 24, 2016.
  3. ↑ Desai, Manish. "Applications of Text Analytics". My Business Analytics @ Blogspot. Retrieved June 24, 2016.
  4. ↑ Chakraborty, Goutam. "Analysis of Unstructured Data: Applications of Text Analytics and Sentiment Mining" (PDF). SAS. Retrieved June 24, 2016.
  5. ↑ Datagrav: A Framework for Knowledge Sharing Using Transclusion Enabled Collaboration Media | Sergey Kochuguev - Academia.edu

External links

This article is issued from Wikipedia - version of the 11/11/2016. The text is available under the Creative Commons Attribution/Share Alike but additional terms may apply for the media files.