Big Data and Hadoop

 Spring.Taiwan

Read the book "Understanding Big Data" today. Half way done.
Here's some excerpts from the book:

Understanding Big Data

2013-02-28 15:45:37

the term Big Data applies to information that can’t be processed or analyzed using tradi-tional processes or tools

2013-02-28 15:46:12

An IBM survey found that over half of the busi-ness leaders today realize they don’t have access to the insights they need to do their jobs

2013-02-28 15:48:45

Even if every bit of this data was relational (and it’s not), it is all going to be raw and have very different for-mats, which makes processing it in a traditional relational system impractical or impossible

2013-02-28 15:49:05

variety combining to create the Big Data problem

2013-02-28 15:51:31

Three characteristics define Big Data: vol-ume, variety, and velocity.

2013-02-28 15:54:17

the opportunity exists, with the right technology platform, to ana-lyze almost all of the data (or at least more of it by identifying the data that’s useful to you) to gain a better understanding of your business, your customers, and the marketplace

2013-02-28 15:55:35

a fundamental shift in analysis require-ments from traditional structured data to include raw, semis-tructured, and unstructured data as part of the decision-making and insight process

2013-02-28 15:56:27

To capi-talize on the Big Data opportunity, enterprises must be able to analyze all types of data, both re-lational and nonrelational: text, sensor data,

2013-02-28 15:56:50

Dealing effectively with Big Data requires that you perform analytics against the volume and variety of data while it is still in motion, not just after it is at rest

2013-02-28 15:58:11

Hadoop-based platform is well suited to deal with semistructured and unstructured data, as well as when a data discovery process is needed

2013-02-28 16:07:26

it is about dis-covery and making the once near-impossible possible from a scalability and analysis per-spec-t

2013-02-28 16:09:19

creator Doug Cutting’s son gave to his stuffed toy elephant

2013-02-28 16:09:44

duce)—more on these in a

2013-02-28 16:10:23

this redundancy provides fault toler-ance and a capability for the Hadoop cluster to heal itself

2013-02-28 16:10:55

Some of the more notable Hadoop-related projects include: Apache Avro (for data serializa-tion), Cassandra and HBase (databases), Chukwa (a monitoring sys-tem spe-cifically designed with large distributed systems in mind), Hive (provides ad hoc SQL-like queries for data aggregation and summariza-tion), Mahout (a machine learning library), Pig (a high-level Hadoop programming language that provides a data-flow language and execution framework for parallel computation), ZooKeeper (provides coordination services for distributed ap-plications), and more

2013-02-28 16:12:23

throughout the cluster

2013-02-28 16:13:54

For Hadoop deployments using a SAN or NAS, the extra network commu-nica-tion overhead can cause performance bottle

2013-02-28 16:14:59

an individual file is actually stored as smaller blocks that are repli-cated across multiple servers in the entire cluster

2013-02-28 16:16:09

default size of these blocks for Apache Hadoop is 64 MB

2013-02-28 16:21:41

All of Hadoop’s data placement logic is managed by a special server called NameNode

2013-02-28 16:28:14

All of the NameNode’s infor-mation is stored in memory, which allows it to provide quick response times

2013-02-28 16:30:33

Any data loss in this metadata will result in a permanent loss of corresponding data in the cluster

2013-02-28 16:32:25

devel-oper doesn’t have to deal with the concepts of the NameNode and where data is stored—Hadoop does that for you

2013-02-28 16:38:42

The first is the map job, which takes a set of data and converts it into another set of data, where individual elements are broken down into tuples

2013-02-28 16:38:51

reduce job takes the output from a map as input and combines those data tuples into a smaller set of tuples

2013-02-28 16:41:20

All five of these output streams would be fed into the reduce tasks, which combine the in-put results and output a single value for each city, producing a final result set

2013-02-28 16:43:08

MapReduce program is referred to as a job. A job is executed by subse-quently breaking it down into pieces called tasks

2013-02-28 16:43:45

An application submits a job to a specific node in a Hadoop cluster, which is running a daemon called the JobTracker

2013-02-28 16:44:39

In a Hadoop cluster, a set of continually running daemons, referred to as TaskTracker agents, monitor the status of each task

2013-02-28 16:46:39

This direct-ing of records to reduce tasks is known as a Shuffle, which takes input from the map tasks and directs the output to a specific re-duce task

2013-02-28 16:47:29

under Hadoop are written in Java, and it is the Java Archive file (jar) that’s distributed by the JobTracker to the various Hadoop cluster nodes to execute the map and reduce tasks

2013-02-28 16:48:00

BigDataUniversity.com and download Info-Sphere BigInsights Basic Edi-tion (www.ibm.com/software/data/infosphere/ biginsights/basic.html

2013-02-28 16:49:58

Hadoop Common Components are a set of libraries that support the var-ious Hadoop subprojects

2013-02-28 16:50:50

When you delete an HDFS file, the data is not actually gone (think of your MAC or Windows-based home computers, and you’ll get the point). Deleted HDFS files can be found in the trash, which is automatically cleaned at some later point in time

2013-02-28 16:52:46

we cover three of the more popular ones, which admit-tedly sound like we’re at a zoo: Pig, Hive, and Jaql

2013-02-28 16:53:09

Pig was initially developed at Yahoo

2013-02-28 16:56:02

you can FIL-TER out rows that are not of interest, JOIN two sets of data files, GROUP data to build aggrega-tions, ORDER results, and much more

2013-02-28 16:58:01

There are three ways to run a Pig program: embedded in a script, embedded in a Java program, or from the Pig command line, called Grunt

2013-02-28 16:58:51

Some folks at Face-book developed a runtime Hadoop sup-port structure that allows anyone who is already fluent with SQL (which is commonplace for rela-tional data-base developers) to leverage the Hadoop platform right out of the gate. Their cre-ation, called Hive

2013-02-28 16:59:50

You can use the Hive Thrift Client within applications written in C++, Java, PHP, Python, or Ruby

2013-02-28 17:01:19

Hive is read-based and therefore not appropriate for transaction processing

2013-02-28 17:03:11

JSON is built on top of two types of struc-tures. The first is a collection of name/value pairs

2013-02-28 17:03:35

The sec-ond JSON structure is the ability to create an or-dered list of values much like an array, list, or se-quence you might have in your existing applica-tion

2013-02-28 17:09:13

The operand used to signify flow from one operand to another is an arrow: ->. Unlike SQL, where the output comes first (for example, the SELECT list)

2013-02-28 17:15:16

Jaql is a flexible infras-tructure for managing and analyzing many kinds of semistructured data such as XML, CSV data, flat files, relational data, and so on

REF:
http://www.ibm.com/developerworks/wikis/display/db2oncampus/FREE+ebook+-+Understanding+Big+Data