• 1 Boeing 737 engine generates 20 terabytes of data an hour. Each aircraft has two of those engines. That’s 40 TB of data an hour.
• So in a standard 6 hour cross country flight, you are talking 240 TB of new data.
• Now multiply that with the number of flights for all commercial jetliners daily in the US.
• For Southwest Airlines alone... Total data generated every day by Southwest Airlines’ fleet of 607 Boeing 737 aircraft:
  (20TB/ hr x 10.8hr avg operation per day) x 2 engines x 607 =
  262,224 terabytes (or 256 petabytes) PER DAY

MapReduce: is a programming model for processing large data sets, and the name of an implementation of the model by Google. MapReduce is typically used to do distributed computing on clusters of computers.

Writing a parallel-executable program has proven over the years to be a very challenging task. MapReduce simples the process by requiring coders to write only the simpler Map() and Reduce() functions, which focus on the logic of the specific problem at hand, while the "MapReduce System" handles the marshalling of the distributed servers, running the various tasks in parallel, managing all communications and data transfers between the various parts of the system, providing for redundancy and failures, and overall management of the whole process.

Not new concept (Just wasn't in parrallel):

     #!/usr/bin/env bash
     for year in ../input/ncdc/all/*
     do
        echo -ne `basename $year .gz`"\t"
        gunzip -c $year | \
        awk '{ temp = substr($0, 88, 5) + 0;
        if (temp !=9999 && q ~ /[01459]/ && temp > max) max = temp }
        END { print max }'
     done
		  

• Fine grained Map and Reduce tasks
• Improved load balancing
• Faster recovery from failed tasks
• Automatic re-execution on failure
• In a large cluster, some nodes are always slow or flaky
• Introduces long tails or failures in computation
• Framework re-executes failed tasks
• Locality optimizations
• With big data, bandwidth to data is a problem
• Map-Reduce + HDFS is a very effective solution
• Map-Reduce queries HDFS for locations of input data
• Map tasks are scheduled local to the inputs when possible

  public class WordCountMapper : MapperBase
  {
      private static int one = 1;
      public override void Map(string inputLine, MapperContext context)
      {
          string[] words = inputLine.Split(' ');
          foreach (String word in words) {
              if (word.Length>1) {
                  //outputs each word into a single line with the number 1
                  //skip spaces
                  context.EmitKeyValue(word, one.ToString());
              }
          }
      }
  }		
		  

  public static class Map extends MapReduceBase 
   	implements Mapper {
     private final static IntWritable one = new IntWritable(1);
     private Text word = new Text();

     public void map(LongWritable key, Text value, 
   		  OutputCollector output, Reporter reporter) throws IOException {
       String line = value.toString();
       StringTokenizer tokenizer = new StringTokenizer(line);
       while (tokenizer.hasMoreTokens()) {
         word.set(tokenizer.nextToken());
         output.collect(word, one);
       }
     }
   }		
		

    public class WordCountReducer : ReducerCombinerBase
    {
        public override void Reduce(string key, IEnumerable values, ReducerCombinerContext context)
        {
            //sum all the ones
           int sum = 0;
           foreach(String value in values)
           {
              sum += int.Parse(value);
           }
           context.EmitKeyValue(key, sum.ToString());
        }
    }		
			  

    public static class Reduce extends MapReduceBase 
    	implements Reducer {
      public void reduce(Text key, Iterator values, 
    		  OutputCollector output, Reporter reporter) throws IOException {
        int sum = 0;
        while (values.hasNext()) {
          sum += values.next().get();
        }
        output.collect(key, new IntWritable(sum));
      }
    }		
		  

	public class WordCountjob : HadoopJob
	{
	    public override HadoopJobConfiguration Configure(ExecutorContext context)
	    {
	        HadoopJobConfiguration config = new HadoopJobConfiguration();
	        config.InputPath = "Input/wordcount";
	        config.OutputFolder = "Output/wordcount";
	        return config;
	    }
	}
	static void Main(string[] args)
	{
	    var hadoop = Hadoop.Connect();
	    var result = hadoop.MapReduceJob.ExecuteJob();

	    Console.In.Read();
	}		
			  

    public static void main(String[] args) throws Exception {
       JobConf conf = new JobConf(WordCount.class);
       conf.setJobName("wordcount");

       conf.setOutputKeyClass(Text.class);
       conf.setOutputValueClass(IntWritable.class);

       conf.setMapperClass(Map.class);
       conf.setCombinerClass(Reduce.class);
       conf.setReducerClass(Reduce.class);

       conf.setInputFormat(TextInputFormat.class);
       conf.setOutputFormat(TextOutputFormat.class);

       FileInputFormat.setInputPaths(conf, new Path(args[0]));
       FileOutputFormat.setOutputPath(conf, new Path(args[1]));

       JobClient.runJob(conf);
    }		
		  

		
  <dependencies>
      <dependency>
          <groupId>org.apache.hadoop</groupId>
          <artifactId>hadoop-core</artifactId>
          <version>1.1.2</version>
      </dependency>
   </dependencies>
		  

		
mvn clean install 
export target/ 
hadoop jar WordCount-0.0.1-SNAPSHOT.jar com.wesleyreisz.bigdata.wordcount.v1.WordCount \
	/home/reiszwt/samples/bible/bible+shakes.nopunc /home/reiszwt/samples/bible/output
	  

What does it mean to be a pig?
The Apache Pig Project has some founding principles that help pig developers decide how the system should grow over time. This page presents those principles.

Pigs Eat Anything
Pig can operate on data whether it has metadata or not. It can operate on data that is relational, nested, or unstructured. And it can easily be extended to operate on data beyond files, including key/value stores, databases, etc.

Pigs Live Anywhere
Pig is intended to be a language for parallel data processing. It is not tied to one particular parallel framework. It has been implemented first on Hadoop, but we do not intend that to be only on Hadoop.

Pigs Are Domestic Animals
Pig is designed to be easily controlled and modified by its users.

Pigs Fly
Pig processes data quickly. We want to consistently improve performance, and not implement features in ways that weigh pig down so it can't fly.

http://pig.apache.org/

Apache Hive
The Apache HiveTM data warehouse software facilitates querying and managing large datasets residing in distributed storage. Built on top of Apache HadoopTM , it provides

Tools to enable easy data extract/transform/load (ETL)
A mechanism to impose structure on a variety of data formats Access to files stored either directly in Apache HDFSTM or in other data storage systems such as Apache HBase

Query execution via MapReduce
Hive defines a simple SQL-like query language, called QL, that enables users familiar with SQL to query the data. At the same time, this language also allows programmers who are familiar with the MapReduce framework to be able to plug in their custom mappers and reducers to perform more sophisticated analysis that may not be supported by the built-in capabilities of the language. QL can also be extended with custom scalar functions (UDF's), aggregations (UDAF's), and table functions (UDTF's).

Hive is not designed for OLTP workloads and does not offer real-time queries or row-level updates. It is best used for batch jobs over large sets of append-only data (like web logs). What Hive values most are scalability (scale out with more machines added dynamically to the Hadoop cluster), extensibility (with MapReduce framework and UDF/UDAF/UDTF), fault-tolerance, and loose-coupling with its input formats.

https://cwiki.apache.org/confluence/display/Hive/Home