Autumn 2014

TIES438 - Big Data Engineering, 5 ECTS

Course teacher

Michael (Michael Cochez)

PhD student at MIT department

• Related work
  • Evolving Knowledge Ecosystems for Big Data Understanding
  • Locality-Sensitive Hashing for Massive String-Based Ontology Matching
• Ongoing:
  • Balanced Large Scale Knowledge Matching Using LSH Forest
  • Scalable hierarchical clustering
  • Scalable graph summarization

Today

• What is Big Data? Why study this?
• Course content
• Practicalities
• Guest lecture by Prof. Pyshkin

What is Big Data?

• Big Data Problems
• Problems
• Lots of data
• See next lectures

Why do you want to study Big Data (Engineering)?

• To understand what might be possible
  • and what not
• To be able to handle larger datasets
• To be able to create your own tools

Why do you want to study Big Data (Engineering)?

• To know what the fuzz is about

source: http://www.socmedsean.com/comic-the-critical-element-of-a-successful-big-data-strategy/

Main course targets

• Learn generalizable techniques and skills, and not …
  • Specific APIs
  • Specific programming languages
  • Certain platforms
  • …
• Side targets
  • Learn the Git distributed version control system
  • Use English as a communication language

This course is simple

• We only study a small part of the data science process
• We focus more on how we analyze as on what we are exactly analyzing for.
• We do not always care whether what we are doing makes sense.
• We will not interpret the final result.
• 

source: http://en.wikipedia.org/wiki/Data_analysis

Content

• Algorithms
• Storage
• Hardware

Algorithms - Processing

• Nearest neighbor search

  • Locality-sensitive hashing

• Map-reduce

  • Joint with TJTSM61 (Business Analytics and Big Data Management)

• Data streams

  • sampling, filtering, dvc, counting ones, decaying windows
  • Clustering

• Graphs

  • page rank

• Recommendation and Frequent item sets ?

• Material

  • Mining massive data sets - Anand Rajaraman, Jure Leskovec, Jeffrey D. Ullman
  • The online course https://www.coursera.org/course/mmds overlaps partially with the course.

Storage

• File system
  • HDFS
  • Lustre
• SQL
  • sharding
  • Object oriented?
• NoSQL
  • Key-value stores
  • Graphs
  • Document store
  • Column

Hardware considerations

• Memory characteristics
  • hierarchy - transfer speeds
• Accelerators
  • GPU
  • Specific hardware
• High Performance Computing
  • Short intro

Computation

• Center for Science (CSC) - high performance computing
  • Sisu (~40000 cores - ~25 in the world )
  • Taito (~10000 cores)
• Amazon Web Services
• oksa3.it.jyu.fi (16 cores)

Non-topics

• Signal processing (See CIBA120)
• Dimensionality reduction (See summer school - except for LSH)
• Explicit NLP (some techniques are used in NLP)
• Explicit BI (See TJTSM61)

Practicalities

• Weekly cycle
  • Two lectures
  • One demo session
• Hands-on experience
  • All content and tools are freely available
• Individual work (later)
• Group work (next week)
• Small tests at the end of lectures

Practicalities

• Grading (Pass/fail)
  • 40% individual work
  • 40% group work
  • 20% quiz
• Uncertain schedule
  • Last lectures likely to drop
  • Check your e-mail

This week

• Today
  • Guest lecture by Prof. Pyshkin
• Thursday
  • Guest lecture (IBM) in Auditorium 3
• On Friday
  • Normal lecture in Agora Beeta
• On Wednesday
  • First assignment