December 2017

TIES438 - Big Data Engineering, 5 ECTS

Course teacher

Today (part I)

• What is Big Data? Why study this?
• Course content
• Practicalities

What is Big Data (1)?

What do you think?

Key/buzzwords

Technologies

Algorithms

What is Big Data (2)?

• Big Data Problems
  • Problems
  • Lots of data
• A belief that if you have enough data, then there will be something in there which is valuable
  • Generously told by vendors of all sorts of tools.

What is Big Data (3)?

What is Big Data (4)?

• Volume: amount and dimensionality of data
• Velocity: speed of data accumulation (cf. datastreams and concept drift)
• Variety: heterogeneity of data and formats (cf. audio&video)
• V…
• Veracity: quality of data can vary significantly (cf. sampling)

Why do you want to study Big Data (Engineering)?

• To understand what might be possible
  • and what not
• To be able to handle larger datasets
• To be able to create your own tools

Why do you want to study Big Data (Engineering)?

• To know what the fuzz is about

source: http://www.socmedsean.com/comic-the-critical-element-of-a-successful-big-data-strategy/

Main course targets

• Learn algorithms and generalizable skills, and not …
  • Specific APIs
  • Specific programming languages
  • Certain tools
  • …
• Side targets
  • Brush up some mathematics
  • Learn the Git distributed version control system
  • Use English as a communication language

Prerequisites

• programming (at least programming 2)
• algorithms and data structures
• computational complexity.
• mathematics
  • basics of sets, probability theory, linear algebra, and statistics.

Content

• Algorithms and Processing
• Storage
• Hardware

Algorithms - Processing

• Hashing
• Nearest neighbor search
• Locality-sensitive hashing
• LSH forest
• Map-reduce
• Practically if resources allow
• Data streams
• sampling, filtering, dvc, counting ones, decaying windows
• Bloom filters
• Clustering
• hierarchical and point assignment

Algorithms - Processing

• Graphs
• (personalized) Page Rank
• Embedding
• graph clustering?
• formal concept analysis?
• Recommendation and Frequent pattern mining
• Set containment

Storage

• NoSQL
  • Key-value stores
  • Graphs
  • Document store
  • Column

Hardware considerations

• Memory characteristics
  • hierarchy - transfer speeds
• High Performance Computing
  • Short intro
  • If resources obtained!!!

Course Material

• Mainly blackboard teaching

• Some parts from the course website (under construction)

• Material

  • Mining massive data sets - Anand Rajaraman, Jure Leskovec, Jeffrey D. Ullman
  • Scientific and other articles
  • Optional: Motwani, and Raghavan. Randomized Algorithms. Cambridge, UK: Cambridge University Press, 1995. ISBN: 0521474655. (available in JYU trough EBSCOhost https://jyu.finna.fi/Record/jykdok.1485577 )
  • Optional: Data Clustering: Algorithms and Applications - Charu C. Aggarwal, Chandan K. Reddy 2013 by Chapman and Hall/CRC ISBN 9781466558212
  • The online course https://www.coursera.org/course/mmds overlaps partially with the course.

This course is simple

• We only study a small part of the data science process
  • We focus more on how we analyze as on what we are exactly analyzing for.
  • We do not always care whether what we are doing makes sense.
  • We will not interpret the final result.

This course is simple

• 

source: http://en.wikipedia.org/wiki/Data_analysis

Non-topics

• Signal processing (See CIBA120)
• Dimensionality reduction (except for LSH see also TIES445)
• Explicit NLP (some techniques are used in NLP)
• Explicit BI or analytics (See TJTSM61)
• A broad overview of data mining (See TIES445)
• Deep Learning for Cognitive Computing (TIES 4910)
• Watson Technologies (ITKA 352)

Computation

• Unclear for now :-(

Practicalities

• Very intensive
  • Two lectures / day
  • One demo session / day
• Hands-on experience
  • All content and tools are freely available
• Individual work
• Pair work

Passing the course

• Two options

• If passing in December required:

  • Exam

• Otherwise:

  • Exam
  • Making all course assignments (pair) + article summary (individual)

Individual work

• Write a summary of a research article

Pair work

• Individual or as a pair
• Mostly programming tasks

Later today:

• Today
  • Start with an approximate algorithm
  • Locality sensitive hashing