Анализ Больших данных. Лекция 1. НЕЙМАРК. Школа хакатонщиков.

Summary

This introductory lecture presents the foundations of big data, machine learning, data mining, and neural networks. It explains why these methods are useful, outlines common ways to organize data-analysis work, and previews the problems and techniques covered later in the course.

Big data: key characteristics

The lecturer describes big data through six dimensions:

• Variety: Data comes in different forms and from many sources—for example, social media text, web content, images, audio, and sensor streams. A single source is less likely to qualify unless the data itself is diverse.
• Volume: The amount of data can range from gigabytes to terabytes or petabytes; there is no single minimum threshold.
• Velocity: Data may arrive or need to be processed rapidly, sometimes continuously and in real time.
• Veracity or reliability: Data quality can be affected by mistakes in collection, storage, or processing.
• Value: Data is useful only in relation to a particular goal. Its relevance and value may change depending on the task.
• Confidence in processing: The lecturer also emphasizes trust in the methods used to analyze data, including the difficulty of interpreting “black box” models and assessing whether their results are dependable.

Big-data systems also require suitable infrastructure. The lecture sketches a layered architecture:

1. Infrastructure: Hardware and computing resources, such as servers, storage, and cloud or distributed systems.
2. Storage: File systems and related technologies that support large-scale, reliable, or parallel access to data.
3. Management and processing: Tools for organizing data and running queries or computations.
4. Analysis and presentation: Reporting, visualization, and business-intelligence tools that help people interpret results.

Machine learning and data mining

Large datasets are often too extensive to analyze manually. The lecture presents algorithms as a way to identify patterns and make decisions at a scale that would be impractical for people alone. Earlier expert systems relied on explicitly written rules, but those rules may not cover the many possible cases in large and complex datasets.

The lecturer distinguishes two overlapping aims:

• Data mining seeks patterns in data and often tries to interpret what those patterns mean.
• Machine learning uses patterns learned from data to make predictions or decisions on new examples; the patterns themselves do not always need to be interpretable.

Both involve finding structure in data, and many of the same methods can be used for both.

Machine learning is introduced as part of the broader field of artificial intelligence. The lecturer characterizes most current AI as weak AI: systems that make task-specific decisions based on algorithms and data, rather than possessing consciousness or general human-like reasoning. Examples include face recognition, translation, search, and recommendation systems.

Main categories of machine learning

The lecture introduces three broad approaches:

• Unsupervised learning: Finds patterns or groups in data when the desired structure is not already specified and labeled examples are unavailable. Clustering is one example.
• Supervised learning: Trains a model using examples with known answers, then uses it to make predictions on new data.
• Reinforcement learning: Improves behavior through feedback: actions that improve results are rewarded, while actions that worsen them are penalized.

The categories can be combined. For example, patterns found through unsupervised learning may help prepare data or features for a supervised model.

Neural networks: structure, training, and limitations

Neural networks are presented as an important family of machine-learning models, inspired loosely by biological neural systems. The lecturer notes that artificial networks are mathematical models, not replicas of the human brain.

A simplified artificial neuron:

1. Receives signals from other neurons.
2. Applies a weight to each input.
3. Combines the weighted signals.
4. Passes the result through an activation function, which determines the output sent onward.

Neurons are arranged in layers: an input layer, one or more hidden layers, and an output layer. Different network architectures and activation functions affect what the network can learn and produce.

A typical training process is:

1. Choose a network architecture, including the number of layers and neurons.
2. Prepare the inputs in a form the network can use; some networks do this internally, while others require preprocessing.
3. Initialize the connections’ weights.
4. Pass training examples through the network to obtain predictions.
5. Compare predictions with the desired outputs to calculate an error.
6. Adjust the weights to reduce that error.
7. Repeat across examples and training rounds until performance is adequate.

The lecture discusses gradient-based optimization and identifies backpropagation as a commonly used approach for calculating how weights should be adjusted. It also contrasts ordinary gradient descent with stochastic gradient descent, noting that stochastic methods are often used because they can reach useful solutions more quickly.

Examples of network types mentioned include convolutional networks for tasks such as image or text processing; recurrent networks for sequential or time-dependent data; Kohonen networks for grouping data; and associative and Boltzmann-style networks for pattern-related tasks.

The lecturer highlights several limitations:

• Neural networks can be difficult to interpret and may act as “black boxes.”
• They can overfit: perform well on training examples but poorly on new data.
• Cross-validation and changes to the model or training process can help assess or reduce overfitting.
• Training may require substantial computing power and many labeled examples.
• Neural networks are useful for many tasks but are not a universal replacement for other methods.

Data-analysis workflows and standards

The lecture presents three established frameworks for structuring data work:

• KDD: A workflow involving data selection or extraction, cleaning, transformation, analysis or mining, and interpretation. Cleaning addresses errors and missing or inconsistent values; transformation can create or enrich variables.
• SEMMA: A framework associated in the lecture with SAS. It includes exploring and modeling data, but the lecturer notes that it does not explicitly begin with defining the business problem.
• CRISP-DM: Presented as the more general, business-oriented process:
  1. Business understanding: Work with the relevant organization or domain to define the problem and desired outcome.
  2. Data understanding: Identify what data exists, how it is stored, and whether it is suitable for the task.
  3. Data preparation: Extract, clean, and transform the data.
  4. Modeling: Select and train an appropriate model.
  5. Evaluation: Check whether the model meets objective quality criteria and actually addresses the original problem.
  6. Deployment: Put the solution into practical use, addressing technical needs such as integration, speed, and who will use it.

The process is iterative. If a model performs poorly or the data proves inadequate, the analyst may need to revisit data preparation, data understanding, or even the original problem definition.

Common data-analysis tasks

The lecture identifies five broad task groups:

• Visualization: Present data or analysis results in a form that supports inspection and interpretation.
• Frequent-pattern and association-rule discovery: Find items or events that commonly occur together—for example, products frequently purchased in the same transaction.
• Prediction: Estimate future or unknown values, including time-series values or equipment failures.
• Classification: Assign an item to a known category, such as deciding whether a message is spam or identifying what class an image belongs to.
• Clustering or segmentation: Divide data into groups based on similarity, which can help reveal structure or support tasks such as data summarization.

The lecture concludes by framing these topics as the course’s foundation. Later sessions will explore machine-learning methods, neural networks, architectures, and practical applications in greater depth.

Speakers and sources featured

• Speaker: Bagirov; the name is rendered in the subtitles as “Mira Bass,” which may reflect an auto-captioning error.
• Frameworks and concepts named: KDD, SEMMA, CRISP-DM, Hadoop/HDFS (the subtitles render the name unclearly), Kolmogorov’s theorem, and the perceptron associated with Rosenblatt.
• Software named: SAS, TensorFlow, and a Python library rendered in the subtitles as “Pie Cake.”
• Other speakers: None identified in the subtitles.