Анализ Больших данных. Лекция. Хранение и извлечение информации. Работа с SQL и NoSQL базами данных.

Summary

The lecture introduces ways to collect, store, query, and retrieve data for analysis. Its central point is that data analysis depends not only on processing data, but also on finding and gathering it—often a substantial task in real-world projects.

Data Sources and Relational Databases

Data may come from databases, file storage, websites, and online services. The lecture first focuses on relational databases, where data is arranged in tables and relationships connect records across tables.

A library database serves as the example. Its four tables represent authors, books, loan records, and visitors. The schema shows what information is stored and how the tables relate. For instance, a book can refer to its author, and a loan record can link a visitor with a borrowed book.

Basic SQL Operations

SQL is presented as a declarative language: a query specifies what data is wanted rather than the steps to retrieve it. The lecture covers:

• SELECT and FROM to choose columns and the table to query.
• WHERE to filter records, including text matching with LIKE.
• Aliases to make table references shorter and queries easier to read.
• JOIN to combine related tables using matching fields, such as an author ID.
• Aggregation and GROUP BY to summarize records. The example counts English-language book loans for each library visitor.

The lecture also explains that SQL results can be retrieved in Python and exported to formats such as CSV, JSON, or a pandas DataFrame.

NoSQL Databases and MongoDB

Relational databases can become less suitable as data volumes grow or when data does not fit naturally into tables. The lecture surveys several NoSQL approaches:

• Document databases, which store flexible records such as JSON-like objects.
• Key–value databases, suited to lookups by key.
• Graph databases, designed for highly connected data.
• Column-oriented databases, useful for some analytical workloads.
• Time-series databases, built for measurements and signals over time.

MongoDB is used to illustrate document databases. Records are stored as documents in collections. The lecture demonstrates how to:

• Filter documents with a query object, including combined conditions and alternatives.
• Select which fields to return with a projection.
• Sort results and limit the number of documents returned.
• Use an aggregation pipeline to filter, group, calculate values such as average salary, and sort results.
• Export or back up data with command-line utilities. The appropriate method depends on whether the goal is a database backup or data in JSON or CSV form.

JSON and Python

JSON represents data as objects containing keys and values. Values can include strings, numbers, Boolean values, nested objects, and arrays. Python’s JSON library can convert JSON text into Python objects, which can then be accessed like other dictionaries and lists.

Collecting Information from the Web

The lecture defines web scraping as the automated retrieval of online resources. It distinguishes two common kinds of interaction:

• Static pages: A server sends a complete HTML document in response to a request. Python’s Beautiful Soup library can parse the HTML, search for tags or classes, and extract text.
• Dynamic pages: The page presentation and its data may be requested separately. Single-page applications commonly use AJAX and exchange data in formats such as JSON, often through an API.

For HTML, the lecture introduces tags, attributes, nested elements, and the role of browser developer tools in inspecting page structure.

REST APIs and Retrieval Workflow

A REST API exposes resources—such as authors, books, or characters—through URLs. Resources may be grouped into collections and linked to related resources. The lecture describes common HTTP methods:

• GET retrieves information.
• POST adds or creates an item.
• PUT creates or replaces an item.
• PATCH updates selected fields.
• DELETE removes an item.

Clients can request particular response formats, such as JSON, when the server supports them. A typical retrieval workflow is to send a request, parse the response, and then follow links to related resources if needed. Python’s requests library is shown as a way to make requests and parse JSON responses.

Practical Considerations

Automated collection can encounter authorization requirements and site restrictions. The lecture notes that access may require a token and that some sites limit or block automated requests. Scraping should respect applicable permissions and site rules; request frequency and the site’s documented access methods also matter.

Speakers and Sources Featured

• Speaker: An unnamed lecturer or presenter; no other speakers are identifiable in the subtitles.
• Technologies and examples mentioned: SQL and relational databases; Python; pandas; MongoDB; JSON; Redis; Cassandra; ClickHouse; Prometheus; Beautiful Soup; HTTP; AJAX; REST APIs; and the Python requests library. Examples also refer to websites and online services such as IMDb and Amazon.