• Hive's database is essentially a folder stored in HDFS, by default located at hdfs://ruochen:8020/user/hive/warehouse/

Hive Table Types Summary

Table Type	Data Storage	Deletion Behavior	Use Case
Managed Table	Hive warehouse	DROP TABLE removes both table and data	When Hive manages the full data lifecycle
External Table	HDFS/S3/GCS	DROP TABLE removes metadata only	When multiple applications share the data
Partitioned Table	HDFS/S3, stored by partitions	DROP TABLE removes table and data	Optimizing queries by partitioning on time, region, etc.
Bucketed Table	HDFS/S3, stored in buckets	DROP TABLE removes table and data	Improving JOIN and GROUP BY performance
Temporary Table	Session-only memory	Automatically removed after session ends	Temporary calculations or short-term queries
ACID Table	Hive warehouse, supports transactions	DROP TABLE removes table and data	When transactional updates (INSERT, UPDATE, DELETE) are needed
View	Only stores query logic	DROP VIEW removes view only	Simplifying complex queries or providing unified data access
Temporary View	Session-only memory	Automatically removed after session ends	Temporary query results without persistence

Scatter -> Gather (This is the pattern of MapReduce)
Central Scheduling -> Step Execution (This is the pattern of big data systems like Spark, Flink)

• The Map function interface provides the "scatter" functionality, allowing servers to process data in a distributed manner.
• The Reduce function interface provides the "gather" (aggregation) functionality, summarizing and aggregating the distributed processing results.

Yarn is a subproject of Hadoop that provides a distributed resource management and job scheduling framework. It is responsible for resource allocation, scheduling, and management, and it is a critical component of Hadoop.

port 8088

• Commonly MapReduce need Yarn to run.

HDFS Architecture

• Main Role: NameNode
• Secondary Role: DataNode
• Auxiliary Role: SecondaryNameNode

NameNode:

• The main role in the HDFS system, it is an independent process.
• Responsible for managing the entire HDFS file system.
• Responsible for managing DataNodes.