High Level Design
Databases
What databases are, how data is stored and retrieved, in-memory vs disk trade-offs, and a tour of relational vs NoSQL data models.
A database is a structured collection of data that provides efficient scaling, data integrity, security, and analytics capabilities.
Is a Database a Server?#
A database is not a server itself but typically runs on a database server — a specialized service that handles data storage, retrieval, and management.
Best practice: Separate the database from application logic. Benefits:
- Improved scalability
- Easier deployment and management
- Increased maintainability
Should Your Database Be In-Memory?#
Use an in-memory database when:
- Real-time analytics, caching layers, gaming leaderboards, or financial trading
- Low latency is critical
- Dataset fits in RAM
- Examples: Redis, Memcached, H2 (for testing)
Avoid in-memory when:
- Data set exceeds available RAM
- Durability is a must — memory is volatile
- Long-term persistent storage is needed (user data, logs, business records)
How Data Is Stored#
| Mechanism | Description |
|---|---|
| Pages / Blocks | Tables are broken into pages; units of storage on disk or memory |
| Indexes | Stored separately; point to actual data locations; speed up queries |
| B+ Trees | Used in relational DBs for efficient range queries and lookups |
| Tables (RDBMS) | Rows and columns; each row = one record; stored on disk in data pages |
| Documents (NoSQL) | JSON-like documents (MongoDB) |
| Key-Value (NoSQL) | Simple key-value pairs (Redis) |
| Columnar (NoSQL) | Column-wise storage, good for analytics (Cassandra) |
| Graph (NoSQL) | Nodes and relationships (Neo4j) |
How Data Is Retrieved#
- Using indexes — Faster reads by avoiding full table scans
- Full table scans — Slower, but necessary when queries touch many rows or use non-indexed columns
- Query Optimizer — Automatically decides whether to use an index or do a full scan