⤵️ Quick shortcut: Select the module you’re currently studying ⤵️
Courses Module 1 Module 2 Module 3 Module 4 Module 5 Module 6
Here you will find materials that cover this topic, which we believe are suitable for completing the activity at the end of the lesson. The choice is left to the student's discretion.
Here are some recommended courses to explore topics like "Overview of Databases, APIs, and File Uploads" and "Real-time vs Batch Pipelines: Pros and Cons".
Below you will find some suggested materials on these topics, which we believe may help you complete the final activity in the lesson. Please note that using these resources is entirely optional, and you are welcome to explore any other sources or courses at your own discretion.
Here are some recommended courses to explore:
This course offers comprehensive training in databases, covering everything from basic concepts to advanced levels, including MySQL, SQL language, EER diagrams, and AWS RDS.
Offered by the University of Colorado on Coursera, this course covers data collection from files, SQL databases, and web APIs, as well as their integration into unified datasets for analysis.
Available on Alison, this free course introduces fundamental concepts of data and databases, including design, management, and SQL usage.
Offered by Microsoft Learn, this module teaches the basics of real-time data processing, including differences between batch and streaming processing, and how to implement real-time analytics solutions.
An article from Educative explaining the differences between batch and stream data processing, providing a clear understanding of when to use each approach.
Offered by Great Learning, this resource provides free courses on APIs, including RESTful API design, web services, and API integrations, essential for understanding how to interact with various data sources.
Learn to create RESTful APIs using Spring Boot, with a focus on proper API design, database integration, and deep understanding of HTTP protocols.
This course introduces you to Apache Kafka and its applications in real-time data processing, comparing it with traditional batch processing, and integrating with Python and Power BI.
These courses provide a solid understanding of the mentioned concepts and help you deepen your knowledge in each area.
✅ Overview of databases, APIs, and file uploads.
✅ Real-time vs batch pipelines: Pros and cons.
Explore a sample SQLite database and understand its schema.
Hands-On Activity Lesson 1
Objective:
Explore and interact with a sample SQLite database to understand its structure and data.
Steps:
Set Up the SQLite Database:
Download the employees.db sample database provided in the resources.
Install Required Libraries:
Ensure you have Python installed (3.8+ recommended).
Install the SQLite library (if not already included with Python):
bash
Copy code
pip install sqlite3
Write a Python Script to Explore the Database Schema:
Connect to the database.
Query the schema to identify available tables and columns.
Fetch a sample of the data to understand its structure.
Sample Script:
python
Copy code
import sqlite3
# Connect to the SQLite database
db_path = "employees.db" # Replace with your database path
connection = sqlite3.connect(db_path)
# Create a cursor object to execute SQL queries
cursor = connection.cursor()
# Fetch and display all tables in the database
cursor.execute("SELECT name FROM sqlite_master WHERE type='table';")
tables = cursor.fetchall()
print("Tables in the database:", tables)
# Explore the schema of a specific table
table_name = "employees" # Replace with your table name
cursor.execute(f"PRAGMA table_info({table_name});")
schema = cursor.fetchall()
print(f"Schema of the '{table_name}' table:", schema)
# Fetch and display a sample of the data
cursor.execute(f"SELECT * FROM {table_name} LIMIT 5;")
sample_data = cursor.fetchall()
print(f"Sample data from '{table_name}' table:", sample_data)
# Close the database connection
connection.close()
A Python script that connects to the SQLite database, retrieves the schema, and displays a sample of the data.
Understand the structure of data stored in databases.
Gain hands-on experience interacting with a database using Python.
Differentiate between real-time and batch pipelines to select the right approach for specific use cases.
Write a Python script to explore the schema and fetch data from a database.
Write a Python script to explore the schema and fetch data from a database.
This activity focuses on gaining hands-on experience with Python and SQLite by exploring the structure (schema) of a database and retrieving sample data.
Understand the database schema: Learn the structure of the database, including table names, columns, and their data types.
Fetch and inspect data: Retrieve and display sample data from tables to understand the stored information.
1. Set Up the Environment
Install Python (3.8+ recommended).
SQLite is included with Python by default. If needed, verify the installation by running:
bash
Copy code
python -c "import sqlite3; print('SQLite is ready')"
2. Download the Sample Database
Obtain the employees.db file from the course resources or create a sample SQLite database for testing.
3. Write the Python Script
Follow these steps in your script:
python
Copy code
import sqlite3
# Step 1: Connect to the SQLite database
db_path = "employees.db" # Path to your SQLite database
try:
connection = sqlite3.connect(db_path)
print("Database connection successful!")
except sqlite3.Error as e:
print(f"Error connecting to database: {e}")
exit()
# Step 2: Create a cursor object to execute SQL queries
cursor = connection.cursor()
# Step 3: Retrieve and display all table names in the database
print("\n--- Tables in the Database ---")
cursor.execute("SELECT name FROM sqlite_master WHERE type='table';")
tables = cursor.fetchall()
for table in tables:
print(f"Table: {table[0]}")
# Step 4: Explore the schema of a specific table
table_name = "employees" # Replace with your desired table name
print(f"\n--- Schema of Table: {table_name} ---")
cursor.execute(f"PRAGMA table_info({table_name});")
schema = cursor.fetchall()
for column in schema:
print(f"Column: {column[1]}, Type: {column[2]}, Nullable: {bool(column[3])}")
# Step 5: Fetch and display sample data from the table
print(f"\n--- Sample Data from Table: {table_name} ---")
cursor.execute(f"SELECT * FROM {table_name} LIMIT 5;")
sample_data = cursor.fetchall()
for row in sample_data:
print(row)
# Step 6: Close the database connection
connection.close()
print("\nDatabase connection closed.")
Connect to the Database:
The sqlite3.connect() function establishes a connection to the database file.
A try-except block is used to handle connection errors.
Retrieve Table Names:
sqlite_master contains metadata about the database.
The query SELECT name FROM sqlite_master WHERE type='table' lists all table names.
Explore Table Schema:
The PRAGMA table_info command returns details about a table's schema, including column names, data types, and nullability.
Fetch Sample Data:
Use SELECT * FROM <table_name> LIMIT 5 to fetch the first 5 rows of data.
Close Connection:
Properly close the database connection to free resources.
For a database containing an employees table:
sql
Copy code
Database connection successful!
--- Tables in the Database ---
Table: employees
--- Schema of Table: employees ---
Column: id, Type: INTEGER, Nullable: False
Column: name, Type: TEXT, Nullable: False
Column: role, Type: TEXT, Nullable: True
--- Sample Data from Table: employees ---
(1, 'Alice', 'Engineer')
(2, 'Bob', 'Manager')
(3, 'Charlie', 'Analyst')
(4, 'Diana', 'Developer')
(5, 'Eve', 'HR')
Database connection closed.
Add command-line arguments to specify the database path and table name.
Format the output for better readability using libraries like tabulate or pandas.
Lesson 1: Understanding Data Sources
Types of data: Databases, APIs, and file uploads.
Differences between real-time and batch ingestion.
Lesson 2: Setting Up Cloud Storage (AWS S3)
Create your own AWS account and set up an S3 bucket.
Learn to upload data programmatically using Python and boto3.
Lesson 3: Building a Data Ingestion Script
Connect to a database, fetch data, and store it locally.
Automate data uploads to AWS S3.
Video lectures and downloadable materials are organized in the dashboard.
Hands-on labs and coding environments are integrated (e.g., AWS, Jupyter).
Sample datasets: Pre-populated SQLite database (employees.db).
Python scripts: Downloadable starter and solution scripts for each activity.
AWS setup guides: Step-by-step instructions for creating an account and setting up S3.