👋 Welcome! I’m documenting my progress through #DEZoomcamp DataTalksClub’s Data Engineering Zoomcamp.

• Goal: Build end-to-end data pipelines with Docker, Terraform, GCP, and more.
• Learning Style: Hands-on projects + public accountability.

📂 Projects

Module 1: Docker ,postgresql& Terraform

Objective

Containerize a Postgres-based data pipeline and automate GCP infrastructure with Terraform.

Tech Stac

Docker | Terraform | Python | Google Cloud Platform (GCP) | Postgres

Code Highlights

# Sample Ingestion Script (taxi_ingest.py)
import pandas as pd
from sqlalchemy import create_engine

def load_data(df: pd.DataFrame, user: str, password: str, host: str, port: str, db: str):
    engine = create_engine(f'postgresql://{user}:{password}@{host}:{port}/{db}')
    df.to_sql('nyc_taxi', engine, if_exists='replace')

Challenges & Solutions

• Problem: Docker containers couldn’t communicate. Fix: Used a custom network in docker-compose.yaml.
• Problem: Terraform state file conflicts. Fix: Stored state in a GCS bucket for team consistency.

GitHub Repo

🔗 Module 1 Code