Posts

Showing posts from January, 2022

Welcome to my Encyclopedia

If you made it here, you found my notes page.  I read a lot of information and then I test out that information so I can put it in my own words.  After trying browser bookmarks, trello boards, evernote, plain old notebook, I'm trying this option to reference all the great stuff I read. I am not taking credit for the great content out there, I only needed a spot where I can find it again and again.   Never stop learning!

Data Engineering

Quick Notes Organize Snowflake Warehouse 3 main components of data warehouse where you ingest raw data (RAW) where data transformations occur (BASE/STAGING - basic casting and field name changes) where you store reporting and experimentation Reverse ETL Census Hub and Spoke model SCD - Slowly Changing Dimensions Type 0   - The passive method Type 1   - Overwriting the old value Type 2   - Creating a new additional record Type 3   - Adding a new column Type 4   - Using historical table Type 6   - Combine approaches of types 1,2,3 (1+2+3=6) Data Modeling Techniques Entity Relationship Diagrams (ERD) Unified Modeling Language (UML) Class Diagrams Data Dictionary Links How to Collect Behavioral Data?  A Guide for Data Engineers and Analysts. CDI: Customer Data Infrastructure https://airbyte.com/blog/collect-behavioral-data-guide Modern Data Stack examples:  https://future.a16z.com/emerging-architectures-modern-data-infrastructure/