Loading public profile
A data engineer with experience building large-scale Spark and Databricks data platforms, Airflow-orchestrated pipelines, incremental processing systems, and production data quality controls. Also brings data analytics experience in compliance, experimentation, user behavior analysis, and statistical modeling across financial and cryptocurrency datasets.
Skills
Experience
Data Engineer — TrendIQ Team — TrendAI
→
Re-engineered the business-critical ARR pipeline on a data platform processing 1B+ rows (~20 TB) daily, using Spark, Change Data Feed–based incremental processing, and Delta Lake partitioning and Z-ordering, cutting end-to-end runtime from 18 hours to 40 minutes. Built a Databricks ETL platform using Spark SQL, dbt, and Delta Lake, integrating multiple sources from OneLake/Fabric and Azure Data Explorer across ingestion, transformation, and egress layers, with reusable data-access and write-back components. Implemented production data quality controls for revenue data, including source-to-target reconciliation of row counts and transfer differences, duplicate detection, data-freshness monitoring, and data quality gates blocking bad loads, backed by CI/CD, automated tests, and peer review. Scheduled a daily correction run across the full pipeline to keep incremental outputs consistent with source data. Established production Airflow orchestration on Amazon EKS using Helm, Docker, and ECR for a company-wide data platform, managing scheduling, cross-DAG dependencies, monitoring, and execution of multi-step pipelines. Built an automated pipeline operations agent that classifies Airflow failures, retries eligible transient failures, tracks impacted downstream DAGs, and escalates data or production-risk issues for human review. Moved monitoring from polling to event-driven alerting, investigating only relevant pipeline events and cutting 98% of monitoring cost.
Data Engineer Intern — IGS
→
Reduced development time by 30% through the implementation of Airflow and Composer. Deployed 8 data pipelines and automated ETL processes with Airflow, Git, and FastAPI, integrating MLOps concepts. Developed and integrated Telegram bots for real-time monitoring and notifications using Python and Telegram Operators.
Data Analytics Intern, Central Data Analytics — Binance
→
Directed the Europe Know Your Customer (EU KYC) program to ensure compliance with new EU cryptography regulations and mitigate illegal activities by analyzing customer risk profiles and implementing necessary account access restrictions. Developed tags with Python and SQL to reduce the time required to extract lists for contacting clients by 90%. Assisted the compliance team in increasing conversion rate by 2–3% and decreasing deposit loss after account restriction to less than 10% through data presentation, data visualization, insight contribution, and daily meetings. Conducted 6 A/B testing experiments and optimized code addressing PB-scale data, reducing runtime from over 1 hour to 7 minutes. Analyzed and validated user behavior data with SQL and dashboards to recommend experiments. Showing non-deposit users the new recommendation module in their apps increased trade rate by over 5%. Directing non-registered users to the registration page after a header click increased registration rate by 2%.
Featured projects
Bitcoin vs. Gold: Safe-Haven Analysis
Python · Pandas · NLP · RoBERTa · Regression · Markov Regime-Switching · Granger Causality · Rolling Correlation
Built a multi-source daily dataset aligning 24/7 Bitcoin prices with exchange-traded gold, the S&P 500, VIX, and macroeconomic releases including inflation and unemployment across different trading calendars and time zones. Tested safe-haven behavior with regression, Markov regime-switching, Granger causality, and rolling-window correlation across stress events including the COVID-19 crash, inflation surge, and bear market. Added Twitter sentiment using RoBERTa and found that Bitcoin trades on hype and equity risk while gold trades on fundamentals.
Vaccine Reservation System
Python · SQL · Azure · Pymssql · Database Design · ETL
Wrote SQL DDL, DML, and ETL functions to manipulate databases in Microsoft Azure and completed the ER diagram design. Applied Pymssql in Python to connect local and online databases and used salt and hash techniques to enhance user password security. Created 11 operations, including reserve, cancel, and search schedule, to complete the system using Python.
Education
University of Michigan, Ann Arbor — Master of Science, Data Science
National ChengChi University (NCCU) — Bachelor of Arts, Economics; Mathematical Finance Program
Last updated
