Home

Show HN: Data Engineering Book – An open source, community-driven guide
xx123122 about 11 hours ago

Show HN: Data Engineering Book – An open source, community-driven guide

Hi HN! I'm currently a Master's student at USTC (University of Science and Technology of China). I've been diving deep into Data Engineering, especially in the context of Large Language Models (LLMs).

The Problem: I found that learning resources for modern data engineering are often fragmented and scattered across hundreds of medium articles or disjointed tutorials. It's hard to piece everything together into a coherent system.

The Solution: I decided to open-source my learning notes and build them into a structured book. My goal is to help developers fast-track their learning curve.

Key Features:

LLM-Centric: Focuses on data pipelines specifically designed for LLM training and RAG systems.

Scenario-Based: Instead of just listing tools, I compare different methods/architectures based on specific business scenarios (e.g., "When to use Vector DB vs. Keyword Search").

Hands-on Projects: Includes full code for real-world implementations, not just "Hello World" examples.

This is a work in progress, and I'm treating it as "Book-as-Code". I would love to hear your feedback on the roadmap or any "anti-patterns" I might have included!

Check it out:

Online: https://datascale-ai.github.io/data_engineering_book/

GitHub: https://github.com/datascale-ai/data_engineering_book

github.com
153 13
Summary
The wonder of modern drywall
jger15 about 10 hours ago

The wonder of modern drywall

The article explores the history and development of modern drywall, highlighting its evolution from plaster walls to a versatile, cost-effective building material that has revolutionized the construction industry. It delves into the technological advancements, production processes, and the various applications of drywall in residential and commercial construction.

worksinprogress.news
103 162
Summary
Show HN: SQL-tap – Real-time SQL traffic viewer for PostgreSQL and MySQL
mickamy about 4 hours ago

Show HN: SQL-tap – Real-time SQL traffic viewer for PostgreSQL and MySQL

sql-tap is a transparent proxy that captures SQL queries by parsing the PostgreSQL/MySQL wire protocol and displays them in a terminal UI. You can run EXPLAIN on any captured query. No application code changes needed — just change the port.

github.com
84 16
Summary
NPMX – a fast, modern browser for the NPM registry
slymax about 7 hours ago

NPMX – a fast, modern browser for the NPM registry

npmx.dev is a website that provides a comprehensive collection of tools and resources for Node.js developers, including package management, project scaffolding, and development tools, to streamline the development process and enhance productivity.

npmx.dev
82 40
Summary
The evolution of OpenAI's mission statement
coloneltcb about 9 hours ago

The evolution of OpenAI's mission statement

The article discusses OpenAI's mission statement, which emphasizes the responsible development of artificial intelligence to benefit humanity while mitigating potential risks. It highlights OpenAI's focus on making progress in alignment research to ensure AI systems are aligned with human values and interests.

simonwillison.net
68 30
Summary
Backblaze Drive Stats for 2025
Brajeshwar about 4 hours ago

Backblaze Drive Stats for 2025

Backblaze's annual hard drive reliability report for 2025 reveals trends in hard drive failure rates, providing insights into the long-term stability and durability of various drive models and manufacturers for consumers and businesses to consider when making storage decisions.

backblaze.com
54 7
Summary
AI safety leader says 'world is in peril' and quits to study poetry
darod about 12 hours ago

AI safety leader says 'world is in peril' and quits to study poetry

The article discusses the growing popularity of electric vehicles (EVs) in the UK, with over 1 million EVs now on the road. It explores factors driving the EV market, such as government incentives, expanding charging infrastructure, and increasing model variety.

bbc.com
29 16
Summary
Stanford Review: Is YC for Cowards?
hedgehog0 about 6 hours ago

Stanford Review: Is YC for Cowards?

The article discusses the debate around the merits of the Y Combinator startup accelerator program, considering whether it provides a valuable resource for entrepreneurs or encourages a risk-averse approach to building companies.

stanfordreview.org
20 3
Summary
Show HN: Long Mem code agent cut 95% costs for Claude with small model reading
lingxiao10 about 1 hour ago

Show HN: Long Mem code agent cut 95% costs for Claude with small model reading

CoSave is a Visual Studio Code extension that provides automatic cloud-based backups of your code projects, ensuring your work is securely stored and easily restorable across devices.

marketplace.visualstudio.com
12 1
Summary
Show HN: ClipPath – Paste screenshots as file paths in your terminal
viniciusborgeis about 8 hours ago

Show HN: ClipPath – Paste screenshots as file paths in your terminal

ClipPath is an open-source library that provides a simple and efficient way to implement clipping paths in web applications. It offers cross-browser compatibility and supports various image formats, making it a useful tool for web developers working with complex visual elements.

github.com
9 0
Summary
Why Stripe paid $1B for Metronome instead of fixing Billing
AnhTho_FR about 9 hours ago

Why Stripe paid $1B for Metronome instead of fixing Billing

The article discusses Stripe's acquisition of Metronome, a startup that focused on building better billing infrastructure. It suggests that Stripe saw value in Metronome's expertise and technology rather than attempting to build an internal solution, highlighting the strategic considerations around build vs. buy decisions in the fintech industry.

getlago.com
9 4
Summary
Conservative activist hands checks to lawmakers on Wyoming House floor
JumpCrisscross about 9 hours ago

Conservative activist hands checks to lawmakers on Wyoming House floor

The article reports on a controversy that erupted in the Wyoming House of Representatives when a conservative activist handed out checks to lawmakers on the House floor, a move that was seen as an attempt to influence legislation through financial incentives.

wyofile.com
8 2
Summary
Why doesn't the CDC care about Chinese biolabs in America?
DustinEchoes about 11 hours ago

Why doesn't the CDC care about Chinese biolabs in America?

The article criticizes the CDC for not investigating potential biosecurity risks posed by Chinese-run biolab facilities in the United States, arguing that the agency prioritizes COVID-19 response over addressing this national security concern.

spectator.com
7 1
Summary
Why exercise isn't much help if you are trying to lose weight
stevenwoo about 4 hours ago

Why exercise isn't much help if you are trying to lose weight

The article discusses how exercise may not be as effective as commonly believed for weight loss, as the body compensates by increasing appetite and reducing energy expenditure in other ways. It highlights the importance of diet and lifestyle in achieving sustainable weight loss.

newscientist.com
7 3
Summary
OpenAI retired its most seductive chatbot – leaving users angry and grieving
i7l about 7 hours ago

OpenAI retired its most seductive chatbot – leaving users angry and grieving

The article explores the use of OpenAI's GPT-4 chatbot to generate personalized Valentine's Day messages, highlighting the potential and limitations of such AI-powered romantic assistance.

theguardian.com
5 0
Summary
14 More Lessons from 14 years at Google
talonx about 3 hours ago

14 More Lessons from 14 years at Google

This article presents 14 lessons that web developers can learn to improve their skills and practices, covering topics such as performance optimization, developer workflow, and code quality.

addyosmani.com
5 1
Summary
Choices for a Self-Hosted eBook Server
ohjeez about 9 hours ago

Choices for a Self-Hosted eBook Server

The article discusses various self-hosting software options for creating your own ebook server, allowing users to access and read ebooks from their own private server. It highlights popular open-source solutions like Calibre, Kobo, Ubooquity, and Paperless-ng, providing an overview of their features and setup process.

itsfoss.com
4 1
Summary
Viral AI Video of Brad Pitt Fighting Tom Cruise Shakes Hollywood
koolba about 6 hours ago

Viral AI Video of Brad Pitt Fighting Tom Cruise Shakes Hollywood

The article reports on a high-profile fight scene between actors Brad Pitt and Tom Cruise, which was created entirely using AI-generated technology. This has sparked debate within the Hollywood film industry about the potential impact of AI on the future of filmmaking and acting.

latimes.com
4 3
Summary
The lifelong exercise that keeps Japan moving (2020)
surprisetalk about 6 hours ago

The lifelong exercise that keeps Japan moving (2020)

The article discusses the Japanese tradition of 'radio calisthenics', a daily exercise routine broadcast on national radio since the 1930s. This communal exercise program is credited with promoting physical and mental well-being, and has become an integral part of Japanese culture, with millions participating daily.

bbc.com
4 0
Summary
Dutch Lawmakers Approve a 36% Tax on Unrealized Crypto, Stock, and Bond Gains
JumpinJack_Cash about 11 hours ago

Dutch Lawmakers Approve a 36% Tax on Unrealized Crypto, Stock, and Bond Gains

The Dutch government has approved a new tax on unrealized gains in cryptocurrencies, stocks, and bonds, setting a 36% tax rate on these assets. This measure aims to generate additional revenue and potentially curb speculative investments.

imidaily.com
4 0
Summary