Add 东莞法拍房 historical data: 31 sub-regions, 24,949 records (2017-2026)
Scraped all 31 Dongguan sub-regions using sortField=2 (end-time ascending) to bypass JD's ~4000-item API cap. Merged 34 CSV files by paimaiId into 24,949 unique records covering 2017-07 to 2026-11. Key findings: - 上架量 grew ~50x: 98 (2017) → 4,813 (2026) - 流拍率 peaked at 81.6% (2024), eased to 67.4% (2026) - 樟木头: 558 records, failure rate peaked 94.2% (2024) - 塘厦: 205 records, 2026 failure rate 51.0% Includes: scrape_history.py, batch_scrape_towns.sh, analyze_trends.py, yearly_stats.py, and updated SKILL.md + url_structure.md documenting the 4000-item cap and sub-region scraping strategy. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.6
parent
bfbb4e6a26
commit
f06d84a003
Executable
+15
@@ -0,0 +1,15 @@
|
||||
#!/bin/bash
|
||||
# Batch scrape all Dongguan sub-regions with sortField=2
|
||||
# Skips 樟木头 and 塘厦 (already scraped)
|
||||
cd /root/agents/wiki
|
||||
|
||||
TOWNS="莞城 东城 南城 万江 石碣 石龙 茶山 石排 企石 桥头 凤岗 清溪 大朗 黄江 长安 虎门 厚街 沙田 道滘 洪梅 麻涌 中堂 高埗 大岭山 望牛墩 常平 寮步 东坑 横沥 谢岗 松山湖"
|
||||
|
||||
for town in $TOWNS; do
|
||||
echo "=== $(date '+%H:%M:%S') Starting $town ==="
|
||||
PYTHONUNBUFFERED=1 python3 -u skills/jd-fapai-scrape/scrape_history.py \
|
||||
--keyword "$town" --batch-label "${town}_all" \
|
||||
--max-scrolls 200 --output-dir "output/法拍"
|
||||
echo "=== $(date '+%H:%M:%S') Done $town ==="
|
||||
done
|
||||
echo "=== ALL DONE ==="
|
||||
Reference in New Issue
Block a user