wiki-agentandClaude Opus 4.6
f06d84a003
Scraped all 31 Dongguan sub-regions using sortField=2 (end-time ascending) to bypass JD's ~4000-item API cap. Merged 34 CSV files by paimaiId into 24,949 unique records covering 2017-07 to 2026-11. Key findings: - 上架量 grew ~50x: 98 (2017) → 4,813 (2026) - 流拍率 peaked at 81.6% (2024), eased to 67.4% (2026) - 樟木头: 558 records, failure rate peaked 94.2% (2024) - 塘厦: 205 records, 2026 failure rate 51.0% Includes: scrape_history.py, batch_scrape_towns.sh, analyze_trends.py, yearly_stats.py, and updated SKILL.md + url_structure.md documenting the 4000-item cap and sub-region scraping strategy. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
16 lines
717 B
Bash
Executable File
16 lines
717 B
Bash
Executable File
#!/bin/bash
|
|
# Batch scrape all Dongguan sub-regions with sortField=2
|
|
# Skips 樟木头 and 塘厦 (already scraped)
|
|
cd /root/agents/wiki
|
|
|
|
TOWNS="莞城 东城 南城 万江 石碣 石龙 茶山 石排 企石 桥头 凤岗 清溪 大朗 黄江 长安 虎门 厚街 沙田 道滘 洪梅 麻涌 中堂 高埗 大岭山 望牛墩 常平 寮步 东坑 横沥 谢岗 松山湖"
|
|
|
|
for town in $TOWNS; do
|
|
echo "=== $(date '+%H:%M:%S') Starting $town ==="
|
|
PYTHONUNBUFFERED=1 python3 -u skills/jd-fapai-scrape/scrape_history.py \
|
|
--keyword "$town" --batch-label "${town}_all" \
|
|
--max-scrolls 200 --output-dir "output/法拍"
|
|
echo "=== $(date '+%H:%M:%S') Done $town ==="
|
|
done
|
|
echo "=== ALL DONE ==="
|