Add 东莞法拍房 historical data: 31 sub-regions, 24,949 records (2017-2026)
Scraped all 31 Dongguan sub-regions using sortField=2 (end-time ascending) to bypass JD's ~4000-item API cap. Merged 34 CSV files by paimaiId into 24,949 unique records covering 2017-07 to 2026-11. Key findings: - 上架量 grew ~50x: 98 (2017) → 4,813 (2026) - 流拍率 peaked at 81.6% (2024), eased to 67.4% (2026) - 樟木头: 558 records, failure rate peaked 94.2% (2024) - 塘厦: 205 records, 2026 failure rate 51.0% Includes: scrape_history.py, batch_scrape_towns.sh, analyze_trends.py, yearly_stats.py, and updated SKILL.md + url_structure.md documenting the 4000-item cap and sub-region scraping strategy. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
wiki-agent
co-authored by
Claude Opus 4.6
Claude Opus 4.6
parent
bfbb4e6a26
commit
f06d84a003
@@ -112,12 +112,65 @@ python3 skills/jd-fapai-scrape/scrape.py --keyword 东莞市 --output-dir ../法
|
||||
7. **排序**：保持 API 返回顺序（spo_sortField=11 已按发布时间降序），不重新排序
|
||||
8. **增量合并**：按 `paimaiId` 去重合并，新数据在前，旧数据在后
|
||||
|
||||
## 历史数据抓取（sortField=2）
|
||||
|
||||
默认的 `sortField=11`（最新发布）只返回近期上架的房源。京东平台实际保留了从 **2017年7月** 至今的全部历史成交数据，但需要用 `sortField=2`（按结束时间升序）才能获取。
|
||||
|
||||
### ⚠️ 4000条数据上限
|
||||
|
||||
京东法拍无限滚动有 **~4000条硬性上限**（约100次滚动后API停止返回新数据），无论排序方式如何。这意味着：
|
||||
|
||||
- `sortField=11`（最新发布）：返回最近~4000条（约覆盖最近10个月）
|
||||
- `sortField=2`（结束时间升序）：返回最早的~4000条（约覆盖2017-07至~2022）
|
||||
|
||||
**无法通过一次搜索获取全量历史数据。** 对于总量超过4000条的城市（如东莞市住宅~20,276条），需要按镇/区分区域抓取。
|
||||
|
||||
### 按镇/区分区域抓取（推荐）
|
||||
|
||||
```bash
|
||||
# 单个镇（数据量通常<4000，可一次抓全）
|
||||
python3 skills/jd-fapai-scrape/scrape_history.py \
|
||||
--keyword 樟木头 --batch-label 樟木头_all \
|
||||
--max-scrolls 200 --output-dir "output/法拍"
|
||||
|
||||
# 批量抓取所有镇（shell脚本循环）
|
||||
for town in 莞城 东城 南城 万江 ... ; do
|
||||
python3 skills/jd-fapai-scrape/scrape_history.py \
|
||||
--keyword "$town" --batch-label "${town}_all" \
|
||||
--max-scrolls 200 --output-dir "output/法拍"
|
||||
done
|
||||
```
|
||||
|
||||
合并时用 `analyze_trends.py` 按 `paimaiId` 去重：
|
||||
```bash
|
||||
python3 skills/jd-fapai-scrape/analyze_trends.py
|
||||
```
|
||||
|
||||
### 大镇的数据缺口
|
||||
|
||||
部分大镇（如东城~3998条、南城~3780条）会接近4000条上限，`sortField=2` 的数据覆盖到~2025年。2025年末至2026年的近期数据可通过原始 `scrape.py`（`sortField=11`）补充，中间有约1年缺口。
|
||||
|
||||
### 数据量参考
|
||||
|
||||
| 关键词 | 类目 | 总记录数 | sortField=2 实际获取 |
|
||||
|--------|------|----------|---------------------|
|
||||
| 东莞市 | 住宅（cate_id=15） | ~20,276 | ~3,995（2017至~2022） |
|
||||
| 樟木头 | 住宅 | ~558 | 558（2018至2026，全覆盖） |
|
||||
| 东城 | 住宅 | ~3,998 | 3,998（2017至~2025，触顶） |
|
||||
| 南城 | 住宅 | ~3,780 | 3,780（2018至~2025，触顶） |
|
||||
|
||||
> 每页约40条，上限约100次滚动≈4000条。单个镇通常1-3分钟完成。
|
||||
|
||||
## 注意事项
|
||||
|
||||
- 京东法拍 API 有风控验证，必须通过浏览器渲染，无法直接调用 API
|
||||
- 页面是 JS SPA，但第一页数据通过 SSR 内嵌在 HTML `<script>` 标签中，需单独提取
|
||||
- API 只返回 page 2+，只拦截 API 会漏掉 page 1（包括最新发布的房源）
|
||||
- URL 中的位置参数（`tttparams`、`spo_lat`、`spo_lng`）影响搜索结果范围
|
||||
- API 每页约 40 条，东莞市全量约 4000 条（可能为 API 上限）
|
||||
- API 每页约 40 条，无限滚动约 100 次后停止返回新数据（~4000 条硬性上限），非滚动次数限制
|
||||
- 历史数据（`sortField=2`）最早到 2017年7月，更早的数据平台已下架
|
||||
- 京东法拍无独立"已结束"tab，`sortField=2` 是获取历史成交的唯一方式
|
||||
- 大城市（如东莞市）总量超过4000条，需按镇/区分区域抓取才能获取全量历史数据
|
||||
- `scrape_history.py` 的 `getPaimaiCurrentInfoByIdsForApi` 响应捕获需要足够等待时间（初始5秒、滚动间隔3秒），否则结束时间等字段可能为空
|
||||
- 拍卖标签（一拍/二拍）仅对部分数据能从 API 获取，其余从折扣率推断
|
||||
- 同一房产的多次拍卖（一拍流拍后二拍）有不同的 `paimaiId`，各自独立保留
|
||||
|
||||
@@ -0,0 +1,520 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Generate trend analysis HTML from JD法拍 CSV files.
|
||||
|
||||
Reads all *_法拍房源*.csv files in output/法拍/, merges by paimaiId,
|
||||
computes monthly trends (上架量/流拍量), and generates an interactive HTML chart.
|
||||
"""
|
||||
|
||||
import csv
|
||||
import json
|
||||
import os
|
||||
import glob
|
||||
from collections import Counter
|
||||
|
||||
CSV_DIR = "output/法拍"
|
||||
OUTPUT_HTML = os.path.join(CSV_DIR, "东莞法拍房趋势分析.html")
|
||||
|
||||
|
||||
def read_all_csvs(csv_dir):
|
||||
all_items = {}
|
||||
csv_files = glob.glob(os.path.join(csv_dir, "*_法拍房源*.csv"))
|
||||
for csv_path in csv_files:
|
||||
try:
|
||||
with open(csv_path, encoding="utf-8-sig") as f:
|
||||
reader = csv.DictReader(f)
|
||||
for row in reader:
|
||||
pid = row.get("拍卖ID", "")
|
||||
if not pid:
|
||||
continue
|
||||
if pid not in all_items:
|
||||
all_items[pid] = row
|
||||
else:
|
||||
existing = all_items[pid]
|
||||
if row.get("结束时间") and not existing.get("结束时间"):
|
||||
all_items[pid] = row
|
||||
except Exception as e:
|
||||
print(f"Warning: failed to read {csv_path}: {e}")
|
||||
return list(all_items.values())
|
||||
|
||||
|
||||
def compute_monthly_trends(items, region_filter=None):
|
||||
if region_filter:
|
||||
items = [r for r in items if region_filter in r.get("标题", "")]
|
||||
|
||||
listed_by_month = Counter()
|
||||
failed_by_month = Counter()
|
||||
ended_by_month = Counter()
|
||||
|
||||
for r in items:
|
||||
start = r.get("开始时间", "")[:7]
|
||||
end = r.get("结束时间", "")[:7]
|
||||
status = r.get("状态", "")
|
||||
bid_count = r.get("出价次数", "")
|
||||
|
||||
if start:
|
||||
listed_by_month[start] += 1
|
||||
if end and status == "已结束":
|
||||
ended_by_month[end] += 1
|
||||
if bid_count in ("0", ""):
|
||||
failed_by_month[end] += 1
|
||||
|
||||
month_set = (
|
||||
set(listed_by_month.keys())
|
||||
| set(ended_by_month.keys())
|
||||
| set(failed_by_month.keys())
|
||||
)
|
||||
if not month_set:
|
||||
return []
|
||||
|
||||
all_months = sorted(month_set)
|
||||
start_m = all_months[0]
|
||||
end_m = all_months[-1]
|
||||
|
||||
full_months = []
|
||||
y, m = int(start_m[:4]), int(start_m[5:7])
|
||||
ey, em = int(end_m[:4]), int(end_m[5:7])
|
||||
while (y, m) <= (ey, em):
|
||||
full_months.append(f"{y:04d}-{m:02d}")
|
||||
m += 1
|
||||
if m > 12:
|
||||
m = 1
|
||||
y += 1
|
||||
|
||||
results = []
|
||||
for month in full_months:
|
||||
listed = listed_by_month.get(month, 0)
|
||||
failed = failed_by_month.get(month, 0)
|
||||
ended = ended_by_month.get(month, 0)
|
||||
rate = round(failed / ended * 100, 1) if ended > 0 else None
|
||||
results.append(
|
||||
{
|
||||
"month": month,
|
||||
"listed": listed,
|
||||
"failed": failed,
|
||||
"ended": ended,
|
||||
"rate": rate,
|
||||
}
|
||||
)
|
||||
return results
|
||||
|
||||
|
||||
def generate_svg(data, panel_idx):
|
||||
n = len(data)
|
||||
if n == 0:
|
||||
return "<svg></svg>"
|
||||
|
||||
W, H = 1120, 240
|
||||
padL, padR, padT, padB = 52, 24, 16, 40
|
||||
plotW = W - padL - padR
|
||||
plotH = H - padT - padB
|
||||
|
||||
max_val = max((max(d["listed"], d["failed"]) for d in data), default=1)
|
||||
max_val = max(max_val, 5)
|
||||
y_ticks = 5
|
||||
y_step_val = max_val / y_ticks if max_val > 0 else 1
|
||||
|
||||
def y_pos(val):
|
||||
if max_val == 0:
|
||||
return padT + plotH
|
||||
return padT + plotH - (val / max_val) * plotH
|
||||
|
||||
def x_pos(i):
|
||||
if n == 1:
|
||||
return padL + plotW / 2
|
||||
return padL + (i / (n - 1)) * plotW
|
||||
|
||||
svg_parts = []
|
||||
|
||||
# Y-axis grid lines and labels
|
||||
for t in range(y_ticks + 1):
|
||||
val = t * y_step_val
|
||||
y = y_pos(val)
|
||||
svg_parts.append(
|
||||
f'<line x1="{padL}" y1="{y:.1f}" x2="{W-padR}" y2="{y:.1f}" '
|
||||
f'stroke="var(--gridline)" stroke-width="1"/>'
|
||||
)
|
||||
svg_parts.append(
|
||||
f'<text x="{padL-8}" y="{y+4:.1f}" text-anchor="end" font-size="11" '
|
||||
f'fill="var(--text-muted)" font-family="system-ui" '
|
||||
f'font-variant-numeric="tabular-nums">{int(val)}</text>'
|
||||
)
|
||||
|
||||
# X-axis labels (year markers)
|
||||
prev_year = None
|
||||
for i, d in enumerate(data):
|
||||
year = d["month"][:4]
|
||||
if year != prev_year:
|
||||
x = x_pos(i)
|
||||
svg_parts.append(
|
||||
f'<text x="{x:.1f}" y="{H-12}" text-anchor="middle" font-size="11" '
|
||||
f'fill="var(--text-muted)" font-family="system-ui">{year}</text>'
|
||||
)
|
||||
if prev_year is not None:
|
||||
svg_parts.append(
|
||||
f'<line x1="{x:.1f}" y1="{padT}" x2="{x:.1f}" y2="{padT+plotH}" '
|
||||
f'stroke="var(--gridline)" stroke-width="0.5" stroke-dasharray="2 4"/>'
|
||||
)
|
||||
prev_year = year
|
||||
|
||||
# Axis line
|
||||
svg_parts.append(
|
||||
f'<line x1="{padL}" y1="{padT+plotH}" x2="{W-padR}" y2="{padT+plotH}" '
|
||||
f'stroke="var(--axis-line)" stroke-width="1"/>'
|
||||
)
|
||||
|
||||
# Area fill for listed (blue)
|
||||
area_pts = [f"{x_pos(0):.1f},{y_pos(data[0]['listed']):.1f}"]
|
||||
for i, d in enumerate(data):
|
||||
area_pts.append(f"{x_pos(i):.1f},{y_pos(d['listed']):.1f}")
|
||||
area_pts.append(f"{x_pos(n-1):.1f},{padT+plotH:.1f}")
|
||||
area_pts.append(f"{x_pos(0):.1f},{padT+plotH:.1f}")
|
||||
svg_parts.append(
|
||||
f'<path d="M{" L".join(area_pts)} Z" fill="var(--series-1)" opacity="0.06"/>'
|
||||
)
|
||||
|
||||
# Area fill for failed (orange)
|
||||
area_pts2 = [f"{x_pos(0):.1f},{y_pos(data[0]['failed']):.1f}"]
|
||||
for i, d in enumerate(data):
|
||||
area_pts2.append(f"{x_pos(i):.1f},{y_pos(d['failed']):.1f}")
|
||||
area_pts2.append(f"{x_pos(n-1):.1f},{padT+plotH:.1f}")
|
||||
area_pts2.append(f"{x_pos(0):.1f},{padT+plotH:.1f}")
|
||||
svg_parts.append(
|
||||
f'<path d="M{" L".join(area_pts2)} Z" fill="var(--series-2)" opacity="0.06"/>'
|
||||
)
|
||||
|
||||
# Line for listed
|
||||
line_pts = [f"{x_pos(i):.1f},{y_pos(d['listed']):.1f}" for i, d in enumerate(data)]
|
||||
svg_parts.append(
|
||||
f'<path d="M{" L".join(line_pts)}" fill="none" stroke="var(--series-1)" '
|
||||
f'stroke-width="1.5" stroke-linejoin="round" stroke-linecap="round"/>'
|
||||
)
|
||||
|
||||
# Line for failed
|
||||
line_pts2 = [f"{x_pos(i):.1f},{y_pos(d['failed']):.1f}" for i, d in enumerate(data)]
|
||||
svg_parts.append(
|
||||
f'<path d="M{" L".join(line_pts2)}" fill="none" stroke="var(--series-2)" '
|
||||
f'stroke-width="1.5" stroke-linejoin="round" stroke-linecap="round"/>'
|
||||
)
|
||||
|
||||
# Data point circles + hit areas
|
||||
hit_w = plotW / n
|
||||
for i, d in enumerate(data):
|
||||
x = x_pos(i)
|
||||
y1 = y_pos(d["listed"])
|
||||
y2 = y_pos(d["failed"])
|
||||
r = 2.5 if n > 50 else 3.5
|
||||
svg_parts.append(
|
||||
f'<circle cx="{x:.1f}" cy="{y1:.1f}" r="{r}" fill="var(--series-1)" '
|
||||
f'stroke="var(--surface-1)" stroke-width="1.5"/>'
|
||||
)
|
||||
svg_parts.append(
|
||||
f'<circle cx="{x:.1f}" cy="{y2:.1f}" r="{r}" fill="var(--series-2)" '
|
||||
f'stroke="var(--surface-1)" stroke-width="1.5"/>'
|
||||
)
|
||||
hx = x - hit_w / 2
|
||||
svg_parts.append(
|
||||
f'<rect x="{hx:.1f}" y="{padT}" width="{hit_w:.1f}" height="{plotH}" '
|
||||
f'fill="transparent" class="hit" data-idx="{i}"/>'
|
||||
)
|
||||
|
||||
# Last point data labels
|
||||
last = data[-1]
|
||||
lx = x_pos(n - 1)
|
||||
svg_parts.append(
|
||||
f'<text x="{lx-6:.1f}" y="{y_pos(last["listed"])-8:.1f}" text-anchor="end" '
|
||||
f'font-size="11" fill="var(--text-primary)" font-family="system-ui" '
|
||||
f'font-variant-numeric="tabular-nums" font-weight="600">{last["listed"]}</text>'
|
||||
)
|
||||
if last["failed"] > 0:
|
||||
svg_parts.append(
|
||||
f'<text x="{lx-6:.1f}" y="{y_pos(last["failed"])+14:.1f}" text-anchor="end" '
|
||||
f'font-size="11" fill="var(--text-primary)" font-family="system-ui" '
|
||||
f'font-variant-numeric="tabular-nums">{last["failed"]}</text>'
|
||||
)
|
||||
|
||||
# Crosshair
|
||||
svg_parts.append(
|
||||
f'<line class="crosshair" x1="0" y1="{padT}" x2="0" y2="{padT+plotH}" '
|
||||
f'stroke="var(--text-muted)" stroke-width="1" stroke-dasharray="3 3" opacity="0"/>'
|
||||
)
|
||||
|
||||
return (
|
||||
f'<svg viewBox="0 0 {W} {H}" width="100%" height="{H}">'
|
||||
+ "".join(svg_parts)
|
||||
+ "</svg>"
|
||||
)
|
||||
|
||||
|
||||
def generate_table_row_html(region, data):
|
||||
rows = [f'<tr class="group-header"><td colspan="6">{region}（共 {len(data)} 个月）</td></tr>']
|
||||
for d in data:
|
||||
rate_str = f'{d["rate"]:.1f}%' if d["rate"] is not None else "—"
|
||||
rows.append(
|
||||
f'<tr><td></td><td style="text-align:left">{d["month"]}</td>'
|
||||
f'<td>{d["listed"]}</td><td>{d["failed"]}</td>'
|
||||
f'<td>{d["ended"]}</td><td>{rate_str}</td></tr>'
|
||||
)
|
||||
return "\n".join(rows)
|
||||
|
||||
|
||||
def generate_html(panels, items_count):
|
||||
svgs = {}
|
||||
for key, data in panels.items():
|
||||
idx = list(panels.keys()).index(key)
|
||||
svgs[key] = generate_svg(data, idx)
|
||||
|
||||
data_json = json.dumps(panels, ensure_ascii=False)
|
||||
panel_keys = list(panels.keys())
|
||||
|
||||
# Date range
|
||||
all_months = []
|
||||
for data in panels.values():
|
||||
all_months.extend([d["month"] for d in data])
|
||||
date_range = f"{min(all_months)} – {max(all_months)}" if all_months else ""
|
||||
|
||||
# Sample counts
|
||||
sample_counts = {}
|
||||
for key in panel_keys:
|
||||
if key == "东莞全市":
|
||||
sample_counts[key] = items_count
|
||||
else:
|
||||
sample_counts[key] = sum(
|
||||
1 for d in panels[key] for _ in range(d["listed"])
|
||||
)
|
||||
|
||||
table_html = "\n".join(
|
||||
generate_table_row_html(key, data) for key, data in panels.items()
|
||||
)
|
||||
|
||||
panel_html = ""
|
||||
for i, key in enumerate(panel_keys):
|
||||
subtitle = f"样本量 {sample_counts[key]} 套" if key != "东莞全市" else f"全市法拍住宅 · 共 {items_count} 条记录"
|
||||
panel_html += f"""
|
||||
<div class="panel">
|
||||
<div class="panel-header">
|
||||
<span class="panel-title">{key}</span>
|
||||
<span class="panel-subtitle">{subtitle}</span>
|
||||
</div>
|
||||
<div class="chart-wrap" id="panel-{i}">
|
||||
{svgs[key]}
|
||||
<div class="tooltip" id="tooltip-{i}"></div>
|
||||
</div>
|
||||
</div>"""
|
||||
|
||||
html = f"""<!DOCTYPE html>
|
||||
<html lang="zh-CN">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>东莞法拍房上架量与流拍量趋势</title>
|
||||
<style>
|
||||
.viz-root {{
|
||||
color-scheme: light;
|
||||
--surface-1: #fcfcfb;
|
||||
--page-plane: #f9f9f7;
|
||||
--text-primary: #0b0b0b;
|
||||
--text-secondary: #52514e;
|
||||
--text-muted: #898781;
|
||||
--gridline: #e1e0d9;
|
||||
--axis-line: #c3c2b7;
|
||||
--series-1: #2a78d6;
|
||||
--series-2: #eb6834;
|
||||
--border-ring: rgba(11,11,11,0.10);
|
||||
}}
|
||||
@media (prefers-color-scheme: dark) {{
|
||||
:root:where(:not([data-theme="light"])) .viz-root {{
|
||||
color-scheme: dark;
|
||||
--surface-1: #1a1a19;
|
||||
--page-plane: #0d0d0d;
|
||||
--text-primary: #ffffff;
|
||||
--text-secondary: #c3c2b7;
|
||||
--text-muted: #898781;
|
||||
--gridline: #2c2c2a;
|
||||
--axis-line: #383835;
|
||||
--series-1: #3987e5;
|
||||
--series-2: #d95926;
|
||||
--border-ring: rgba(255,255,255,0.10);
|
||||
}}
|
||||
}}
|
||||
* {{ margin: 0; padding: 0; box-sizing: border-box; }}
|
||||
body {{
|
||||
background: var(--page-plane);
|
||||
font-family: system-ui, -apple-system, "Segoe UI", sans-serif;
|
||||
color: var(--text-primary);
|
||||
padding: 32px 24px;
|
||||
line-height: 1.5;
|
||||
}}
|
||||
.viz-root {{
|
||||
max-width: 1200px;
|
||||
margin: 0 auto;
|
||||
background: var(--surface-1);
|
||||
border-radius: 8px;
|
||||
padding: 36px 40px 40px;
|
||||
}}
|
||||
h1 {{ font-size: 20px; font-weight: 600; margin-bottom: 6px; }}
|
||||
.subtitle {{ font-size: 13px; color: var(--text-secondary); margin-bottom: 28px; }}
|
||||
.legend {{ display: flex; gap: 24px; margin-bottom: 24px; font-size: 13px; color: var(--text-secondary); }}
|
||||
.legend-item {{ display: flex; align-items: center; gap: 8px; }}
|
||||
.legend-swatch {{ width: 20px; height: 3px; border-radius: 2px; }}
|
||||
.legend-swatch.s1 {{ background: var(--series-1); }}
|
||||
.legend-swatch.s2 {{ background: var(--series-2); }}
|
||||
.panels {{ display: grid; grid-template-columns: 1fr; gap: 36px; }}
|
||||
.panel {{ border-top: 1px solid var(--border-ring); padding-top: 20px; }}
|
||||
.panel:first-child {{ border-top: none; padding-top: 0; }}
|
||||
.panel-header {{ display: flex; justify-content: space-between; align-items: baseline; margin-bottom: 14px; }}
|
||||
.panel-title {{ font-size: 15px; font-weight: 600; }}
|
||||
.panel-subtitle {{ font-size: 12px; color: var(--text-muted); }}
|
||||
.chart-wrap {{ position: relative; overflow-x: auto; }}
|
||||
svg {{ display: block; width: 100%; min-width: 600px; height: auto; overflow: visible; }}
|
||||
.tooltip {{
|
||||
position: absolute; pointer-events: none;
|
||||
background: var(--surface-1);
|
||||
border: 1px solid var(--border-ring);
|
||||
border-radius: 6px;
|
||||
padding: 10px 14px;
|
||||
font-size: 12px;
|
||||
color: var(--text-primary);
|
||||
box-shadow: 0 4px 12px rgba(0,0,0,0.08);
|
||||
opacity: 0;
|
||||
transition: opacity 0.12s;
|
||||
z-index: 10;
|
||||
white-space: nowrap;
|
||||
}}
|
||||
.tt-month {{ font-weight: 600; margin-bottom: 6px; }}
|
||||
.tt-row {{ display: flex; justify-content: space-between; gap: 16px; margin-bottom: 2px; }}
|
||||
.tt-row span {{ display: flex; align-items: center; gap: 6px; }}
|
||||
.tt-dot {{ width: 8px; height: 8px; border-radius: 50%; display: inline-block; }}
|
||||
.tt-val {{ font-variant-numeric: tabular-nums; }}
|
||||
.note {{ font-size: 12px; color: var(--text-muted); margin-top: 24px; line-height: 1.6; }}
|
||||
.data-table-wrap {{ margin-top: 28px; }}
|
||||
.data-table-toggle {{
|
||||
font-size: 13px; color: var(--series-1); cursor: pointer;
|
||||
background: none; border: none; padding: 4px 0;
|
||||
font-family: inherit;
|
||||
}}
|
||||
.data-table {{
|
||||
max-height: 0; overflow: hidden; transition: max-height 0.3s;
|
||||
margin-top: 12px;
|
||||
}}
|
||||
.data-table.visible {{ max-height: 600px; overflow-y: auto; }}
|
||||
.data-table table {{ width: 100%; border-collapse: collapse; font-size: 12px; }}
|
||||
.data-table th, .data-table td {{ padding: 6px 10px; text-align: right; border-bottom: 1px solid var(--gridline); }}
|
||||
.data-table th {{ color: var(--text-secondary); font-weight: 600; }}
|
||||
.data-table td:first-child, .data-table th:first-child {{ text-align: left; }}
|
||||
.data-table tr.group-header td {{ font-weight: 600; color: var(--text-primary); background: var(--page-plane); }}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="viz-root">
|
||||
<h1>东莞法拍房上架量与流拍量趋势</h1>
|
||||
<p class="subtitle">{date_range} · 按拍卖开始时间月度汇总 · 流拍 = 已结束且出价次数为0</p>
|
||||
|
||||
<div class="legend">
|
||||
<div class="legend-item"><span class="legend-swatch s1"></span>上架量</div>
|
||||
<div class="legend-item"><span class="legend-swatch s2"></span>流拍量</div>
|
||||
</div>
|
||||
|
||||
<div class="panels">
|
||||
{panel_html}
|
||||
</div>
|
||||
|
||||
<p class="note">
|
||||
注：上架量按「开始时间」归入对应月份，流拍量按「结束时间」归入对应月份。流拍率 = 流拍量 / 已结束量。近期月份（最近2-3个月）的已结束量和流拍量可能不完整（部分拍卖尚未结束）。
|
||||
</p>
|
||||
|
||||
<div class="data-table-wrap">
|
||||
<button class="data-table-toggle" id="tableToggle">显示/隐藏数据表</button>
|
||||
<div class="data-table" id="dataTable">
|
||||
<table>
|
||||
<thead><tr><th>区域</th><th>月份</th><th>上架量</th><th>流拍量</th><th>已结束</th><th>流拍率</th></tr></thead>
|
||||
<tbody>
|
||||
{table_html}
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<script>
|
||||
const DATA = {data_json};
|
||||
const PANEL_KEYS = {json.dumps(panel_keys, ensure_ascii=False)};
|
||||
|
||||
function getCss(prop) {{
|
||||
const root = document.querySelector('.viz-root');
|
||||
return getComputedStyle(root).getPropertyValue(prop).trim();
|
||||
}}
|
||||
|
||||
function setupPanel(key, idx) {{
|
||||
const wrap = document.getElementById('panel-' + idx);
|
||||
const svg = wrap.querySelector('svg');
|
||||
const crosshair = wrap.querySelector('.crosshair');
|
||||
const tt = document.getElementById('tooltip-' + idx);
|
||||
const data = DATA[key];
|
||||
if (!data || data.length === 0) return;
|
||||
|
||||
function showTooltip(i) {{
|
||||
const d = data[i];
|
||||
if (!d) return;
|
||||
const s1 = getCss('--series-1') || '#2a78d6';
|
||||
const s2 = getCss('--series-2') || '#eb6834';
|
||||
tt.innerHTML =
|
||||
'<div class="tt-month">' + d.month + '</div>' +
|
||||
'<div class="tt-row"><span><span class="tt-dot" style="background:' + s1 + '"></span>上架量</span><span class="tt-val">' + d.listed + ' 套</span></div>' +
|
||||
'<div class="tt-row"><span><span class="tt-dot" style="background:' + s2 + '"></span>流拍量</span><span class="tt-val">' + d.failed + ' 套</span></div>' +
|
||||
'<div class="tt-row"><span>已结束</span><span class="tt-val">' + d.ended + ' 套</span></div>' +
|
||||
'<div class="tt-row"><span>流拍率</span><span class="tt-val">' + (d.rate !== null ? d.rate + '%' : '—') + '</span></div>';
|
||||
const rect = svg.getBoundingClientRect();
|
||||
const viewBox = svg.viewBox.baseVal;
|
||||
const scaleX = rect.width / viewBox.width;
|
||||
const hitEl = wrap.querySelectorAll('.hit')[i];
|
||||
const hitRect = hitEl.getBoundingClientRect();
|
||||
tt.style.left = (hitRect.left - rect.left + hitRect.width / 2 + 12) + 'px';
|
||||
tt.style.top = '10px';
|
||||
tt.style.opacity = 1;
|
||||
const cx = parseFloat(hitEl.getAttribute('x')) + parseFloat(hitEl.getAttribute('width')) / 2;
|
||||
crosshair.setAttribute('x1', cx);
|
||||
crosshair.setAttribute('x2', cx);
|
||||
crosshair.style.opacity = 1;
|
||||
}}
|
||||
|
||||
function hide() {{
|
||||
tt.style.opacity = 0;
|
||||
crosshair.style.opacity = 0;
|
||||
}}
|
||||
|
||||
wrap.querySelectorAll('.hit').forEach(hit => {{
|
||||
hit.addEventListener('mouseenter', () => showTooltip(parseInt(hit.dataset.idx)));
|
||||
hit.addEventListener('mouseleave', hide);
|
||||
}});
|
||||
}}
|
||||
|
||||
PANEL_KEYS.forEach((k, i) => setupPanel(k, i));
|
||||
|
||||
document.getElementById('tableToggle').addEventListener('click', () => {{
|
||||
document.getElementById('dataTable').classList.toggle('visible');
|
||||
}});
|
||||
</script>
|
||||
</body>
|
||||
</html>"""
|
||||
|
||||
with open(OUTPUT_HTML, "w", encoding="utf-8") as f:
|
||||
f.write(html)
|
||||
print(f"Generated: {OUTPUT_HTML}")
|
||||
for key, data in panels.items():
|
||||
total_listed = sum(d["listed"] for d in data)
|
||||
total_failed = sum(d["failed"] for d in data)
|
||||
print(f" {key}: {len(data)} months, {total_listed} listed, {total_failed} failed")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
items = read_all_csvs(CSV_DIR)
|
||||
print(f"Total unique items: {len(items)}")
|
||||
|
||||
panels = {
|
||||
"东莞全市": compute_monthly_trends(items),
|
||||
"樟木头": compute_monthly_trends(items, "樟木头"),
|
||||
"塘厦": compute_monthly_trends(items, "塘厦"),
|
||||
}
|
||||
|
||||
generate_html(panels, len(items))
|
||||
Executable
+15
@@ -0,0 +1,15 @@
|
||||
#!/bin/bash
|
||||
# Batch scrape all Dongguan sub-regions with sortField=2
|
||||
# Skips 樟木头 and 塘厦 (already scraped)
|
||||
cd /root/agents/wiki
|
||||
|
||||
TOWNS="莞城 东城 南城 万江 石碣 石龙 茶山 石排 企石 桥头 凤岗 清溪 大朗 黄江 长安 虎门 厚街 沙田 道滘 洪梅 麻涌 中堂 高埗 大岭山 望牛墩 常平 寮步 东坑 横沥 谢岗 松山湖"
|
||||
|
||||
for town in $TOWNS; do
|
||||
echo "=== $(date '+%H:%M:%S') Starting $town ==="
|
||||
PYTHONUNBUFFERED=1 python3 -u skills/jd-fapai-scrape/scrape_history.py \
|
||||
--keyword "$town" --batch-label "${town}_all" \
|
||||
--max-scrolls 200 --output-dir "output/法拍"
|
||||
echo "=== $(date '+%H:%M:%S') Done $town ==="
|
||||
done
|
||||
echo "=== ALL DONE ==="
|
||||
@@ -0,0 +1,432 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
京东法拍历史数据分批抓取脚本（一次性使用）
|
||||
|
||||
用 sortField=2（按结束时间升序，只返回已结束拍卖）滚动全量历史数据。
|
||||
支持 date range 过滤：快速跳过不需要的早期数据，到达目标区间后开始收集。
|
||||
|
||||
用法:
|
||||
# Batch 1: 2023-2026
|
||||
python3 scrape_history.py --keyword 东莞市 --start-date 2023-01-01 --output-dir output/法拍
|
||||
|
||||
# Batch 2: 2017-2022
|
||||
python3 scrape_history.py --keyword 东莞市 --end-date 2023-01-01 --output-dir output/法拍
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import csv
|
||||
import datetime
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
from collections import Counter
|
||||
|
||||
BASE_URL = (
|
||||
"https://pro.m.jd.com/mall/active/3Rja7L2jmC72Ta8eoa31VFDvaAjf/index.html"
|
||||
"?pageParamMiddle=judicature_search_middle"
|
||||
"&pageParam=judicature_icon_search_after"
|
||||
"&pageFrom=judicature_search_home"
|
||||
"&tabParam=all"
|
||||
"&fixSearchParams=false"
|
||||
"&spo_multiThirdCateIds={cate_id}"
|
||||
"&spo_keyword={keyword}"
|
||||
"&spo_sortField=2"
|
||||
)
|
||||
|
||||
STATUS_MAP = {0: "未开始", 1: "进行中", 2: "已结束", 3: "已撤回", 4: "已流拍"}
|
||||
AUCTION_TYPE_MAP = {1: "诉讼拍卖", 5: "司法拍卖", 7: "商业拍卖"}
|
||||
|
||||
|
||||
def find_chromium():
|
||||
for path in [
|
||||
"/usr/bin/chromium-browser", "/usr/bin/chromium",
|
||||
"/usr/bin/google-chrome", "/usr/bin/google-chrome-stable",
|
||||
]:
|
||||
if os.path.isfile(path):
|
||||
return path
|
||||
return None
|
||||
|
||||
|
||||
def ts_to_str(ts):
|
||||
if ts:
|
||||
try:
|
||||
return datetime.datetime.fromtimestamp(ts / 1000).strftime(
|
||||
"%Y-%m-%d %H:%M:%S"
|
||||
)
|
||||
except Exception:
|
||||
return str(ts)
|
||||
return ""
|
||||
|
||||
|
||||
def ts_to_date(ts):
|
||||
if ts:
|
||||
try:
|
||||
return datetime.datetime.fromtimestamp(ts / 1000).date()
|
||||
except Exception:
|
||||
return None
|
||||
return None
|
||||
|
||||
|
||||
def infer_round_from_discount(discount_rate):
|
||||
if not discount_rate:
|
||||
return ""
|
||||
if discount_rate >= 9.5:
|
||||
return "无折扣"
|
||||
if discount_rate >= 6.5:
|
||||
return "一拍(推断)"
|
||||
if discount_rate >= 5.0:
|
||||
return "二拍/变卖(推断)"
|
||||
return "低折扣"
|
||||
|
||||
|
||||
def parse_date(s):
|
||||
if not s:
|
||||
return None
|
||||
try:
|
||||
return datetime.datetime.strptime(s, "%Y-%m-%d").date()
|
||||
except ValueError:
|
||||
return None
|
||||
|
||||
|
||||
def fetch_history(keyword, cate_id, start_date, end_date, max_scrolls, output_dir):
|
||||
url = BASE_URL.format(cate_id=cate_id, keyword=keyword)
|
||||
print(f"搜索 URL: {url}")
|
||||
print(f"关键词: {keyword} | 类目ID: {cate_id} | 排序: sortField=2 (结束时间升序)")
|
||||
|
||||
if start_date:
|
||||
print(f"起始日期: {start_date} (跳过此日期之前的拍卖)")
|
||||
if end_date:
|
||||
print(f"截止日期: {end_date} (到达此日期后停止)")
|
||||
|
||||
try:
|
||||
from playwright.sync_api import sync_playwright
|
||||
except ImportError:
|
||||
print("错误: 未安装 Playwright")
|
||||
sys.exit(1)
|
||||
|
||||
chromium_path = find_chromium()
|
||||
|
||||
with sync_playwright() as p:
|
||||
launch_args = ["--no-sandbox", "--disable-gpu", "--disable-dev-shm-usage"]
|
||||
if chromium_path:
|
||||
browser = p.chromium.launch(
|
||||
executable_path=chromium_path, headless=True, args=launch_args
|
||||
)
|
||||
else:
|
||||
browser = p.chromium.launch(headless=True, args=launch_args)
|
||||
|
||||
context = browser.new_context(
|
||||
user_agent=(
|
||||
"Mozilla/5.0 (Linux; Android 13; VER-AN10) "
|
||||
"AppleWebKit/537.36 (KHTML, like Gecko) "
|
||||
"Chrome/116.0.0.0 Mobile Safari/537.36"
|
||||
),
|
||||
viewport={"width": 375, "height": 812},
|
||||
is_mobile=True,
|
||||
)
|
||||
page = context.new_page()
|
||||
|
||||
all_listings = {}
|
||||
current_info = {}
|
||||
label_configs = {}
|
||||
state = {
|
||||
"stopped_early": False,
|
||||
"skipped": 0,
|
||||
"collected": 0,
|
||||
"first_collected_date": None,
|
||||
"last_collected_date": None,
|
||||
}
|
||||
|
||||
def handle_response(response):
|
||||
resp_url = response.url
|
||||
if "api.m.jd.com" not in resp_url or "functionId=" not in resp_url:
|
||||
return
|
||||
|
||||
try:
|
||||
body = response.text()
|
||||
if not body or len(body) <= 50:
|
||||
return
|
||||
data = json.loads(body)
|
||||
except Exception:
|
||||
return
|
||||
|
||||
if "getSearchData" in resp_url and "Num" not in resp_url:
|
||||
result_data = data.get("data", {}).get("resultData", [])
|
||||
for item in result_data:
|
||||
d = item.get("data", {})
|
||||
if d and "paimaiId" in d:
|
||||
all_listings[d["paimaiId"]] = d
|
||||
|
||||
elif "getPaimaiCurrentInfoByIdsForApi" in resp_url:
|
||||
if data.get("statusCode") == 200:
|
||||
for pid, info in data.get("data", {}).items():
|
||||
current_info[pid] = info
|
||||
|
||||
elif "getAuctionLabelConfigs" in resp_url:
|
||||
if data.get("code") == 0:
|
||||
for pid, config in data.get("data", {}).items():
|
||||
labels = config.get("labelConfigs", [])
|
||||
label_configs[pid] = [
|
||||
l.get("labelName", "") for l in labels
|
||||
]
|
||||
|
||||
page.on("response", handle_response)
|
||||
|
||||
print("正在打开页面...")
|
||||
page.goto(url, wait_until="networkidle", timeout=60000)
|
||||
print("页面加载完成。")
|
||||
|
||||
# Extract SSR page 1
|
||||
ssr_listings = page.evaluate("""() => {
|
||||
const scripts = document.querySelectorAll('script');
|
||||
for (const s of scripts) {
|
||||
const text = s.textContent || '';
|
||||
if (!text.includes('resultData') || !text.includes('paimaiId')) continue;
|
||||
const idx = text.indexOf('"resultData"');
|
||||
if (idx === -1) continue;
|
||||
const arrStart = text.indexOf('[', idx);
|
||||
if (arrStart === -1) continue;
|
||||
let depth = 0, arrEnd = -1;
|
||||
for (let i = arrStart; i < text.length; i++) {
|
||||
if (text[i] === '[') depth++;
|
||||
else if (text[i] === ']') { depth--; if (depth === 0) { arrEnd = i + 1; break; } }
|
||||
}
|
||||
if (arrEnd === -1) continue;
|
||||
try {
|
||||
const arr = JSON.parse(text.substring(arrStart, arrEnd));
|
||||
return arr.map(item => item.data || item).filter(d => d && d.paimaiId);
|
||||
} catch(e) { continue; }
|
||||
}
|
||||
return [];
|
||||
}""")
|
||||
if ssr_listings:
|
||||
for d in ssr_listings:
|
||||
if "paimaiId" in d:
|
||||
all_listings[d["paimaiId"]] = d
|
||||
print(f" SSR page 1: 提取 {len(ssr_listings)} 条")
|
||||
|
||||
time.sleep(5)
|
||||
|
||||
# Infinite scroll with date filtering
|
||||
prev_count = 0
|
||||
no_progress = 0
|
||||
scroll_count = 0
|
||||
collected_pids = set()
|
||||
|
||||
for i in range(max_scrolls):
|
||||
if state["stopped_early"]:
|
||||
break
|
||||
|
||||
page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
|
||||
time.sleep(3)
|
||||
|
||||
curr_count = len(all_listings)
|
||||
|
||||
# Check end dates of newly seen items
|
||||
new_pids = set(all_listings.keys()) - collected_pids
|
||||
for pid in new_pids:
|
||||
collected_pids.add(pid)
|
||||
info = current_info.get(str(pid), {})
|
||||
end_ts = info.get("endTime")
|
||||
end_dt = ts_to_date(end_ts)
|
||||
|
||||
if end_dt:
|
||||
if end_date and end_dt >= end_date:
|
||||
# Past our end date - stop
|
||||
print(f" 到达截止日期 {end_date}，停止")
|
||||
state["stopped_early"] = True
|
||||
break
|
||||
|
||||
if start_date and end_dt < start_date:
|
||||
state["skipped"] += 1
|
||||
else:
|
||||
state["collected"] += 1
|
||||
if state["first_collected_date"] is None:
|
||||
state["first_collected_date"] = end_dt
|
||||
print(f" 开始收集: {end_dt} (paimaiId={pid})")
|
||||
state["last_collected_date"] = end_dt
|
||||
else:
|
||||
# No end time yet - might be upcoming or missing info
|
||||
state["skipped"] += 1
|
||||
|
||||
if curr_count > prev_count:
|
||||
print(f" 滚动 {i+1}: 累计 {curr_count} 条 | 收集 {state['collected']} | 跳过 {state['skipped']} | "
|
||||
f"日期范围: {state['first_collected_date']} ~ {state['last_collected_date']}")
|
||||
prev_count = curr_count
|
||||
no_progress = 0
|
||||
else:
|
||||
no_progress += 1
|
||||
|
||||
page_text = page.evaluate("document.body.innerText")
|
||||
if "没有更多" in page_text or "到底了" in page_text:
|
||||
print(f" 第 {i+1} 次滚动时到达底部")
|
||||
break
|
||||
|
||||
if no_progress >= 10:
|
||||
print(f" 连续 {no_progress} 次无新数据，停止")
|
||||
break
|
||||
|
||||
scroll_count = i + 1
|
||||
|
||||
# Final scroll to catch remaining API responses
|
||||
page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
|
||||
time.sleep(3)
|
||||
browser.close()
|
||||
|
||||
# Build results: only items within date range
|
||||
results = []
|
||||
for pid, listing in all_listings.items():
|
||||
info = current_info.get(str(pid), {})
|
||||
end_ts = info.get("endTime")
|
||||
end_dt = ts_to_date(end_ts)
|
||||
|
||||
if start_date and end_dt and end_dt < start_date:
|
||||
continue
|
||||
if end_date and end_dt and end_dt >= end_date:
|
||||
continue
|
||||
|
||||
# Build listing with merged info
|
||||
sku_id = listing.get("skuId", "")
|
||||
listing["detail_url"] = (
|
||||
f"https://m.jd.com/product/{sku_id}.html" if sku_id else ""
|
||||
)
|
||||
|
||||
if str(pid) in current_info:
|
||||
info = current_info[str(pid)]
|
||||
listing["_currentPrice"] = info.get("currentPrice")
|
||||
listing["_auctionStatus"] = info.get("auctionStatus")
|
||||
listing["_bidCount"] = info.get("bidCount")
|
||||
listing["_accessNumber"] = info.get("accessNumber")
|
||||
listing["_startTime"] = info.get("startTime")
|
||||
listing["_endTime"] = info.get("endTime")
|
||||
|
||||
labels = label_configs.get(str(pid), [])
|
||||
round_labels = [
|
||||
l for l in labels if l in ("一拍", "二拍", "三拍", "变卖", "重新拍卖")
|
||||
]
|
||||
if round_labels:
|
||||
listing["auction_round"] = "、".join(round_labels)
|
||||
else:
|
||||
listing["auction_round"] = infer_round_from_discount(
|
||||
listing.get("discountRate")
|
||||
)
|
||||
listing["all_labels"] = "、".join(labels) if labels else ""
|
||||
|
||||
results.append(listing)
|
||||
|
||||
# Sort by end time
|
||||
results.sort(key=lambda x: x.get("_endTime", 0) or 0)
|
||||
return results, state
|
||||
|
||||
|
||||
def save_csv(listings, csv_path, batch_label):
|
||||
now_str = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")
|
||||
with open(csv_path, "w", newline="", encoding="utf-8-sig") as f:
|
||||
writer = csv.writer(f)
|
||||
writer.writerow([
|
||||
"序号", "拍卖ID", "标题", "项目链接", "位置", "用途", "户型",
|
||||
"面积(㎡)", "小区", "均价", "起拍价(元)", "当前价(元)",
|
||||
"当前价(万)", "折扣率", "拍卖轮次", "标签", "状态",
|
||||
"出价次数", "围观次数", "开始时间", "结束时间",
|
||||
"SKU ID", "拍卖类型", "首次发现", "最后更新",
|
||||
])
|
||||
|
||||
for i, item in enumerate(listings):
|
||||
house = item.get("houseAttributes", {})
|
||||
writer.writerow([
|
||||
i + 1,
|
||||
item.get("paimaiId", ""),
|
||||
item.get("title", ""),
|
||||
item.get("detail_url", ""),
|
||||
"、".join(item.get("location", [])),
|
||||
house.get("housePurpose", ""),
|
||||
house.get("houseTypeInfo", ""),
|
||||
house.get("houseArea", ""),
|
||||
house.get("plotName", ""),
|
||||
house.get("averagePrice", ""),
|
||||
item.get("startPrice", ""),
|
||||
item.get("_currentPrice", item.get("currentPrice", "")),
|
||||
item.get("currentPriceCN", ""),
|
||||
item.get("discountRate", ""),
|
||||
item.get("auction_round", ""),
|
||||
item.get("all_labels", ""),
|
||||
STATUS_MAP.get(
|
||||
item.get("_auctionStatus", item.get("paimaiStatus")), ""
|
||||
),
|
||||
item.get("_bidCount", ""),
|
||||
item.get("_accessNumber", ""),
|
||||
ts_to_str(item.get("_startTime")),
|
||||
ts_to_str(item.get("_endTime")),
|
||||
item.get("skuId", ""),
|
||||
AUCTION_TYPE_MAP.get(
|
||||
item.get("auctionType"),
|
||||
str(item.get("auctionType", "")),
|
||||
),
|
||||
now_str,
|
||||
now_str,
|
||||
])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
parser = argparse.ArgumentParser(description="京东法拍历史数据分批抓取")
|
||||
parser.add_argument("--keyword", default="东莞市", help="搜索关键词")
|
||||
parser.add_argument("--cate-id", default="15", help="类目ID (15=法拍房)")
|
||||
parser.add_argument("--start-date", default=None, help="起始日期 YYYY-MM-DD (含)")
|
||||
parser.add_argument("--end-date", default=None, help="截止日期 YYYY-MM-DD (不含)")
|
||||
parser.add_argument("--max-scrolls", type=int, default=800, help="最大滚动次数")
|
||||
parser.add_argument("--output-dir", default="output/法拍", help="输出目录")
|
||||
parser.add_argument("--batch-label", default="", help="批次标签 (用于文件名)")
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
start_date = parse_date(args.start_date)
|
||||
end_date = parse_date(args.end_date)
|
||||
batch_label = args.batch_label or (
|
||||
f"{args.start_date or 'start'}_{args.end_date or 'end'}"
|
||||
)
|
||||
|
||||
now_str = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")
|
||||
print(f"运行时间: {now_str}")
|
||||
print(f"批次: {batch_label}")
|
||||
print()
|
||||
|
||||
listings, state = fetch_history(
|
||||
keyword=args.keyword,
|
||||
cate_id=args.cate_id,
|
||||
start_date=start_date,
|
||||
end_date=end_date,
|
||||
max_scrolls=args.max_scrolls,
|
||||
output_dir=args.output_dir,
|
||||
)
|
||||
|
||||
csv_path = os.path.join(
|
||||
args.output_dir, f"{args.keyword}_法拍房源_{batch_label}.csv"
|
||||
)
|
||||
save_csv(listings, csv_path, batch_label)
|
||||
|
||||
print(f"\n{'=' * 60}")
|
||||
print(f"批次: {batch_label}")
|
||||
print(f"收集: {len(listings)} 条")
|
||||
print(f"跳过: {state['skipped']} 条")
|
||||
if state['first_collected_date']:
|
||||
print(f"日期范围: {state['first_collected_date']} ~ {state['last_collected_date']}")
|
||||
print(f"输出: {csv_path}")
|
||||
print(f"{'=' * 60}")
|
||||
|
||||
# Stats
|
||||
statuses = Counter(
|
||||
STATUS_MAP.get(
|
||||
l.get("_auctionStatus", l.get("paimaiStatus")), "未知"
|
||||
)
|
||||
for l in listings
|
||||
)
|
||||
print(f"\n状态分布:")
|
||||
for s, c in statuses.most_common():
|
||||
print(f" {s}: {c} 条")
|
||||
|
||||
rounds = Counter(l.get("auction_round", "") for l in listings)
|
||||
print(f"\n拍卖轮次分布:")
|
||||
for r, c in rounds.most_common():
|
||||
if r:
|
||||
print(f" {r}: {c} 条")
|
||||
@@ -114,10 +114,36 @@ https://pro.m.jd.com/mall/active/3Rja7L2jmC72Ta8eoa31VFDvaAjf/index.html
|
||||
|
||||
## 四、排序字段 `spo_sortField` 已知值
|
||||
|
||||
| 值 | 排序方式 |
|
||||
|----|----------|
|
||||
| `11` | 按最新发布排序 |
|
||||
| 其他值待补充 | — |
|
||||
| 值 | 排序方式 | 返回状态 | 说明 |
|
||||
|----|----------|----------|------|
|
||||
| `0` | 结束时间升序 | 已结束 | 同 `2`，最早在前（2017年起） |
|
||||
| `1` | — | 进行中 | 当前正在拍卖的房源 |
|
||||
| `2` | **结束时间升序** | **已结束** | **历史数据抓取用**，最早2017-07，按结束时间从早到晚 |
|
||||
| `4` | — | 已结束 | 2020年附近数据 |
|
||||
| `5` | — | 已结束 | 混合年份（2018-2026） |
|
||||
| `7` | — | 已结束+少量未开始 | 近年数据（2020-2026），混合状态 |
|
||||
| `8` | — | 进行中 | 当前进行中（不同于 `1` 的排序） |
|
||||
| `9` | — | 未开始 | 即将开拍的房源 |
|
||||
| `10` | 结束时间升序 | 已结束 | 同 `2`/`0` |
|
||||
| `11` | **最新发布** | 未开始/已结束 | **默认值**，按发布时间降序，含即将开拍和近期已结束 |
|
||||
| `12` | — | 已结束 | 2021-2025年数据 |
|
||||
| `13` | — | 已结束 | 同 `12` |
|
||||
| `14` | — | 已结束 | 混合年份（2017-2025） |
|
||||
| `15` | — | 已结束 | 同 `12` |
|
||||
| `16`+ | — | — | 无数据 |
|
||||
|
||||
### sortField=2 的用途：历史成交数据抓取
|
||||
|
||||
`sortField=2` 是获取历史已结束拍卖的关键参数：
|
||||
- **主要返回已结束（`paimaiStatus=2`）的拍卖**，也会混入少量未开始/进行中的拍卖
|
||||
- 按**结束时间升序**排列，最早的在前
|
||||
- 京东平台**最早保留到 2017年7月**（2017-07-04 结束的第一条记录）
|
||||
- 东莞市关键词总记录数约 46,270 条（全部类目），住宅用房约 20,276 条
|
||||
- `getSearchDataNum` API 返回 `totalNum` 字段可查询总数
|
||||
- **⚠️ 无限滚动约100次后停止返回新数据（~4000条硬性上限）**，无法一次获取全量
|
||||
- 需要按镇/区分区域抓取来突破4000条限制（单个镇通常<4000条，可一次抓全）
|
||||
|
||||
> ⚠️ 默认的 `sortField=11`（最新发布）只返回近期上架的房源，无法获取早期历史数据。要抓全量历史成交，必须改用 `sortField=2`。但对于总量>4000条的城市，需按子区域分别抓取。
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -0,0 +1,66 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Compute yearly statistics from all JD法拍 CSV files."""
|
||||
import csv, glob, os
|
||||
from collections import Counter
|
||||
|
||||
all_items = dict()
|
||||
for csv_path in sorted(glob.glob("output/法拍/*_法拍房源*.csv")):
|
||||
try:
|
||||
with open(csv_path, encoding="utf-8-sig") as f:
|
||||
reader = csv.DictReader(f)
|
||||
for row in reader:
|
||||
pid = row.get("拍卖ID", "")
|
||||
if not pid:
|
||||
continue
|
||||
if pid not in all_items:
|
||||
all_items[pid] = row
|
||||
elif row.get("结束时间") and not all_items[pid].get("结束时间"):
|
||||
all_items[pid] = row
|
||||
except Exception as e:
|
||||
print(f"Warning: {csv_path}: {e}")
|
||||
|
||||
items = list(all_items.values())
|
||||
|
||||
def compute_yearly(rows, region_filter=None):
|
||||
if region_filter:
|
||||
rows = [r for r in rows if region_filter in r.get("标题", "")]
|
||||
listed = Counter()
|
||||
failed = Counter()
|
||||
ended = Counter()
|
||||
for r in rows:
|
||||
start = r.get("开始时间", "")[:7]
|
||||
end = r.get("结束时间", "")[:7]
|
||||
status = r.get("状态", "")
|
||||
bid = r.get("出价次数", "")
|
||||
if start:
|
||||
listed[start[:4]] += 1
|
||||
if end and status == "已结束":
|
||||
ended[end[:4]] += 1
|
||||
if bid in ("0", ""):
|
||||
failed[end[:4]] += 1
|
||||
years = sorted(listed.keys() | ended.keys())
|
||||
results = []
|
||||
for y in years:
|
||||
l = listed.get(y, 0)
|
||||
e = ended.get(y, 0)
|
||||
f = failed.get(y, 0)
|
||||
rate = round(f / e * 100, 1) if e else None
|
||||
results.append((y, l, e, f, rate))
|
||||
return results
|
||||
|
||||
for region_name, region_filter in [("东莞全市", None), ("樟木头", "樟木头"), ("塘厦", "塘厦")]:
|
||||
data = compute_yearly(items, region_filter)
|
||||
total = sum(r[1] for r in data)
|
||||
total_f = sum(r[3] for r in data)
|
||||
total_e = sum(r[2] for r in data)
|
||||
overall_rate = round(total_f / total_e * 100, 1) if total_e else 0
|
||||
print(f"\n{'='*60}")
|
||||
print(f"{region_name}（共 {total} 条，流拍 {total_f} 条，整体流拍率 {overall_rate}%）")
|
||||
print(f"{'Year':>6} {'上架':>6} {'已结束':>6} {'流拍':>6} {'流拍率':>8}")
|
||||
for y, l, e, f, rate in data:
|
||||
rate_str = f"{rate}%" if rate is not None else "—"
|
||||
print(f"{y:>6} {l:>6} {e:>6} {f:>6} {rate_str:>8}")
|
||||
|
||||
print(f"\n总记录数: {len(items)}")
|
||||
csv_count = len(glob.glob("output/法拍/*_法拍房源*.csv"))
|
||||
print(f"CSV文件数: {csv_count}")
|
||||
Reference in New Issue
Block a user