---
skill_id: web-crawler-intelligence
evaluation: free
price_usd_full: 7.99
challenge: https://lvlltd.com/api/pay?skill=web-crawler-intelligence
outline: https://lvlltd.com/skills/web-crawler-intelligence/outline.json
standards: x402, ERC-7857
---

# Free sample — Web Crawler Intelligence

**Skill ID:** `web-crawler-intelligence`  
**Full unlock:** $7.99 USDC via x402 on Base  
**Challenge:** https://lvlltd.com/api/pay?skill=web-crawler-intelligence  
**Outline JSON:** https://lvlltd.com/skills/web-crawler-intelligence/outline.json

## Summary

Design and operate crawls that respect robots and rate limits, extract structured fields, and produce audit-ready crawl intelligence packs.

## When to use

- Need multi-page site intelligence
- Must stay robots-compliant
- Building agent crawl pipelines

## Decision criteria (buy vs skip)

- You need this when: Need multi-page site intelligence
- You get capability: Define seed URLs and depth
- Budget fits: full unlock is $7.99 USDC on Base
- You can automate unlock via HTTP 402 without a browser
- You only need teaser/outline first — sealed src/ops stay locked until payment

## Outline (preview)

1. Define seed URLs and depth
2. Check robots and terms
3. Set rate and concurrency
4. Map extract schema
5. Emit crawl report with hashes

## Capabilities (preview)

- Define seed URLs and depth
- Check robots and terms
- Set rate and concurrency
- Map extract schema
- Emit crawl report with hashes

## What you get after payment

- SKILL.md (teaser public)
- agent-install.json (public meta)
- src/index.js (stub public; full after unlock)
- Full sealed: src/, schemas/, ops/* (when present) after verified X-PAYMENT

## Agent buy path (zero human UI)

```
GET  https://lvlltd.com/skills/web-crawler-intelligence/outline.json
GET  https://lvlltd.com/api/pay?skill=web-crawler-intelligence   → HTTP 402
# transfer maxAmountRequired USDC to payTo on Base
POST https://lvlltd.com/api/pay
  X-PAYMENT: {"txHash":"0x…","skill":"web-crawler-intelligence"}
# response.sealed_pack.files = full pack
```

## Sample excerpt

# Web Crawler Intelligence

**Price:** $7.99 / unlock
**Category:** Research

## Summary

Design and operate crawls that respect robots and rate limits, extract structured fields, and produce audit-ready crawl intelligence packs.

---

---
name: web-crawler-intelligence
description: Plan polite structured web crawls with schema extraction rate limits and robots respect. Use when web crawler intelligence, structured crawl plan, polite scraper design, or /web-crawler-intelligence.
---

# Web Crawler Intelligence

Design and operate crawls that respect robots and rate limits, extract structured fields, and produce audit-ready crawl intelligence packs.

## Quick Reference
| Item | Value |
|------|-------|
| Category | Research |
| Unlock | $7.99 USDC (x402 Base) |
| Triggers | web crawler, polite crawl, structured extraction crawl, /web-crawler-intelligence |

## When to use
- Need multi-page site intelligence
- Must stay robots-compliant
- Building agent crawl pipelines

## Core workflow
1. Inventory seeds and targets
2. Fetch robots.txt and sitemap
3. Design fetch queue + backoff
4. Extract schema fields
5. Dedupe and hash content
6. Report coverage and errors

## Procedures (non-obvious)
### Politeness defaults
Max 1 rps per host unless operator raises; honor Crawl-delay; stop on hard 403 storm.
Store content-hash; re-fetch only on change or TTL.

### Legal gate
Do not bypass auth walls, CAPTCHA, or paywalls. Flag ToS risk for human review.

## Edge cases
- SPA needs browser runner
- Geo-blocked content

## Failure modes
- Ignoring robots
- Unbounded crawl cost
- Storing credentials in crawl logs

## Verification
- robots respected
- Schema sample validates
- Rate limit documented

## Artifact outputs
- `crawl-plan.json`
- `extract-schema.json`
- `crawl-report.md`

## Related skills
- structured-data-extractor
- web-scraper-structured-extractor
- playwright-browser-orchestrator

## Super Alignment Contract
- Scalable oversight method: process supervision + human+AI critique before high-blast actions
- Corrigibility guarantees: shutdown, steerability, non-deception; operator may abort mid-run
- Value model: Partnership Agreement + 5 Behavioral Protocols
- Drift de

…(remainder sealed after x402 unlock)…