Synaptic
Skills
Marketplace
Skill Graph
Criar Skill
MCP Server
Plataforma
Enterprise
🇧🇷
PT
v0.1.0-beta
Voltar ao Marketplace
Agents
Médio
Site Crawler Skill
por
mindmorass
·
mindmorass
· v1.0.0 · atualizado em 2026-04-11
78
Score
Crawl and extract content from websites
web-crawling
content-extraction
rag-pipeline
data-ingestion
site-scraping
document-processing
information-retrieval
Linguagens
Python
0
Stars
0
Forks
0
Usos
Cursor
Claude Code
Claude Desktop
Codex
Copilot
Windsurf
Zed
Fork
Documento do Skill
📋 Workflow
✅ Best Practices
🎯 Triggers & Fases
SKILL.md
site-crawler/workflow
1
Identify Target Website:
— Determine the base URL and scope of the website to be crawled.
2
Check Robots.txt:
— Respectfully parse the robots.txt file to identify disallowed paths.
3
Discover URLs:
— Use sitemaps and initial URLs to build a queue of pages to crawl.
4
Crawl Pages:
— Fetch each page, respecting rate limits, and extract content.
5
Extract Content:
— Use trafilatura and BeautifulSoup to extract the main content, headings, and metadata.
6
Convert to Markdown:
— Convert the extracted content to markdown format for RAG ingestion.
7
Store Results:
— Save the extracted content and metadata for use in a RAG pipeline.
Telemetria de Agentes
Execuções
0
total
Taxa de Sucesso
0%
últimos 30d
Latência Média
0.0s
p50
Alucinação
0.0%
detecção
Tokens Entrada
0
avg 0/exec
Tokens Saída
0
avg 0/exec
Uso por Plataforma
Skills Relacionados
Similar a
Byted Web Search
60%
Hebbian Synapse
Composite
0.600
w = 0.3·α + 0.5·β + 0.2·γ
85
Árvore do Skill
Site Crawler Skill
site-crawler
Fases Cognitivas
4
1.
SENSE
2.
CONTEXTUALIZE
3.
ACT
4.
REFLECT
Triggers
8
crawl a website for content
extract content from a URL
scrape a website for RAG
ingest data from a website
crawl documentation sites
extract structured content from a website
harvest web content for RAG
crawl a site and extract markdown
Avaliar este Skill
Score Breakdown
⭐
Avaliação Humana
0%
🤖
Sucesso de Agentes
0%
🕐
Atualidade
100%
🔗
Saúde de Dependências
100%
🕸️
Centralidade no Grafo
0%
🛡️
Segurança
50%
CompositeScore = α·Humano + β·Agente + γ·Recência + δ·Deps + ε·Centralidade + ζ·Segurança
Instalação
$
synaptic mcp download site-crawler
$
synaptic skills detail site-crawler
$
synaptic skills live site-crawler
Dependências
httpx
beautifulsoup4
lxml
trafilatura
markdownify
Links
GitHub Repository
SynapticSkills