計畫簡介
ScrapeGraphAI 是一個基於Python的Web抓取庫，使用大型語言模型和直接圖邏輯來建立針對網站、文件和XML檔的抓取流程。使用者只需指定想要抽取的資訊，該庫便能自動完成抓取任務。該計畫強調易用性和高效性，支持透過命令列界面或程式碼實作靈活的數據抓取，並提供豐富的文件支持，幫助使用者快速上手。
掃碼加入交流群
獲得更多技術支持和交流
（請註明自己的職業）
快速安裝
這是 Scrapegraph-ai 的官方 PyPI 頁面的參考資訊
https://pypi.org/project/scrapegraphai/
pip install scrapegraphai
你還需要安裝 Playwright 用於基於 JavaScript 的網頁抓取：
playwright install
註意：建議在虛擬環境中安裝該庫，以避免與其他庫沖突
DEMO
https://scrapegraph-ai-demo.streamlit.app/
使用
你可以使用 SmartScraper 類透過提示從網站中提取資訊。
SmartScraper 類是一個直接圖實作，使用了網頁抓取流程中最常見的節點。更多資訊請檢視文件。
https://scrapegraph-ai.readthedocs.io/en/latest/
案例 1：使用 Ollama 提取資訊記得要單獨下載 Ollama 的模型！
from scrapegraphai.graphs import SmartScraperGraphgraph_config = {"llm": {"model": "ollama/mistral","temperature": 0,"format": "json", # Ollama needs the format to be specified explicitly"base_url": "http://localhost:11434", # set Ollama URL },"embeddings": {"model": "ollama/nomic-embed-text","base_url": "http://localhost:11434", # set Ollama URL }}smart_scraper_graph = SmartScraperGraph( prompt="List me all the articles",# also accepts a string with the already downloaded HTML code source="https://perinim.github.io/projects", config=graph_config)result = smart_scraper_graph.run()print(result)
案例 2：使用 Docker 提取資訊
註意：在使用本地模型之前，記得建立 Docker 容器！
docker-compose up -d docker exec -it ollama ollama pull stablelm-zephyr
你可以使用 Ollama 上可用的模型或者你自己的模型，而不是使用 stablelm-zephyr。
from scrapegraphai.graphs import SmartScraperGraphgraph_config = {"llm": {"model": "ollama/mistral","temperature": 0,"format": "json", # Ollama needs the format to be specified explicitly # "model_tokens": 2000, # set context length arbitrarily },}smart_scraper_graph = SmartScraperGraph( prompt="List me all the articles",# also accepts a string with the already downloaded HTML code source="https://perinim.github.io/projects", config=graph_config)result = smart_scraper_graph.run()print(result)
案例 3：使用 OpenAI 模型提取資訊
from scrapegraphai.graphs import SmartScraperGraphOPENAI_API_KEY = "YOUR_API_KEY"graph_config = {"llm": {"api_key": OPENAI_API_KEY,"model": "gpt-3.5-turbo", },}smart_scraper_graph = SmartScraperGraph( prompt="List me all the articles",# also accepts a string with the already downloaded HTML code source="https://perinim.github.io/projects", config=graph_config)result = smart_scraper_graph.run()print(result)
案例 4：使用 Groq 提取資訊
from scrapegraphai.graphs import SmartScraperGraphfrom scrapegraphai.utils import prettify_exec_infogroq_key = os.getenv("GROQ_APIKEY")graph_config = {"llm": {"model": "groq/gemma-7b-it","api_key": groq_key,"temperature": 0 },"embeddings": {"model": "ollama/nomic-embed-text","temperature": 0,"base_url": "http://localhost:11434", },"headless": False}smart_scraper_graph = SmartScraperGraph( prompt="List me all the projects with their description and the author.", source="https://perinim.github.io/projects", config=graph_config)result = smart_scraper_graph.run()print(result)
案例 5：使用 Azure 提取資訊
from langchain_openai import AzureChatOpenAIfrom langchain_openai import AzureOpenAIEmbeddingslm_model_instance = AzureChatOpenAI( openai_api_version=os.environ["AZURE_OPENAI_API_VERSION"], azure_deployment=os.environ["AZURE_OPENAI_CHAT_DEPLOYMENT_NAME"])embedder_model_instance = AzureOpenAIEmbeddings( azure_deployment=os.environ["AZURE_OPENAI_EMBEDDINGS_DEPLOYMENT_NAME"], openai_api_version=os.environ["AZURE_OPENAI_API_VERSION"],)graph_config = {"llm": {"model_instance": llm_model_instance},"embeddings": {"model_instance": embedder_model_instance}}smart_scraper_graph = SmartScraperGraph( prompt="""List me all the events, with the following fields: company_name, event_name, event_start_date, event_start_time, event_end_date, event_end_time, location, event_mode, event_category, third_party_redirect, no_of_days, time_in_hours, hosted_or_attending, refreshments_type, registration_available, registration_link""", source="https://www.hmhco.com/event", config=graph_config)
案例 6：使用 Gemini 提取資訊
from scrapegraphai.graphs import SmartScraperGraphGOOGLE_APIKEY = "YOUR_API_KEY"# Define the configuration for the graphgraph_config = {"llm": {"api_key": GOOGLE_APIKEY,"model": "gemini-pro", },}# Create the SmartScraperGraph instancesmart_scraper_graph = SmartScraperGraph( prompt="List me all the articles", source="https://perinim.github.io/projects", config=graph_config)result = smart_scraper_graph.run()print(result)
以上三個案例的輸出都將是一個包含提取資訊的字典，例如：
{'titles': ['Rotary Pendulum RL' ],'descriptions': ['Open Source project aimed at controlling a real life rotary pendulum using RL algorithms' ]}
計畫連結
https://github.com/VinciGit00/Scrapegraph-ai
關註「 開源AI計畫落地 」公眾號
與AI時代更靠近一點
