NotexAI
  • Quickstart
  • Python SDK
  • Authentication
  • Examples
    • Navigation
    • Scraping Flow
  • Browser Sessions
    • Session Management
    • Start Session
    • Close Session
    • Health
  • Browser Navigation
    • Observe Page
    • Step In Page
    • Scrape Data
Powered by GitBook
On this page
  1. Examples

Scraping Flow

Creating a Scraping Flow with NotexAI

This guide demonstrates how to use NotexAI to scrape an HBS news article and summarize it using OpenAI.


1. Use the Scrape Endpoint

Send a request to scrape a webpage:

curl --location 'https://api.notexai.pro/env/scrape/' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer your-api-key' \
--data '{
    "url": "https://www.hbs.edu/news/articles/Pages/awa-ambra-seck-profile-2024.aspx",
    "scrape_images": false,
    "screenshot": false
}'

Response:

{
    "session_id": "31f613a4-068e-464d-88b1-b8eb5d4d5c6f",
    "error": null,
    "title": "New Faculty Profiles: Awa Ambra Seck - News - Harvard Business School",
    "url": "https://www.hbs.edu/news/articles/Pages/awa-ambra-seck-profile-2024.aspx",
    "timestamp": "2025-01-08T14:23:32.969858",
    "screenshot": null,
    "data": {
        "markdown": "...",  
        "images": null,
        "structured": null
    },
    "space": null
}

2. Extracted Data

The response includes structured data extracted from the page, such as the article title, metadata, and content. Here is an example:

# New Faculty Profiles: Awa Ambra Seck
## Article Metadata
- Date: 18 DEC 2024
- Title: New Faculty Profiles: Awa Ambra Seck
- Author: Harvard Business School

## Article Content
### Introduction
HBS faculty comprises scholars and practitioners who bring leading-edge research, extensive experience, and deep insights into the classroom, to organizations, and to leaders across the globe.

### Interview with Awa Ambra Seck
#### Educational Background
Awa Ambra Seck has a rich educational background, including a bachelor’s degree from the University of Torino and a PhD from Harvard University, with a focus on development economics and political economy.

#### Area of Research
Her research focuses on the intersection of development economics and economic history, particularly in Africa.

#### Teaching and Interests
She teaches Business, Government & the International Economy and has interests in cooking, dancing, and painting.

3. Summarize with OpenAI

Use OpenAI’s API to summarize the extracted content:

curl https://api.openai.com/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -d '{
     "model": "gpt-4o-mini",
     "messages": [
        {"role": "system", "content": "You are a helpful assistant that summarizes news articles in max 100 words."},
        {"role": "user", "content": "...extracted article content..."}
     ],
     "temperature": 0.7
   }'

Summary Output:

Awa Ambra Seck is a new faculty member at Harvard Business School specializing in development economics and political economy with a focus on Africa. She has degrees from the University of Torino, Bocconi University, and a PhD from Harvard. Her research explores the impact of cultural traits and colonial history on Africa’s economic systems. Outside academia, she enjoys cooking, dancing, and painting.

That’s it! You’ve successfully created a scraping flow and summarized the content using NotexAI and OpenAI. 🌌

PreviousNavigationNextSession Management

Last updated 4 months ago