Skip to main content

sc_crawler.insert

Functions:

func can_bulk_insert​

can_bulk_insert(session)

Checks if bulk insert is supported for the engine dialect of a SQLModel session.

func validate_items​

validate_items(model, items, vendor=None, prefix='')

Validates a list of items against a pydantic.BaseModel definition.

Parameters:

  • model (BaseModel) – An SQLModel model to be used for validation.
  • items (List[dict]) – List of dictionaries to be checked against model.
  • vendor (Optional[Vendor]) – Optional Vendor instance used for logging and progress bar updates.
  • prefix (str) – Optional extra description for the model added in front of the model name in logs and progress bar updates.

Returns:

  • List[dict] – List of validated dicts in the same order. Note that missing fields has been filled in with default values (needed for bulk inserts).

func bulk_insert_items​

bulk_insert_items(model, items, vendor=None, session=None, progress=None, prefix='')

Bulk inserts items into a SQLModel table with ON CONFLICT update.

Parameters:

  • model (SQLModel) – An SQLModel table definition with primary key(s).
  • items (List[dict]) – List of dicts with all columns of the model.
  • vendor (Optional[Vendor]) – Optional related Vendor instance used for logging and progress bar updates.
  • session (Optional[Session]) – Optional database connections. When not provided, defaults to the vendor's session.
  • progress (Optional[Progress]) – Optional progress bar to use instead of vendor's progress bar.
  • prefix (str) – Optional extra description for the model added in front of the model name in logs and progress bar updates.

func insert_items​

insert_items(model, items, vendor=None, session=None, progress=None, prefix='')

Insert items into the related database table using bulk or merge.

Bulk insert is only supported with SQLite, other databases fall back to the default session.merge (slower) approach.

Parameters:

  • model (SQLModel) – An SQLModel table definition with primary key(s).
  • items (List[dict]) – List of dicts with all columns of the model.
  • vendor (Optional[Vendor]) – Optional related Vendor instance used for database connection, logging and progress bar updates.
  • session (Optional[Session]) – Optional database connections. When not provided, defaults to the vendor's session.
  • progress (Optional[Progress]) – Optional progress bar to use instead of vendor's progress bar.
  • prefix (str) – Optional extra description for the model added in front of the model name in logs and progress bar updates.