Skip to main content

sc_crawler.vendor_helpers

Functions:

func server_status_from_availability_categories​

server_status_from_availability_categories(categories, category_to_status, *, missing_status=Status.INACTIVE)

Map vendor availability categories to a single Server/Database status.

Each category is mapped via category_to_status. If more than one status results, Status.best selects the best. Empty or unmapped categories return missing_status.

func fetch_servers​

fetch_servers(fn, where, vendor)

Fetch servers of a region/zone.

Parameters:

  • fn (Callable) – A function that takes the region or zone id as its first and only argument. The returning list must conform with the Server object, or need to be in a format that preprocess_servers's fn can manage.
  • where (str) – A Region or Zone api_reference or similar that is passed to fn.
  • vendor (Optional[Vendor]) – Optional Vendor instance used for logging and progress bar updates.

func parallel_fetch_servers​

parallel_fetch_servers(vendor, fn, id_col, by)

Fetch servers of all regions/zones in parallel on 8 threads.

Parameters:

  • vendor (Vendor) – Required Vendor instance used for the regions lookup, logging and progress bar updates.
  • fn (Callable) – A function to be passed to fetch_servers.
  • id_col (str) – Field name to be used to deduplicate the list of server dicts.
  • by (Literal['regions', 'zones']) – What objects of the vendor to iterate on.

func preprocess_servers​

preprocess_servers(servers, vendor, fn)

Preprocess servers before inserting into the database.

Takes a list of dicts and tranform to a list of dicts that follows the Server schema.

Parameters:

  • servers (List[dict]) – To be passed to fn.
  • vendor (Vendor) – The related Vendor instance used for database connection, logging and progress bar updates.
  • fn (Callable) – A function that takes a server from servers (one-by-one) and the vendor.

func add_vendor_id​

add_vendor_id(obj, vendor)

Adds vendor_id field to a dict.

func get_region_by_id​

get_region_by_id(region_id, vendor)

Get a region by its ID or alias.

Parameters:

  • region_id (str) – The ID or alias of the region to get.
  • vendor (Vendor) – The vendor to get the region from.

Returns: