Skip to main content

sc_crawler.table_bases

Tiny helper classes for the most commonly used fields to be inherited by sc_crawler.tables.

Classes:

class ScMetaModel​

Bases: __class__

Custom class factory to auto-update table models.

  • Reuse description of the table and its fields as SQL comment.

    Checking if the table and its fields have explicit comment set to be shown in the CREATE TABLE statements, and if not, reuse the optional table and field descriptions. Table docstrings are truncated to first line.

  • Reuse description of the fields to dynamically append to the docstring in the Attributes section.

  • Set __validator__ to the parent Pydantic model without table=True, which is useful for running validations. The Pydantic model is found by the parent class' name ending in "Base".

  • Auto-generate SCD table docs from the non-SCD table docs.

class ScModel​

Bases: SQLModel

Custom extensions to SQLModel objects and tables.

Extra features:

  • auto-generated table names using snake_case,
  • support for hashing table rows,
  • reuse description field of tables/columns as SQL comment,
  • reuse description field of columns to extend the Attributes section of the docstring.

Functions:

  • get_columns – Return the table's column names in a dict for all, primary keys, and attributes.
  • get_table_name – Return the SQLModel object's table name.
  • get_validator – Return the parent Base Pydantic model (without a table definition).
  • get_scd – Return the SCD version of the SQLModel table.
  • hash – Hash the content of the rows.

meth 
ScModel.get_columns
​

get_columns()

Return the table's column names in a dict for all, primary keys, and attributes.

meth 
ScModel.get_table_name
​

get_table_name()

Return the SQLModel object's table name.

meth 
ScModel.get_validator
​

get_validator()

Return the parent Base Pydantic model (without a table definition).

meth 
ScModel.get_scd
​

get_scd()

Return the SCD version of the SQLModel table.

meth 
ScModel.hash
​

hash(session, ignored=['observed_at'], progress=None)

Hash the content of the rows.

Parameters:

  • session (Session) – Database connection to use for object lookups.
  • ignored (List[str]) – List of column names to exclude from hashing.
  • progress (Optional[Progress]) – Optional progress bar to track the status of the hashing.

Returns:

  • dict – Dictionary of the row hashes keyed by the JSON dump of primary keys.

class MetaColumns​

Bases: ScModel

Helper class to add the status and observed_at columns.

class HasPriceFieldsBase​

Bases: ScModel

class ServerFields​

Bases: HasDisplayName, HasApiReference, HasName, HasServerIdPK, HasVendorPKFK

class BenchmarkFields​

Bases: HasBenchmarkIdPK

class ResourceIdComparator​

Bases: Comparator

Compile server_id/database_id to CASE + typed filter.

class BenchmarkScoreFields​

Bases: HasBenchmarkPKFK, HasVendorPKFK

Functions:

meth 
BenchmarkScoreFields.update_config_to_hashable
​

update_config_to_hashable(values)

Accept server_id / database_id keys; hashably sort config.