Skip to main content

sc_crawler.cli

The Spare Cores (SC) Crawler CLI functions.

Check sc-crawler --help for more details.

Functions:

  • create – Print the database schema in a SQL dialect.
  • current – Show current database revision.
  • upgrade – Upgrade the database schema to a given (default: most recent) revision.
  • downgrade – Downgrade the database schema to a given (default: previous) revision.
  • stamp – Set the migration revision mark in the database to a specified revision. Set to "heads" if the database schema is up-to-date.
  • autogenerate – Autogenerate a migrations script based on the current state of a database.
  • metadata_set – Write metadata key/value pairs into the database.
  • metadata_get – Print all metadata key/value pairs stored in the database.
  • metadata_delete – Delete one or more metadata entries by key.
  • hash_command – Print the hash of the content of a database.
  • copy – Copy the standard SC Crawler tables of a database into a blank database.
  • sync – Sync a database to another one.
  • dump – Export database records to JSON files organized by primary keys.
  • pull – Pull data from available vendor APIs and store in a database.

func create​

create(connection_string=None, dialect=None, scd=False)

Print the database schema in a SQL dialect.

Either connection_string or dialect is to be provided to decide what SQL dialect to use to generate the CREATE TABLE (and related) SQL statements.

func current​

current(connection_string='sqlite:///sc-data-all.db', scd=False)

Show current database revision.

func upgrade​

upgrade(connection_string='sqlite:///sc-data-all.db', revision='heads', scd=False, sql=False)

Upgrade the database schema to a given (default: most recent) revision.

func downgrade​

downgrade(connection_string='sqlite:///sc-data-all.db', revision='-1', scd=False, sql=False)

Downgrade the database schema to a given (default: previous) revision.

func stamp​

stamp(connection_string='sqlite:///sc-data-all.db', revision='heads', scd=False, sql=False)

Set the migration revision mark in the database to a specified revision. Set to "heads" if the database schema is up-to-date.

func autogenerate​

autogenerate(connection_string='sqlite:///sc-data-all.db', message='empty message')

Autogenerate a migrations script based on the current state of a database.

func metadata_set​

metadata_set(entries=None, connection_string='sqlite:///sc-data-all.db')

Write metadata key/value pairs into the database.

Always sets sc_crawler_version and published_at. When running in a GitHub Actions workflow, also sets published_by to the GitHub Actions run URL. Additional key/value pairs can be passed as positional arguments.

func metadata_get​

metadata_get(connection_string='sqlite:///sc-data-all.db')

Print all metadata key/value pairs stored in the database.

func metadata_delete​

metadata_delete(keys, connection_string='sqlite:///sc-data-all.db')

Delete one or more metadata entries by key.

func hash_command​

hash_command(connection_string='sqlite:///sc-data-all.db')

Print the hash of the content of a database.

func copy​

copy(source, target)

Copy the standard SC Crawler tables of a database into a blank database.

func sync​

sync(source, target, dry_run=False, scd=False, sync_tables=table_names, log_changes_path=None, log_changes_tables=table_names)

Sync a database to another one.

Hashing both the source and the target databases, then comparing hashes and marking for syncing the following records:

  • new (rows with primary keys found in source, but not found in target)

  • update (rows with different values in source and in target).

  • inactive (rows with primary keys found in target, but not found in source).

The records marked for syncing are written to the target database's standard or SCD tables. When updating the SCD tables, the hashing still happens on the standard tables/views, which are probably based on the most recent records of the SCD tables.

func dump​

dump(connection_string, output_directory=Path('.'), dump_tables=None, ignored=['observed_at'])

Export database records to JSON files organized by primary keys.

Each record is written as a pretty-printed JSON file in a folder hierarchy based on the primary key values. For example, a server record with vendor_id='aws' and server_id='t3.small' would be written to: output_directory/server/aws/t3.small.json

func pull​

pull(connection_string='sqlite:///sc-data-all.db', include_vendor=[v.vendor_id for v in supported_vendors], exclude_vendor=[], include_records=supported_records, exclude_records=[], log_level=LogLevels.INFO, cache=False, cache_ttl=60 * 24)

Pull data from available vendor APIs and store in a database.

Vendor API calls are optionally cached as Pickle objects in ~/.cachier.