Vendor support
Each file in the src/sc_crawler/vendors folder provides the required helpers for a given Vendor, named as the id of the vendor prefixed with an underscore. For example, _aws.py provides functions to be used by its Vendor instance, called aws.
First steps
- Define the new Vendor instance in
src/sc_crawler/vendors/vendors.py. - Copy the below template file as a starting point to
src/sc_crawler/vendors/_<vendor_id>.py. - Update
src/sc_crawler/vendors/__init__.pyto include the new vendor. - Update
docs/add_vendor.mdwith the credential requirements for the new vendor. - Implement the
inventorymethods. - Preferably also implement the related cloud-discovery tool in the
resource-trackerpackage. - Add the new vendor to
_boot_from_attached_network_drivein_find_storage_disks_from_lsblk(src/sc_crawler/inspector.py). The value (TrueorFalse) must match whether the vendor's Pulumi provisioning script in the sc-runner repo attaches a network block volume as the boot drive: checksrc/sc_runner/resources/<vendor_id>.py.
Inventory methods
Each vendor module should provide the below functions:
inventory_compliance_frameworks: DefineVendorComplianceLinkinstances to describe which frameworks the vendor complies with. Optionally include references in thecommentfield. To avoid duplicatingComplianceFrameworkinstances, easiest is to use thecompliance_framework_idfield instead of thecompliance_frameworkrelationship, preferably via sc_crawler.lookup.map_compliance_frameworks_to_vendor.inventory_regions: DefineRegioninstances with location, energy source etc for each region the vendor has.inventory_zones: Define aZoneinstance for each availability zone of the vendor in each region.inventory_servers: DefineServerinstances for the vendor's server/instance types.inventory_server_prices: Define theServerPriceinstances for the standard/ondemand (or optionally also for the reserved) pricing of the instance types per region and zone. When applicable, include the monthly cap for tiered pricing in theprice_tieredfield.inventory_server_prices_spot: Similar to the above, defineServerPriceinstances but theallocationfield set toAllocation.SPOT. Very likely to see different spot prices per region/zone.inventory_storage_prices: DefineStoragePriceinstances to describe the available storage options that can be attached to the servers.inventory_databases: DefineDatabaseinstances for managed database instances (e.g. PostgreSQL).inventory_database_prices: DefineDatabasePriceinstances for compute pricing per region.haandha_strategyare scalar primary-key fields (one price row per HA deployment). Use hourlypricewith optional monthly cap inprice_tiered.inventory_database_storages: DefineDatabaseStorageinstances for decoupled or add-on storage products.inventory_database_storage_prices: DefineDatabaseStoragePriceinstances for storage pricing per region.inventory_traffic_prices: DefineTrafficPriceinstances to describe the pricing of ingress/egress traffic.inventory_ipv4_prices: DefineIpv4Priceinstances on the price of an IPv4 address.
Each function will be picked up as the related Vendor instance's instance methods, so each function should take a single argument, that is the Vendor instance. E.g. sc_crawler.vendors._aws.inventory_regions is called by sc_crawler.tables.Vendor.inventory_regions.
The functions should return an array of dict representing the related objects. The vendor's inventory method will pass the array to sc_crawler.insert.insert_items along with the table object.
Other functions and variables must be prefixed with an underscore to suggest those are internal tools.
Status and retirement checks
Set status on every Server and Database row. Possible values: ACTIVE, INACTIVE, PLANNED_FOR_RETIREMENT, RETIRED.
- Map the vendor's lifecycle API fields to these four states. Document the mapping in the
inventory_servers/inventory_databasesdocstring. - Use only explicit API or documentation signals for
PLANNED_FOR_RETIREMENTandRETIRED. Do not infer retirement from naming patterns like "previous generation" unless there is a confirmed mapping. - If multiple signals exist (e.g. per-region availability), pick the best:
ACTIVE>PLANNED_FOR_RETIREMENT>INACTIVE>RETIRED.
Progress bars
To create progress bars, you can use the Vendor's progress_tracker attribute with the below methods:
The start_task will register a task in the "Current tasks"
progress bar list with the provided name automatically prefixed by the vendor name, and the provided number of expected
steps. You should call advance_task after each step finished,
which will by default update the most recently created task's progress bar. If making updates in parallel, store the
TaskID returned by start_task and pass
to advance_task
and hide_task explicitly. Make sure to call hide_task when the
progress bar is not to be shown anymore. It's a good practice to log the number of fetched/synced objects afterwards
with logger.info. See the manual of VendorProgressTracker for more
details.
Basic example:
def inventory_zones(vendor):
zones = range(5)
vendor.progress_tracker.start_task(name="Searching zones", total=len(zones))
for zone in zones:
# do something
vendor.progress_tracker.advance_task()
vendor.progress_tracker.hide_task()
return zones
Template file for new vendors
def inventory_compliance_frameworks(vendor):
return map_compliance_frameworks_to_vendor(
vendor.vendor_id,
[
# "hipaa",
# "soc2t2",
# "iso27001",
],
)
def inventory_regions(vendor):
items = []
# for region in []:
# items.append(
# {
# "vendor_id": vendor.vendor_id,
# "region_id": "",
# "name": "",
# "api_reference": "",
# "display_name": "",
# "aliases": [],
# "country_id": "",
# "state": None,
# "city": None,
# "address_line": None,
# "zip_code": None,
# "lon": None,
# "lat": None,
# "founding_year": None,
# "green_energy": None,
# }
# )
return items
def inventory_zones(vendor):
items = []
# for zone in []:
# items.append({
# "vendor_id": vendor.vendor_id,
# "region_id": "",
# "zone_id": "",
# "name": "",
# "api_reference": "",
# "display_name": "",
# })
return items
def inventory_servers(vendor):
items = []
# for server in []:
# items.append(
# {
# "vendor_id": vendor.vendor_id,
# "server_id": ,
# "name": ,
# "api_reference": ,
# "display_name": ,
# "description": None,
# "family": None,
# "vcpus": ,
# "hypervisor": None,
# "cpu_allocation": CpuAllocation....,
# "cpu_cores": None,
# "cpu_speed": None,
# "cpu_architecture": CpuArchitecture....,
# "cpu_manufacturer": None,
# "cpu_family": None,
# "cpu_model": None,
# "cpu_l1d_cache": None,
# "cpu_l1d_cache_total": None,
# "cpu_l1i_cache": None,
# "cpu_l1i_cache_total": None,
# "cpu_l2_cache": None,
# "cpu_l2_cache_total": None,
# "cpu_l3_cache": None,
# "cpu_l3_cache_total": None,
# "cpu_flags": [],
# "cpus": [],
# "memory_amount": ,
# "memory_generation": None,
# "memory_speed": None,
# "memory_ecc": None,
# "gpu_count": 0,
# "gpu_memory_min": None,
# "gpu_memory_total": None,
# "gpu_manufacturer": None,
# "gpu_family": None,
# "gpu_model": None,
# "gpus": [],
# "storage_size": 0,
# "storage_type": None,
# "storages": [],
# "network_speed": None,
# "inbound_traffic": 0,
# "outbound_traffic": 0,
# "ipv4": 0,
# }
# )
return items
def inventory_server_prices(vendor):
items = []
# for server in []:
# items.append({
# "vendor_id": ,
# "region_id": ,
# "zone_id": ,
# "server_id": ,
# "operating_system": ,
# "allocation": Allocation....,
# "unit": PriceUnit.HOUR,
# "price": ,
# "price_upfront": 0,
# "price_tiered": [
# {"lower": 0, "upper": monthly_cap, "price": hourly_price},
# {"lower": monthly_cap + 1, "upper": "Infinity", "price": 0},
# ],
# "currency": "USD",
# })
return items
def inventory_server_prices_spot(vendor):
return []
def inventory_storages(vendor):
items = []
# for storage in []:
# items.append(
# {
# "storage_id": ,
# "vendor_id": vendor.vendor_id,
# "name": ,
# "description": None,
# "storage_type": StorageType....,
# "max_iops": None,
# "max_throughput": None,
# "min_size": None,
# "max_size": None,
# }
# )
return items
def inventory_storage_prices(vendor):
items = []
# for price in []:
# items.append(
# {
# "vendor_id": vendor.vendor_id,
# "region_id": ,
# "storage_id": ,
# "unit": PriceUnit.GB_MONTH,
# "price": ,
# "currency": "USD",
# }
# )
return items
def inventory_traffic_prices(vendor):
items = []
# for price in []:
# items.append(
# {
# "vendor_id": vendor.vendor_id,
# "region_id": ,
# "price": ,
# "price_tiered": [],
# "currency": "USD",
# "unit": PriceUnit.GB_MONTH,
# "direction": TrafficDirection....,
# }
# )
return items
def inventory_ipv4_prices(vendor):
items = []
# for price in []:
# items.append(
# {
# "vendor_id": vendor.vendor_id,
# "region_id": ,
# "price": ,
# "currency": "USD",
# "unit": PriceUnit.MONTH,
# }
# )
return items
def inventory_databases(vendor):
items = []
# for database in []:
# items.append(
# {
# "vendor_id": vendor.vendor_id,
# "database_id": ,
# "name": ,
# "api_reference": ,
# # Named API args for provisioning (e.g. Pulumi/Terraform).
# "api_reference_object": None,
# "display_name": ,
# "description": None,
# "family": None,
# "server_id": None,
# "vcpus": None,
# "memory_amount": None, # MiB
# "engine": DatabaseEngine.POSTGRESQL,
# "wire_protocol": DatabaseWireProtocol.POSTGRESQL,
# "engine_versions": ["16", "17"],
# "auto_upgrade_versions": None,
# # Ordered JSON lists (highest tier first); defaults to [NONE].
# "ha": [DatabaseHaLevel.NONE],
# "ha_strategy": [DatabaseHaStrategy.NONE],
# "max_read_replicas": None,
# "custom_config": None,
# "custom_extensions": None,
# "storage_size": None, # bundled GB
# "storage_extra_min": None, # GB
# "storage_extra_max": None, # GB
# "storage_extra_autosize": None,
# "disk_encryption": None,
# "scheduled_backups": None,
# "continuous_backups": None, # max PITR retention (days)
# "connection_pool": None,
# "system_monitoring": None,
# "database_monitoring": None,
# "autotuning_advice": None,
# "autotuning_apply": None,
# "sla": None, # e.g. 99.95
# # Supported security capabilities for this db instance.
# "security_features": [
# # DatabaseSecurityFeature.IP_FILTERING,
# # DatabaseSecurityFeature.PRIVATE_NETWORK,
# # DatabaseSecurityFeature.NETWORK_PEERING,
# # DatabaseSecurityFeature.IDENTITY_BASED_AUTH,
# # DatabaseSecurityFeature.CLIENT_CERT_AUTH,
# # DatabaseSecurityFeature.ENFORCED_TLS,
# # DatabaseSecurityFeature.CUSTOMER_MANAGED_KEYS,
# # DatabaseSecurityFeature.AUDIT_LOGGING,
# ],
# }
# )
return items
def inventory_database_prices(vendor):
items = []
# for price in []:
# items.append(
# {
# "vendor_id": vendor.vendor_id,
# "region_id": ,
# "database_id": ,
# "allocation": Allocation.ONDEMAND,
# # Scalar PK fields: one row per HA deployment (e.g. Single-AZ vs Multi-AZ).
# "ha": DatabaseHaLevel.NONE,
# "ha_strategy": DatabaseHaStrategy.NONE,
# "unit": PriceUnit.HOUR,
# "price": ,
# "price_upfront": 0,
# "price_tiered": [],
# "currency": "USD",
# }
# )
return items
def inventory_database_storages(vendor):
items = []
# for storage in []:
# items.append(
# {
# "vendor_id": vendor.vendor_id,
# "database_storage_id": ,
# "name": ,
# "description": None,
# "scope": DatabaseStorageScope.DATA,
# "min_size": None,
# "max_size": None,
# }
# )
return items
def inventory_database_storage_prices(vendor):
items = []
# for price in []:
# items.append(
# {
# "vendor_id": vendor.vendor_id,
# "region_id": ,
# "database_storage_id": ,
# "unit": PriceUnit.GB_MONTH,
# "price": ,
# "currency": "USD",
# }
# )
return items