scystream.sdk package

Subpackages

• scystream.sdk.config package
  • SDKConfig
    • SDKConfig.__new__()
    • SDKConfig.app_name
    • SDKConfig.cb_spark_master
    • SDKConfig.get_cb_spark_master()
    • SDKConfig.set_cb_spark_master()
  • generate_config_from_compute_block()
  • get_compute_block()
  • load_config()
  • validate_config_with_code()
  • Submodules
  • scystream.sdk.config.compute_block_utils module
    • get_compute_block()
  • scystream.sdk.config.config_loader module
    • SDKConfig
      • SDKConfig.__new__()
      • SDKConfig.app_name
      • SDKConfig.cb_spark_master
      • SDKConfig.get_cb_spark_master()
      • SDKConfig.set_cb_spark_master()
    • generate_config_from_compute_block()
    • load_config()
    • validate_config_with_code()
  • scystream.sdk.config.entrypoints module
    • TEST_reset_registered_functions()
    • get_registered_functions()
    • register_entrypoint()
  • scystream.sdk.config.models module
    • ComputeBlock
      • ComputeBlock.author
      • ComputeBlock.check_entrypoints()
      • ComputeBlock.description
      • ComputeBlock.docker_image
      • ComputeBlock.entrypoints
      • ComputeBlock.model_config
      • ComputeBlock.name
    • Entrypoint
      • Entrypoint.description
      • Entrypoint.envs
      • Entrypoint.inputs
      • Entrypoint.model_config
      • Entrypoint.outputs
    • InputOutputModel
      • InputOutputModel.config
      • InputOutputModel.description
      • InputOutputModel.model_config
      • InputOutputModel.type
• scystream.sdk.database_handling package
  • Submodules
  • scystream.sdk.database_handling.postgres_manager module
    • BaseDatabaseOperations
      • BaseDatabaseOperations.MAX_TABLE_NAME_LENGTH
      • BaseDatabaseOperations.__init__()
      • BaseDatabaseOperations.read()
      • BaseDatabaseOperations.write()
    • PandasDatabaseOperations
      • PandasDatabaseOperations.__init__()
      • PandasDatabaseOperations.read()
      • PandasDatabaseOperations.write()
    • SparkDatabaseOperations
      • SparkDatabaseOperations.__init__()
      • SparkDatabaseOperations.read()
      • SparkDatabaseOperations.write()
• scystream.sdk.env package
  • Submodules
  • scystream.sdk.env.settings module
    • CustomSettings
      • CustomSettings.model_config
    • DatabaseSettings
      • DatabaseSettings.DB_DSN
      • DatabaseSettings.DB_SCHEMA
      • DatabaseSettings.DB_TABLE
      • DatabaseSettings.model_config
    • EnvSettings
      • EnvSettings.__init__()
      • EnvSettings.from_env()
      • EnvSettings.get_settings()
      • EnvSettings.model_config
    • FileSettings
      • FileSettings.BUCKET_NAME
      • FileSettings.FILE_EXT
      • FileSettings.FILE_NAME
      • FileSettings.FILE_PATH
      • FileSettings.S3_ACCESS_KEY
      • FileSettings.S3_HOST
      • FileSettings.S3_PORT
      • FileSettings.S3_SECRET_KEY
      • FileSettings.model_config
    • InputSettings
      • InputSettings.model_config
    • OutputSettings
      • OutputSettings.model_config
• scystream.sdk.file_handling package
  • Submodules
  • scystream.sdk.file_handling.s3_manager module
    • S3Config
      • S3Config.S3_ACCESS_KEY
      • S3Config.S3_HOST
      • S3Config.S3_PORT
      • S3Config.S3_SECRET_KEY
      • S3Config.model_config
    • S3Operations
      • S3Operations.__init__()
      • S3Operations.download()
      • S3Operations.download_file()
      • S3Operations.download_file_from_settings()
      • S3Operations.upload()
      • S3Operations.upload_file()
      • S3Operations.upload_file_from_settings()

Submodules

scystream.sdk.core module

scystream.sdk.core.entrypoint(settings_class: Type[EnvSettings] | None = None)

Decorator to mark a function as an entrypoint. It also loads and injects the settings of the entrypoint.

scystream.sdk.scheduler module

class scystream.sdk.scheduler.Scheduler

Bases: object

static execute_function(name, *args, **kwargs)

Validate the in code defined entrypoints with the settings defined in the cfg file

static list_entrypoints()

List all registered entrypoint functions.

scystream.sdk.spark_manager module

class scystream.sdk.spark_manager.SparkManager

Bases: object

__init__()

setup_pg(dsn: str) → SparkDatabaseOperations

stop_session()