Snowflake Container Service Deployment

Snowflake Container Service Deployment

Pre-requisites:

1. An active compute pool in Snowflake
2. BigGeo image and proxy-server image in the snowflake registry.
3. an encrypted named stage on snowflake that the compute pool has access to.

We will need to deploy the docker_compose.yaml to a named stage in snowflake in order to deploy BigGeo as a container named bg-search alongside a proxy-server.

Here, we require a proxy-server because UDFs cannot direct link to BG search's gRPC API. So, we use the proxy-server as an intermediary that transforms BG search's gRPC API into a rest API that can be utilized by the snowflake UDFs. For instance, if a collection was to be created in BG search, we would create a UDF that invokes the 'create collection' REST API of the proxy-server. Subsequently, the proxy-server would make a call to BG search’s 'create collection' gRPC API.

spec:
  containers:
  - name: proxy-server
    image: <snowflake_registry>/<your docker image>:latest
    env:
      SERVER_PORT: 5001
      BG_SEARCH_PORT: 50051
      BGSEARCH_PSWD: <you passworkd>
      ENABLE_LOGGING: false
    readinessProbe:
      port: 5001
      path: /healthcheck
  - name: bg-search
    image: <snowflake_registry>/bg_server:<bg-search-version-here>
    env:
      BGSEARCH_PSWD: <you passworkd>
      BGSEARCH_AUTH_ENABLED: false
      BGSEARCH_PATH_PREFIX: /storage/
    volumeMounts:                       # optional list
      - name: vol
        mountPath: /storage
  endpoint:
  - name: bgpoint
    port: 5001
    public: true
  volumes:                               # optional volume list
    - name: vol
      source: "@persistent_storage"
---
contentSecurityPolicy: frame-ancestors https://*.snowflakecomputing.app

We will need to ensure that we have the right permissions to be able to deploy this service in Snowflake compute pools, please refer to Snowflakes tutorial to make sure you are familiar with granting the required permissions.

Deploy the service in the active compute pool using the following command

CREATE SERVICE BigGeo
  IN COMPUTE POOL <ComputePoolName>
  FROM @STAGE_NAME
  SPECIFICATION_FILE='docker_compose.yaml'
  MIN_INSTANCES=1
  MAX_INSTANCES=1;

We can then create external functions used to interact with BG search, an example external function that would create a collection is presented below:

CREATE FUNCTION bg_search_createCollection(collection varchar)
    RETURNS variant
    SERVICE=BigGeo
    ENDPOINT=bgpoint
    AS '/createCollection';

SELECT bg_search_createCollection('hello_world');

Note: External functions will need to be created for every endpoint offered in the proxy-server identified by viewing the logs of the proxy-server service running in the active compute pool.

• The arguments positions and names in the external function must align with the arguments position provided in the proto file mentioned here.