Snowflake Container Service Deployment
Snowflake Container Service Deployment
Pre-requisites:
- An active compute pool in Snowflake
- BigGeo image and
proxy-server
image in the snowflake registry. - an encrypted named stage on snowflake that the compute pool has access to.
We will need to deploy the docker_compose.yaml
to a named stage in snowflake in order to deploy BigGeo as a container named bg-search
alongside a proxy-server
.
Here, we require a proxy-server
because UDFs cannot direct link to BG search's gRPC API. So, we use the proxy-server
as an intermediary that transforms BG search's gRPC API into a rest API that can be utilized by the snowflake UDFs. For instance, if a collection was to be created in BG search, we would create a UDF that invokes the 'create collection' REST API of the proxy-server
. Subsequently, the proxy-server
would make a call to BG search’s 'create collection' gRPC API.
spec:
containers:
- name: proxy-server
image: <snowflake_registry>/<your docker image>:latest
env:
SERVER_PORT: 5001
BG_SEARCH_PORT: 50051
BGSEARCH_PSWD: <you passworkd>
ENABLE_LOGGING: false
readinessProbe:
port: 5001
path: /healthcheck
- name: bg-search
image: <snowflake_registry>/bg_server:<bg-search-version-here>
env:
BGSEARCH_PSWD: <you passworkd>
BGSEARCH_AUTH_ENABLED: false
BGSEARCH_PATH_PREFIX: /storage/
volumeMounts: # optional list
- name: vol
mountPath: /storage
endpoint:
- name: bgpoint
port: 5001
public: true
volumes: # optional volume list
- name: vol
source: "@persistent_storage"
---
contentSecurityPolicy: frame-ancestors https://*.snowflakecomputing.app
We will need to ensure that we have the right permissions to be able to deploy this service in Snowflake compute pools, please refer to Snowflakes tutorial to make sure you are familiar with granting the required permissions.
Deploy the service in the active compute pool using the following command
CREATE SERVICE BigGeo
IN COMPUTE POOL <ComputePoolName>
FROM @STAGE_NAME
SPECIFICATION_FILE='docker_compose.yaml'
MIN_INSTANCES=1
MAX_INSTANCES=1;
We can then create external functions used to interact with BG search, an example external function that would create a collection is presented below:
CREATE FUNCTION bg_search_createCollection(collection varchar)
RETURNS variant
SERVICE=BigGeo
ENDPOINT=bgpoint
AS '/createCollection';
SELECT bg_search_createCollection('hello_world');
Note: External functions will need to be created for every endpoint offered in the proxy-server
identified by viewing the logs of the proxy-server
service running in the active compute pool.
- The arguments positions and names in the external function must align with the arguments position provided in the proto file mentioned here.
Updated 3 months ago