ベクトルデータベース Weaviate (GPU使用) を Jetson Orin Nano NX 16G で動かす

Jul 24, 2024, 7:35 AM
Jetson で weaviate (text2vec-transformers,sum-transformers) を起動する必要がありましたが、Jetson オーナーなら誰でも悩む「一般的なNvidia製品との違い」があり、通常のDockerイメージではインストールできませんでした。なので、Jetson用にリビルドして起動しました。その時のログです。リビルドしたイメージも公開しておきます。

時間がない人のために

以下の内容で docker-compose.yaml を作成し起動してください。
※2024-07-24時点では、weaviate のバージョンは、1.26.1 です。最新のバージョンに書き換えても動くと思います。
services:
  weaviate:
    command:
    - --host
    - 0.0.0.0
    - --port
    - '8080'
    - --scheme
    - http
    image: cr.weaviate.io/semitechnologies/weaviate:1.26.1
    ports:
    - 8080:8080
    - 50051:50051
    volumes:
    - /home/jetson-01/weaviate/data:/var/lib/weaviate
    restart: on-failure:0
    environment:
      TRANSFORMERS_INFERENCE_API: 'http://t2v-transformers:8080'
      SUM_INFERENCE_API: 'http://sum-transformers:8080'
      QUERY_DEFAULTS_LIMIT: 25
      AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
      PERSISTENCE_DATA_PATH: '/var/lib/weaviate'
      DEFAULT_VECTORIZER_MODULE: 'text2vec-transformers'
      ENABLE_MODULES: 'text2vec-transformers,sum-transformers'
      CLUSTER_HOSTNAME: 'node1'
  t2v-transformers:
    image: ghcr.io/microaijp/jetson-transformers-inference:sentence-transformers-paraphrase-multilingual-mpnet-base-v2
    environment:
      ENABLE_CUDA: '1'
      NVIDIA_VISIBLE_DEVICES: 'all'
    deploy:
      resources:
        reservations:
          devices:
          - capabilities: 
            - 'gpu'
  sum-transformers:
    image: ghcr.io/microaijp/jetson-sum-transformers:facebook-bart-large-cnn-1.0.0
    environment:
      ENABLE_CUDA: '1'
      NVIDIA_VISIBLE_DEVICES: 'all'
    deploy:
      resources:
        reservations:
          devices:
          - capabilities: 
            - 'gpu'
volumes:
  weaviate_data:
※/home/jetson-01/weaviate/data は、↓の絶対パスに書き換えてください

データ保存用ディレクトリ作成

mkdir -o ~/weaviate/data

起動

docker compose up -d
で起動できます。


ベースとなるDocker Compose

ここからリビルドしたときのログです。以下の docker-compose.yaml をベースにしています。
services:
  weaviate:
    command:
    - --host
    - 0.0.0.0
    - --port
    - '8080'
    - --scheme
    - http
    image: cr.weaviate.io/semitechnologies/weaviate:1.26.1
    ports:
    - 8080:8080
    - 50051:50051
    volumes:
    - weaviate_data:/var/lib/weaviate
    restart: on-failure:0
    environment:
      TRANSFORMERS_INFERENCE_API: 'http://t2v-transformers:8080'
      SUM_INFERENCE_API: 'http://sum-transformers:8080'
      QUERY_DEFAULTS_LIMIT: 25
      AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
      PERSISTENCE_DATA_PATH: '/var/lib/weaviate'
      DEFAULT_VECTORIZER_MODULE: 'text2vec-transformers'
      ENABLE_MODULES: 'text2vec-transformers,sum-transformers'
      CLUSTER_HOSTNAME: 'node1'
  t2v-transformers:
    image: cr.weaviate.io/semitechnologies/transformers-inference:sentence-transformers-paraphrase-multilingual-mpnet-base-v2
    environment:
      ENABLE_CUDA: '1'
      NVIDIA_VISIBLE_DEVICES: 'all'
    deploy:
      resources:
        reservations:
          devices:
          - capabilities: 
            - 'gpu'
  sum-transformers:
    image: cr.weaviate.io/semitechnologies/sum-transformers:facebook-bart-large-cnn-1.0.0
    environment:
      ENABLE_CUDA: '1'
      NVIDIA_VISIBLE_DEVICES: 'all'
    deploy:
      resources:
        reservations:
          devices:
          - capabilities: 
            - 'gpu'
volumes:
  weaviate_data:

  • weaviate
  • t2v-transformers
  • sum-transformers
の３つを動かす必要があります。

weaviate 自体は GPU を使わないので、t2v-transformers と sum-transformers をリビルドしていきます。

ソースコードを抜き出す

ホスト側でソースコードのコピーを保存するディレクトリを作る

$ cd
$ mkdir weaviate
$ mkdir weaviate/t2v-transformers
$ mkdir weaviate/sum-transformers

一時的にコンテナを起動して、ソースコードをホストにコピーする

t2v-transformers

docker pull cr.weaviate.io/semitechnologies/transformers-inference:sentence-transformers-paraphrase-multilingual-mpnet-base-v2
docker run -it --rm --name temp_container cr.weaviate.io/semitechnologies/transformers-inference:sentence-transformers-paraphrase-multilingual-mpnet-base-v2 /bin/bash
ホスト側で
docker cp temp_container:/app ~/weaviate/t2v-transformers
コンテナ側
exit

sum-transformers

docker pull cr.weaviate.io/semitechnologies/sum-transformers:facebook-bart-large-cnn-1.0.0
docker run -it --rm --name temp_container cr.weaviate.io/semitechnologies/sum-transformers:facebook-bart-large-cnn-1.0.0 /bin/bash
ホスト側で
docker cp temp_container:/app ~/weaviate/sum-transformers
コンテナ側
exit

Dockerfile の作成&ビルド

t2v-transformers

cd ~/weaviate/t2v-transformers
nano requirements.txt
不足ライブラリを追加
transformers
fastapi
uvicorn
nltk
sentencepiece
sentence-transformers
optimum
onnxruntime
もともとある requirements.txt を削除しておく
rm ~/weaviate/t2v-transformers/app/requirements.txt

nano Dockerfile

FROM dustynv/l4t-pytorch:r36.2.0

WORKDIR /app

RUN apt-get update
RUN apt-get install -y libsndfile1
RUN pip install --upgrade pip setuptools

COPY requirements.txt .
RUN pip3 install -r requirements.txt

COPY app .

ENV MODEL_NAME=sentence-transformers/paraphrase-multilingual-mpnet-base-v2

ARG TARGETARCH
ARG MODEL_NAME
ARG ONNX_RUNTIME
ENV ONNX_CPU=${TARGETARCH}
RUN ./download.py

COPY . .

ENTRYPOINT ["/bin/sh", "-c"]
CMD ["uvicorn app:app --host 0.0.0.0 --port 8080"]

リビルト実行
sudo docker build -t microaijp/transformers-inference:sentence-transformers-paraphrase-multilingual-mpnet-base-v2 .

起動確認
sudo docker run --runtime=nvidia --rm \
  --name microai-t2v-transformers \
  --gpus all \
  -e ENABLE_CUDA=1 \
  -e NVIDIA_VISIBLE_DEVICES=all \
  -p 8080:8080 \
  microaijp/transformers-inference:sentence-transformers-paraphrase-multilingual-mpnet-base-v2

タグ付け＆私の ghcr に pushします。
echo "ghp_XXXXXXXXXXXXXXXXX" | sudo docker login ghcr.io -u microaijp --password-stdin
sudo docker tag microaijp/transformers-inference:sentence-transformers-paraphrase-multilingual-mpnet-base-v2 ghcr.io/microaijp/jetson-transformers-inference:sentence-transformers-paraphrase-multilingual-mpnet-base-v2
sudo docker push ghcr.io/microaijp/jetson-transformers-inference:sentence-transformers-paraphrase-multilingual-mpnet-base-v2

sum-transformers

cd ~/weaviate/sum-transformers
nano requirements.txt
不足ライブラリを追加
pydantic
transformers
fastapi
uvicorn
もともとある requirements.txt を削除しておく
rm ~/weaviate/sum-transformers/app/requirements.txt

nano Dockerfile

FROM dustynv/l4t-pytorch:r36.2.0

WORKDIR /app

COPY requirements.txt .
RUN pip3 install -r requirements.txt

ENV MODEL_NAME=facebook/bart-large-cnn

# COPY download.py .
COPY app .
RUN chmod +x ./download.py
RUN ./download.py

COPY . .

ENTRYPOINT ["/bin/sh", "-c"]
CMD ["uvicorn app:app --host 0.0.0.0 --port 8080"]

リビルド
sudo docker build -t microaijp/sum-transformers:facebook-bart-large-cnn-1.0.0 .

起動確認
sudo docker run --runtime=nvidia --rm \
  --name microai-sum-transformers \
  --gpus all \
  -e ENABLE_CUDA=1 \
  -e NVIDIA_VISIBLE_DEVICES=all \
  -p 8080:8080 \
  microaijp/sum-transformers:facebook-bart-large-cnn-1.0.0

タグ付け＆私の ghcr に pushします。
echo "ghp_XXXXXXXXXXXXXXXXX" | sudo docker login ghcr.io -u microaijp --password-stdin
sudo docker tag microaijp/sum-transformers:facebook-bart-large-cnn-1.0.0 ghcr.io/microaijp/jetson-sum-transformers:facebook-bart-large-cnn-1.0.0
sudo docker push ghcr.io/microaijp/jetson-sum-transformers:facebook-bart-large-cnn-1.0.0

動作確認
sudo docker run --runtime=nvidia -d \
  --name sum-transformers \
  --gpus all \
  -e ENABLE_CUDA=1 \
  -e NVIDIA_VISIBLE_DEVICES=all \
  -p 8003:8080 \
  cr.weaviate.io/semitechnologies/sum-transformers:facebook-bart-large-cnn-1.0.0

Docker Compose 作成

データ保存用ディレクトリ作成

mkdir -o ~/weaviate/data

docker-compose.yaml 作成

cd ~/weaviate
nano docker-compose.yaml
※/home/jetson-01/weaviate/data は、↑の絶対パスに書き換えてください
services:
  weaviate:
    command:
    - --host
    - 0.0.0.0
    - --port
    - '8080'
    - --scheme
    - http
    image: cr.weaviate.io/semitechnologies/weaviate:1.26.1
    ports:
    - 8080:8080
    - 50051:50051
    volumes:
    - /home/jetson-01/weaviate/data:/var/lib/weaviate
    restart: on-failure:0
    environment:
      TRANSFORMERS_INFERENCE_API: 'http://t2v-transformers:8080'
      SUM_INFERENCE_API: 'http://sum-transformers:8080'
      QUERY_DEFAULTS_LIMIT: 25
      AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
      PERSISTENCE_DATA_PATH: '/var/lib/weaviate'
      DEFAULT_VECTORIZER_MODULE: 'text2vec-transformers'
      ENABLE_MODULES: 'text2vec-transformers,sum-transformers'
      CLUSTER_HOSTNAME: 'node1'
  t2v-transformers:
    image: ghcr.io/microaijp/jetson-transformers-inference:sentence-transformers-paraphrase-multilingual-mpnet-base-v2
    environment:
      ENABLE_CUDA: '1'
      NVIDIA_VISIBLE_DEVICES: 'all'
    deploy:
      resources:
        reservations:
          devices:
          - capabilities: 
            - 'gpu'
  sum-transformers:
    image: ghcr.io/microaijp/jetson-sum-transformers:facebook-bart-large-cnn-1.0.0
    environment:
      ENABLE_CUDA: '1'
      NVIDIA_VISIBLE_DEVICES: 'all'
    deploy:
      resources:
        reservations:
          devices:
          - capabilities: 
            - 'gpu'
volumes:
  weaviate_data:

起動

docker compose up -d

起動確認

docker ps
CONTAINER ID   IMAGE                                                                                                         COMMAND                  CREATED         STATUS         PORTS                                                                                      NAMES
fe2551847f2d   ghcr.io/microaijp/jetson-sum-transformers:facebook-bart-large-cnn-1.0.0                                       "/bin/sh -c 'uvicorn…"   4 seconds ago   Up 2 seconds                                                                                              weaviate-sum-transformers-1
b389211d8818   ghcr.io/microaijp/jetson-transformers-inference:sentence-transformers-paraphrase-multilingual-mpnet-base-v2   "/bin/sh -c 'uvicorn…"   4 seconds ago   Up 2 seconds                                                                                              weaviate-t2v-transformers-1
491adae1b4a7   cr.weaviate.io/semitechnologies/weaviate:1.25.4                                                               "/bin/weaviate --hos…"   4 seconds ago   Up 2 seconds   0.0.0.0:8080->8080/tcp, :::8080->8080/tcp, 0.0.0.0:50051->50051/tcp, :::50051->50051/tcp   weaviate-weaviate-1

ログ確認

docker logs weaviate-weaviate-1
{"action":"startup","default_vectorizer_module":"text2vec-transformers","level":"info","msg":"the default vectorizer modules is set to \"text2vec-transformers\", as a result all new schema classes without an explicit vectorizer setting, will use this vectorizer","time":"2024-07-23T21:38:14Z"}
{"action":"startup","auto_schema_enabled":true,"level":"info","msg":"auto schema enabled setting is set to \"true\"","time":"2024-07-23T21:38:14Z"}
{"level":"info","msg":"No resource limits set, weaviate will use all available memory and CPU. To limit resources, set LIMIT_RESOURCES=true","time":"2024-07-23T21:38:14Z"}
{"level":"info","msg":"open cluster service","servers":{"node1":8300},"time":"2024-07-23T21:38:14Z"}
{"address":"172.19.0.2:8301","level":"info","msg":"starting cloud rpc server ...","time":"2024-07-23T21:38:14Z"}
{"level":"info","msg":"starting raft sub-system ...","time":"2024-07-23T21:38:14Z"}
{"address":"172.19.0.2:8300","level":"info","msg":"tcp transport","tcpMaxPool":3,"tcpTimeout":10000000000,"time":"2024-07-23T21:38:14Z"}
{"level":"info","msg":"loading local db","time":"2024-07-23T21:38:14Z"}
{"level":"info","msg":"database has been successfully loaded","n":0,"time":"2024-07-23T21:38:14Z"}
{"level":"info","metadata_only_voters":false,"msg":"construct a new raft node","name":"node1","time":"2024-07-23T21:38:14Z"}
{"action":"raft","index":813,"level":"info","msg":"raft initial configuration","servers":"[[{Suffrage:Voter ID:node1 Address:172.18.0.2:8300}]]","time":"2024-07-23T21:38:14Z"}
{"last_snapshot_index":0,"last_store_applied_index":0,"last_store_log_applied_index":0,"level":"info","msg":"raft node constructed","raft_applied_index":0,"raft_last_index":862,"time":"2024-07-23T21:38:14Z"}
{"action":"raft","follower":{},"leader-address":"","leader-id":"","level":"info","msg":"raft entering follower state","time":"2024-07-23T21:38:14Z"}
{"action":"bootstrap","error":"could not join a cluster from [172.19.0.2:8300]","level":"warning","msg":"failed to join cluster, will notify next if voter","servers":["172.19.0.2:8300"],"time":"2024-07-23T21:38:16Z","voter":true}
{"action":"bootstrap","candidates":[{"Suffrage":0,"ID":"node1","Address":"172.19.0.2:8300"}],"level":"info","msg":"starting cluster bootstrapping","time":"2024-07-23T21:38:16Z"}
{"action":"bootstrap","error":"bootstrap only works on new clusters","level":"error","msg":"could not bootstrapping cluster","time":"2024-07-23T21:38:16Z"}
{"action":"bootstrap","level":"info","msg":"notified peers this node is ready to join as voter","servers":["172.19.0.2:8300"],"time":"2024-07-23T21:38:16Z"}
{"action":"raft","last-leader-addr":"","last-leader-id":"","level":"warning","msg":"raft heartbeat timeout reached, starting election","time":"2024-07-23T21:38:16Z"}
{"action":"raft","level":"info","msg":"raft entering candidate state","node":{},"term":862,"time":"2024-07-23T21:38:16Z"}
{"action":"raft","level":"info","msg":"raft election won","tally":1,"term":862,"time":"2024-07-23T21:38:16Z"}
{"action":"raft","leader":{},"level":"info","msg":"raft entering leader state","time":"2024-07-23T21:38:16Z"}
{"address":"172.19.0.2:8300","level":"info","msg":"current Leader","time":"2024-07-23T21:38:16Z"}
{"level":"info","msg":"starting migration from old schema","time":"2024-07-23T21:38:16Z"}
{"level":"info","msg":"legacy schema is empty, nothing to migrate","time":"2024-07-23T21:38:16Z"}
{"level":"info","msg":"migration from the old schema has been successfully completed","time":"2024-07-23T21:38:16Z"}
{"action":"sum_remote_wait_for_startup","error":"send check ready request: Get \"http://sum-transformers:8080/.well-known/ready\": dial tcp 172.19.0.4:8080: connect: connection refused","level":"warning","msg":"sum remote service not ready","time":"2024-07-23T21:38:17Z"}
{"action":"bootstrap","level":"info","msg":"node reporting ready, node has probably recovered cluster from raft config. Exiting bootstrap process","time":"2024-07-23T21:38:18Z"}
{"action":"sum_remote_wait_for_startup","error":"send check ready request: Get \"http://sum-transformers:8080/.well-known/ready\": dial tcp 172.19.0.4:8080: connect: connection refused","level":"warning","msg":"sum remote service not ready","time":"2024-07-23T21:38:18Z"}
{"action":"sum_remote_wait_for_startup","error":"send check ready request: Get \"http://sum-transformers:8080/.well-known/ready\": dial tcp 172.19.0.4:8080: connect: connection refused","level":"warning","msg":"sum remote service not ready","time":"2024-07-23T21:38:19Z"}
{"action":"sum_remote_wait_for_startup","error":"send check ready request: Get \"http://sum-transformers:8080/.well-known/ready\": dial tcp 172.19.0.4:8080: connect: connection refused","level":"warning","msg":"sum remote service not ready","time":"2024-07-23T21:38:20Z"}
{"action":"sum_remote_wait_for_startup","error":"send check ready request: Get \"http://sum-transformers:8080/.well-known/ready\": dial tcp 172.19.0.4:8080: connect: connection refused","level":"warning","msg":"sum remote service not ready","time":"2024-07-23T21:38:21Z"}
{"action":"sum_remote_wait_for_startup","error":"send check ready request: Get \"http://sum-transformers:8080/.well-known/ready\": dial tcp 172.19.0.4:8080: connect: connection refused","level":"warning","msg":"sum remote service not ready","time":"2024-07-23T21:38:22Z"}
{"action":"sum_remote_wait_for_startup","error":"send check ready request: Get \"http://sum-transformers:8080/.well-known/ready\": dial tcp 172.19.0.4:8080: connect: connection refused","level":"warning","msg":"sum remote service not ready","time":"2024-07-23T21:38:23Z"}
{"action":"transformer_remote_wait_for_startup","error":"send check ready request: Get \"http://t2v-transformers:8080/.well-known/ready\": dial tcp 172.19.0.3:8080: connect: connection refused","level":"warning","msg":"transformer remote inference service not ready","time":"2024-07-23T21:38:25Z"}
{"action":"transformer_remote_wait_for_startup","error":"send check ready request: Get \"http://t2v-transformers:8080/.well-known/ready\": dial tcp 172.19.0.3:8080: connect: connection refused","level":"warning","msg":"transformer remote inference service not ready","time":"2024-07-23T21:38:26Z"}
{"action":"grpc_startup","level":"info","msg":"grpc server listening at [::]:50051","time":"2024-07-23T21:38:27Z"}
{"action":"restapi_management","level":"info","msg":"Serving weaviate at http://[::]:8080","time":"2024-07-23T21:38:27Z"}
{"action":"telemetry_push","level":"info","msg":"telemetry started","payload":"\u0026{MachineID:43b74468-1464-4d4e-b84c-39a6f5375127 Type:INIT Version:1.25.4 NumObjects:0 OS:linux Arch:arm64 UsedModules:[]}","time":"2024-07-23T21:38:28Z"}
起動に少し時間がかかるため、数回 error ログがでていますが、起動するとでなくくなります。

docker logs weaviate-t2v-transformers-1
INFO:     Started server process [19]
INFO:     Waiting for application startup.
INFO:     CUDA_PER_PROCESS_MEMORY_FRACTION set to 1.0
INFO:     CUDA_CORE set to cuda:0
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)
INFO:     172.19.0.2:44372 - "GET /.well-known/ready HTTP/1.1" 204 No Content

docker logs weaviate-sum-transformers-1
INFO:     Started server process [18]
INFO:     Waiting for application startup.
INFO:     CUDA_CORE set to cuda:0
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)
INFO:     172.19.0.2:55678 - "GET /.well-known/ready HTTP/1.1" 204 No Content
上記は両方とも、204 で問題ありません。

ブラウザでアクセスしてみる

http://10.0.0.21:8080/v1



データを投入してみる

curl -X POST http://10.0.0.21:8080/v1/objects \
-H 'Content-Type: application/json' \
-d '{
  "class": "Article",
  "properties": {
    "title": "Persistent Data Test",
    "content": "This data should persist across container restarts."
  }
}'

{"class":"Article","creationTimeUnix":1721771820813,"id":"694bf6a9-4843-451e-bf04-77d5aa446cd7","lastUpdateTimeUnix":1721771820813,"properties":{"content":"This data should persist across contain...


データの一覧を取得してみる

curl -X POST http://10.0.0.21:8080/v1/graphql \
-H 'Content-Type: application/json' \
-d '{"query":"{ Get { Article { title content _additional { id } } } }"}' | jq

{
  "data": {
    "Get": {
      "Article": [
        {
          "_addit
:

ベクトル検索してみる

curl -X POST http://10.0.0.21:8080/v1/graphql \
-H 'Content-Type: application/json' \
-d '{"query":"{ Get { Article(nearText: {concepts: [\"Persistent Data Test\"]}) { title content _additional { id } } } }"}' | jq

{
  "data": {
    "Get": {
      "Article": [
        {
          "_additional": {
            "id": "b27668f6-1f7a-403a-98ef-5d75917ce76c"
          },
          "content": "This data should persist across container restarts.",
          "title": "Persisten
:
すべてうまく動いてくれました🍻
※jtop で GPU の動きをみると、一瞬すぎてなかなかGPUが動いているところを見れませんが、何度も行っているとGPUが動くところが確認できます。
※データはコンテナ内に保存されるため、再起動しても消えません。


以下、個人用のメモです

sudo docker run --runtime nvidia -it --rm --network=host -v /home/jetson-01/weaviate/sum-transformers/app:/app dustynv/l4t-pytorch:r36.2.0
pydantic==1.8.2
transformers==4.20.1
torch==1.12.0
fastapi==0.78.0
uvicorn==0.18.2

pip install pydantic transformers fastapi uvicorn
export MODEL_NAME=facebook/bart-large-cnn
chmod +x ./download.py
./download.py

export ENABLE_CUDA=1
export NVIDIA_VISIBLE_DEVICES=all
uvicorn app:app --host 0.0.0.0 --port 8080

Kubernetes に Redis をインストールする
NASを持っていないので、GlusterFSで分散ファイルシステムを構築してNFSとして運用するNASを持っていないので、GlusterFSで分散ファイルシステムを構築してNFSとして運用する