Облачное соединение SQL для Kubernetes с использованием прокси

Question

Облачное соединение SQL для Kubernetes с использованием прокси

В настоящее время я работаю в Spring Boot Pod в Кубернетесе. В контейнере есть сторонняя машина для облачного прокси-сервера SQL.

Ниже приведена моя весенняя настройка Boot application.properties:

server.port=8081
spring.datasource.driver-class-name=com.mysql.cj.jdbc.Driver
spring.jpa.hibernate.ddl-auto=create-drop
spring.datasource.continue-on-error=true
spring.datasource.url=jdbc:mysql://localhost:3306/<database_name>
spring.datasource.username=<user_name>
spring.datasource.password=<password>

Ниже приведен мой pom.xml экстракт с плагинами и зависимостями:

<properties>
    <java.version>1.8</java.version>
</properties>

<dependencies>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-actuator</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-data-jpa</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-web</artifactId>
    </dependency>

    <dependency>
        <groupId>mysql</groupId>
        <artifactId>mysql-connector-java</artifactId>
    </dependency>

    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-devtools</artifactId>
        <scope>runtime</scope>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-test</artifactId>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>com.jayway.jsonpath</groupId>
        <artifactId>json-path</artifactId>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-hateoas</artifactId>
    </dependency>

    <dependency>
        <groupId>com.h2database</groupId>
        <artifactId>h2</artifactId>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>ca.performance.common</groupId>
        <artifactId>common-http</artifactId>
        <version>1.1.1</version>
    </dependency>
    <dependency>
        <groupId>com.google.cloud.sql</groupId>
        <artifactId>mysql-socket-factory</artifactId>
        <version>1.0.10</version>
    </dependency>
    <dependency>
        <groupId>org.springframework</groupId>
        <artifactId>spring-jdbc</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-gcp-starter-sql-mysql</artifactId>
        <version>1.1.0.RELEASE</version>
    </dependency>
</dependencies>

<build>
    <plugins>
        <plugin>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-maven-plugin</artifactId>
        </plugin>
    </plugins>
</build>

И это мой файл deploy.yaml:

apiVersion: v1
kind: Service
metadata:
  name: app-dummy-name
spec:
  selector:
    app: app-dummy-name
  ports:
  - port: 81
    name: http-app-dummy-name
    targetPort: http-api
  type: LoadBalancer
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: app-dummy-name
spec:
  replicas: 1
  selector:
    matchLabels:
      app: app-dummy-name
  template:
    metadata:
      labels:
        app: app-dummy-name
    spec:
      containers:
      - name: app-dummy-name
        image: <image url>
        ports:
        - containerPort: 8081
          name: http-api
        env:
        - name: DB_HOST
          value: 127.0.0.1:3306
        - name: DB_USER
          valueFrom:
            secretKeyRef:
              name: cloudsql-db-credentials
              key: username
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: cloudsql-db-credentials
              key: password
      - name: cloudsql-proxy
        image: gcr.io/cloudsql-docker/gce-proxy:1.11
        command: ["/cloud_sql_proxy",
                    "-instances=<INSTANCE_CONNECTION_NAME>=:3306",
                    "-credential_file=/secrets/cloudsql/credentials.json"]
        securityContext:
          runAsUser: 2  # non-root user
          allowPrivilegeEscalation: false
        volumeMounts:
          - name: cloudsql-instance-credentials
            mountPath: /secrets/cloudsql
            readOnly: true
      volumes:
        - name: cloudsql-instance-credentials
          secret:
            secretName: cloudsql-instance-credentials

Я следовал инструкциям по этой ссылке, поэтому создал секреты и учетную запись сервиса. Тем не менее, я постоянно получаю ошибки отказа соединения при развертывании предыдущего файла yaml в Kubernetes после создания секретов:

org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta-data; 
nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection; 
nested exception is com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure. 
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.

Я даже тестировал загрузочное приложение Spring локально, используя прокси и ту же конфигурацию application.properties, и оно работало нормально.

3

mysql spring spring-boot kubernetes cloud-sql-proxy

Источник

user9546332 15 фев '19 в 16:23

2 ответа

Другие вопросы по тегам mysql spring spring-boot kubernetes cloud-sql-proxy

user5980143 16 фев '19 в 19:59 2019-02-16 19:59 · Answer 1 · 2019-02-16 19:59

Я добавляю свой yaml развертывания, который работал для меня, проверьте, поможет ли добавление следующего:

по объемам:

  volumes:
  - name: cloudsql
    emptyDir:

в связи: `--dir=/cloudsql`

  - name: cloudsql-proxy
    image: gcr.io/cloudsql-docker/gce-proxy:1.11
    command: ["/cloud_sql_proxy", "--dir=/cloudsql",
        "-instances=<INSTANCE_CONNECTION_NAME=tcp:5432>",
        "-credential_file=/secrets/cloudsql/credentials.json"]

также убедитесь, что вы включили API администрирования Cloud SQL

вот мое полное развертывание yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: app-dummy-name
spec:
  replicas: 1
  revisionHistoryLimit: 1
  strategy:
      type: RollingUpdate
  template:
    metadata:
      labels:
        app: app-dummy-name
        tier: backend
    spec:
      securityContext:
        runAsUser: 0
        runAsNonRoot: false
      containers:
      - name: app-dummy-name
        image: <image url>
        ports:
        - containerPort: 80
        env:
        - name: DB_HOST
          value: localhost
        - name: DB_USER
          valueFrom:
            secretKeyRef:
              name: cloudsql-db-credentials
              key: username
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: cloudsql-db-credentials
              key: password
      # proxy_container
      - name: cloudsql-proxy
        image: gcr.io/cloudsql-docker/gce-proxy:1.11
        command: ["/cloud_sql_proxy", "--dir=/cloudsql",
          "-instances=my-project-id:us-central1:postgres-instance-name=tcp:5432",
          "-credential_file=/secrets/cloudsql/credentials.json"]
        volumeMounts:
          - name: cloudsql-instance-credentials
            mountPath: /secrets/cloudsql
            readOnly: true
          - name: cloudsql
            mountPath: /cloudsql
      # volumes
      volumes:
      - name: cloudsql-instance-credentials
        secret:
          secretName: cloudsql-instance-credentials
      - name: cloudsql
        emptyDir:

вот мой сценарий перед дельпой:

#!/bin/bash
# https://cloud.google.com/sql/docs/mysql/connect-kubernetes-engine
# 1. Go to the Cloud SQL Service accounts page of the Google Cloud Platform Console.
# GO TO THE SERVICE ACCOUNTS PAGE
# 2.  If needed, select the project that contains your Cloud SQL instance.
# 3. Click Create service account.
# 4. In the Create service account dialog, provide a descriptive name for the service account.
# 5. For Role, select Cloud SQL > Cloud SQL Client.
# Alternatively, you can use the primitive Editor role by selecting Project > Editor, but the Editor role includes permissions across Google Cloud Platform.
#
# 6. If you do not see these roles, your Google Cloud Platform user might not have the resourcemanager.projects.setIamPolicy permission. You can check your permissions by going to the IAM page in the Google Cloud Platform Console and searching for your user id.
# Change the Service account ID to a unique value that you will recognize so you can easily find this service account later if needed.
# 7. Click Furnish a new private key.
# 8. The default key type is JSON, which is the correct value to use.
# 9. Click Create.
# 10. enable Cloud SQL Administration API [here](https://console.developers.google.com/apis/api/sqladmin.googleapis.com/overview)
# make sure to choose your project


echo "create cloudsql secret"
kubectl create secret generic cloudsql-instance-credentials \
   --from-file=credentials.json=postgres-sql-credential.json

echo "create cloudsql user and password"
kubectl create secret generic cloudsql-db-credentials \
   --from-literal=username=postgres --from-literal=password=123456789

Файл postgres-sql-credential.json:

{
  "type": "service_account",
  "project_id": "my-project",
  "private_key_id": "1234567890",
  "private_key": "-----BEGIN PRIVATE KEY-----\n123445556\n123445\n-----END PRIVATE KEY-----\n",
  "client_email": "postgres-sql@my-project.iam.gserviceaccount.com",
  "client_id": "1234567890",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/postgres-sq%my-project.iam.gserviceaccount.com"
}

user5877576 09 ноя '21 в 16:25 2021-11-09 16:25 · Answer 2 · 2021-11-09 16:25

Исходя из этого обсуждения, я разработал yaml, как показано ниже. Он предназначен для подключения к cloud-sql через pgbouncer.

      apiVersion: apps/v1
kind: Deployment
metadata:
  name: pgproxy
  namespace: var_namespace_k8s
spec:
  replicas: 1
  selector:
    matchLabels:
      app: pgproxy
  revisionHistoryLimit: 1
  strategy:
      type: RollingUpdate
  template:
    metadata:
      labels:
        app: pgproxy
        tier: backend
    spec:
      securityContext:
        runAsUser: 0
        runAsNonRoot: false
      containers:
      - name: pgproxy
        image: var_image_pgproxy_k8s
        ports:
        - containerPort: 6432
        volumeMounts:
          - name: pgbouncer-init-volume
            mountPath: /home/pgbouncer/pgbouncer.ini
            subPath: pgbouncer.ini
          - name: pgbouncer-userlist-volume
            mountPath: /home/pgbouncer/userlist.txt
            subPath: userlist.txt
      # proxy_container
      - name: cloudsql-proxy
        image: var_image_cloudsqlproxy_k8s
        command: ["/cloud_sql_proxy", "--dir=/cloudsql",
          "-instances=gcp:us-east:databse=tcp:0.0.0.0:3306",
          "-credential_file=/secrets/cloudsql/cloudsql-service-acnt-key.json"]
        volumeMounts:
          - name: sqlauthproxy-svcaccount-volume
            mountPath: /secrets/cloudsql/cloudsql-service-acnt-key.json
            subPath: cloudsql-service-acnt-key.json
            readOnly: true
          - name: cloudsql
            mountPath: /cloudsql
      # volumes
      volumes:
      - name: pgbouncer-init-volume
        secret:
          secretName: pgbouncer-init
          items:
          - key: pgbouncer.ini
            path: pgbouncer.ini
      - name: pgbouncer-userlist-volume
        secret:
          secretName: pgbouncer-userlist
          items:
          - key: userlist.txt
            path: userlist.txt
      - name: sqlauthproxy-svcaccount-volume
        secret:
          secretName: sqlauthproxy-svcaccount
          items:
          - key: cloudsql-service-acnt-key.json
            path: cloudsql-service-acnt-key.json
      - name: cloudsql
        emptyDir:

---
apiVersion: v1
kind: Service
metadata:
  name: pgproxy
  namespace: var_namespace_k8s
  annotations:
    cloud.google.com/load-balancer-type: "Internal"
spec:
  type: LoadBalancer
  selector:
    app: pgproxy
  ports:
    - port: 6432
      targetPort: 6432

Результат:

      NAME                      READY   STATUS             RESTARTS   AGE
pgproxy-6b84bdb84-7g2l4   1/2     CrashLoopBackOff   7          14m

Выход kubectl logs pod/pgproxy-6b84bdb84-7g2l4 -c cloudsql-proxy -n db-auth-proxy-gcp

      2021/11/09 13:05:59 current FDs rlimit set to 1048576, wanted limit is 8500. Nothing to do here.
2021/11/09 13:05:59 using credential file for authentication; email=svc-keng-edap-poc-user@gcp-keng01.iam.gserviceaccount.com
2021/11/09 13:05:59 Listening on 0.0.0.0:3306 for gcp:us-east:databse
2021/11/09 13:05:59 Ready for new connections
2021/11/09 13:05:59 Generated RSA key in 412.982017ms

Несмотря на отсутствие ошибки, статус - «CrashLoopBackOff». Что я делаю не так?

Облачное соединение SQL для Kubernetes с использованием прокси

2 ответа

по объемам:

в связи: --dir=/cloudsql

вот мое полное развертывание yaml

Файл postgres-sql-credential.json:

в связи: `--dir=/cloudsql`