Kafka/Ruby on Rails - я не понимаю журналы, созданные после запуска `bundle exec karafka server`
Я новичок в мире микросервисов. Я переношу монолитное веб-приложение на микросервисное. У меня проблема с подключением веб-приложения Ruby on Rails, отвечающего за аутентификацию, с помощью KAFKA.
У самого Ruby on Rails нет никаких проблем. POC доступен здесь https://github.com/maelfosso/authentication-kafka. HTTP-запрос работает правильно.
Теперь я хочу связать его с Apache Kafka. Я использую Карафку. Благодаря его автору я наконец успешно его настроил. Но при запуске у меня получился такой лог, что я не понимаю. Вы можете прочитать это здесь https://gist.github.com/maelfosso/3be57de6bede85f67b36b0143fd6ae58
Это мой первый опыт работы с Apache Kafka, и я начинаю его как можно проще, как указано в разделе "Быстрый запуск".
С помощью микросервиса NodeJs я могу создать сообщение. Я могу увидеть созданное сообщение, используяbin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic business.
Вот некоторые части, которые повторяются много раз
Fetching cluster metadata from kafka://192.168.8.101:9092
[topic_metadata] Opening connection to 192.168.8.101:9092 with client id nkapsi_accounts_development...
[topic_metadata] Sending topic_metadata API request 1 to 192.168.8.101:9092
[topic_metadata] Waiting for response 1 from 192.168.8.101:9092
[topic_metadata] Received response 1 from 192.168.8.101:9092
Discovered cluster metadata; nodes: 192.168.8.101:9092 (node_id=0)
Closing socket to 192.168.8.101:9092
Я не понимаю, почему он закрывается.
[[nkapsi_accounts_development_business] {business: 0}:] Fetching cluster metadata from kafka://192.168.8.101:9092
[[nkapsi_accounts_development_business] {business: 0}:] [topic_metadata] Opening connection to 192.168.8.101:9092 with client id nkapsi_accounts_development...
[[nkapsi_accounts_development_business] {business: 0}:] [topic_metadata] Sending topic_metadata API request 1 to 192.168.8.101:9092
[[nkapsi_accounts_development_business] {business: 0}:] [topic_metadata] Waiting for response 1 from 192.168.8.101:9092
[[nkapsi_accounts_development_business] {business: 0}:] [topic_metadata] Received response 1 from 192.168.8.101:9092
[[nkapsi_accounts_development_business] {business: 0}:] Discovered cluster metadata; nodes: 192.168.8.101:9092 (node_id=0)
[[nkapsi_accounts_development_business] {business: 0}:] Closing socket to 192.168.8.101:9092
[[nkapsi_accounts_development_business] {business: 0}:] [fetch] Opening connection to 192.168.8.101:9092 with client id nkapsi_accounts_development...
[[nkapsi_accounts_development_business] {business: 0}:] [fetch] Sending fetch API request 1 to 192.168.8.101:9092
[[nkapsi_accounts_development_business] {business: 0}:] [fetch] Waiting for response 1 from 192.168.8.101:9092
[[nkapsi_accounts_development_business] {business: 0}:] [fetch] Received response 1 from 192.168.8.101:9092
[[nkapsi_accounts_development_business] {business: 0}:] Fetching batches
[[nkapsi_accounts_development_business] {business: 0}:] [fetch] Sending fetch API request 2 to 192.168.8.101:9092
[[nkapsi_accounts_development_business] {business: 0}:] [fetch] Waiting for response 2 from 192.168.8.101:9092
Жду ответа 2 (до 9, если я его не перестану)? Чего он ждет? Есть конфигурация, которую я пропустил?
New [Business] event:
Params deserialization error for business topic: 767: unexpected token at '{
activities: [],
_id: 5f0dd8df919fec6c2294fd28,
delivrables: [],
__v: 0
}'
[[nkapsi_accounts_development_business] {}:] Exception raised when processing business/0 in offset range 0..0 -- Karafka::Errors::DeserializationError: 767: unexpected token at '{
activities: [],
_id: 5f0dd8df919fec6c2294fd28,
delivrables: [],
__v: 0
}'
/home/maelfosso/.rvm/gems/ruby-2.6.5/gems/karafka-1.3.6/lib/karafka/serialization/json/deserializer.rb:22:in `rescue in call'
/home/maelfosso/.rvm/gems/ruby-2.6.5/gems/karafka-1.3.6/lib/karafka/serialization/json/deserializer.rb:19:in `call'
/home/maelfosso/.rvm/gems/ruby-2.6.5/gems/karafka-1.3.6/lib/karafka/params/params.rb:60:in `block in deserialize'
/home/maelfosso/.rvm/gems/ruby-2.6.5/gems/dry-monitor-0.3.2/lib/dry/monitor/notifications.rb:43:in `block in instrument'
/home/maelfosso/.rvm/gems/ruby-2.6.5/gems/dry-monitor-0.3.2/lib/dry/monitor/notifications.rb:11:in `measure'
/home/maelfosso/.rvm/gems/ruby-2.6.5/gems/dry-monitor-0.3.2/lib/dry/monitor/notifications.rb:43:in `instrument'
/home/maelfosso/.rvm/gems/ruby-2.6.5/gems/karafka-1.3.6/lib/karafka/params/params.rb:59:in `deserialize'
/home/maelfosso/.rvm/gems/ruby-2.6.5/gems/karafka-1.3.6/lib/karafka/params/params.rb:51:in `deserialize!'
/home/maelfosso/.rvm/gems/ruby-2.6.5/gems/karafka-1.3.6/lib/karafka/params/params_batch.rb:42:in `first'
/home/maelfosso/.rvm/gems/ruby-2.6.5/gems/karafka-1.3.6/lib/karafka/consumers/single_params.rb:11:in `params'
/home/maelfosso/Documents/Projects/Nkapsi/api/accounts/app/consumers/business_consumer.rb:5:in `consume'
/home/maelfosso/.rvm/gems/ruby-2.6.5/gems/karafka-1.3.6/lib/karafka/backends/inline.rb:12:in `block in process'
/home/maelfosso/.rvm/gems/ruby-2.6.5/gems/dry-monitor-0.3.2/lib/dry/monitor/notifications.rb:43:in `block in instrument'
/home/maelfosso/.rvm/gems/ruby-2.6.5/gems/dry-monitor-0.3.2/lib/dry/monitor/notifications.rb:11:in `measure'
/home/maelfosso/.rvm/gems/ruby-2.6.5/gems/dry-monitor-0.3.2/lib/dry/monitor/notifications.rb:43:in `instrument'
/home/maelfosso/.rvm/gems/ruby-2.6.5/gems/karafka-1.3.6/lib/karafka/backends/inline.rb:12:in `process'
/home/maelfosso/.rvm/gems/ruby-2.6.5/gems/karafka-1.3.6/lib/karafka/base_consumer.rb:38:in `call'
/home/maelfosso/.rvm/gems/ruby-2.6.5/gems/karafka-1.3.6/lib/karafka/connection/batch_delegator.rb:43:in `block (2 levels) in call'
/home/maelfosso/.rvm/gems/ruby-2.6.5/gems/karafka-1.3.6/lib/karafka/connection/batch_delegator.rb:38:in `each'
/home/maelfosso/.rvm/gems/ruby-2.6.5/gems/karafka-1.3.6/lib/karafka/connection/batch_delegator.rb:38:in `block in call'
/home/maelfosso/.rvm/gems/ruby-2.6.5/gems/dry-monitor-0.3.2/lib/dry/monitor/notifications.rb:43:in `block in instrument'
/home/maelfosso/.rvm/gems/ruby-2.6.5/gems/dry-monitor-0.3.2/lib/dry/monitor/notifications.rb:11:in `measure'
/home/maelfosso/.rvm/gems/ruby-2.6.5/gems/dry-monitor-0.3.2/lib/dry/monitor/notifications.rb:43:in `instrument'
/home/maelfosso/.rvm/gems/ruby-2.6.5/gems/karafka-1.3.6/lib/karafka/connection/batch_delegator.rb:18:in `call'
/home/maelfosso/.rvm/gems/ruby-2.6.5/gems/karafka-1.3.6/lib/karafka/connection/listener.rb:46:in `block in fetch_loop'
/home/maelfosso/.rvm/gems/ruby-2.6.5/gems/karafka-1.3.6/lib/karafka/connection/client.rb:36:in `block in fetch_loop'
/home/maelfosso/.rvm/gems/ruby-2.6.5/gems/ruby-kafka-1.1.0/lib/kafka/consumer.rb:331:in `block (3 levels) in each_batch'
/home/maelfosso/.rvm/gems/ruby-2.6.5/gems/activesupport-6.0.3.2/lib/active_support/notifications.rb:182:in `instrument'
/home/maelfosso/.rvm/gems/ruby-2.6.5/gems/ruby-kafka-1.1.0/lib/kafka/instrumenter.rb:21:in `instrument'
/home/maelfosso/.rvm/gems/ruby-2.6.5/gems/ruby-kafka-1.1.0/lib/kafka/instrumenter.rb:35:in `instrument'
/home/maelfosso/.rvm/gems/ruby-2.6.5/gems/ruby-kafka-1.1.0/lib/kafka/consumer.rb:329:in `block (2 levels) in each_batch'
/home/maelfosso/.rvm/gems/ruby-2.6.5/gems/ruby-kafka-1.1.0/lib/kafka/consumer.rb:310:in `each'
/home/maelfosso/.rvm/gems/ruby-2.6.5/gems/ruby-kafka-1.1.0/lib/kafka/consumer.rb:310:in `block in each_batch'
/home/maelfosso/.rvm/gems/ruby-2.6.5/gems/karafka-1.3.6/lib/karafka/patches/ruby_kafka.rb:28:in `block in consumer_loop'
/home/maelfosso/.rvm/gems/ruby-2.6.5/gems/ruby-kafka-1.1.0/lib/kafka/consumer.rb:414:in `block in consumer_loop'
/home/maelfosso/.rvm/gems/ruby-2.6.5/gems/activesupport-6.0.3.2/lib/active_support/notifications.rb:182:in `instrument'
/home/maelfosso/.rvm/gems/ruby-2.6.5/gems/ruby-kafka-1.1.0/lib/kafka/instrumenter.rb:21:in `instrument'
/home/maelfosso/.rvm/gems/ruby-2.6.5/gems/ruby-kafka-1.1.0/lib/kafka/instrumenter.rb:35:in `instrument'
/home/maelfosso/.rvm/gems/ruby-2.6.5/gems/ruby-kafka-1.1.0/lib/kafka/consumer.rb:412:in `consumer_loop'
/home/maelfosso/.rvm/gems/ruby-2.6.5/gems/karafka-1.3.6/lib/karafka/patches/ruby_kafka.rb:16:in `consumer_loop'
/home/maelfosso/.rvm/gems/ruby-2.6.5/gems/ruby-kafka-1.1.0/lib/kafka/consumer.rb:307:in `each_batch'
/home/maelfosso/.rvm/gems/ruby-2.6.5/gems/karafka-1.3.6/lib/karafka/connection/client.rb:36:in `fetch_loop'
/home/maelfosso/.rvm/gems/ruby-2.6.5/gems/karafka-1.3.6/lib/karafka/connection/listener.rb:39:in `fetch_loop'
/home/maelfosso/.rvm/gems/ruby-2.6.5/gems/karafka-1.3.6/lib/karafka/connection/listener.rb:24:in `call'
/home/maelfosso/.rvm/gems/ruby-2.6.5/gems/karafka-1.3.6/lib/karafka/fetcher.rb:18:in `block (2 levels) in call'
[[nkapsi_accounts_development_business] {business: 0}:] [fetch] Received response 2 from 192.168.8.101:9092
[[nkapsi_accounts_development_business] {business: 0}:] Handling fetcher command: stop
[nkapsi_accounts_development_business] {business: 0}: Fetcher thread exited.
[[nkapsi_accounts_development_business] {}:] Leaving group `nkapsi_accounts_development_business`
[[nkapsi_accounts_development_business] {}:] [leave_group] Sending leave_group API request 6 to 192.168.8.101:9092
[[nkapsi_accounts_development_business] {}:] [leave_group] Waiting for response 6 from 192.168.8.101:9092
[[nkapsi_accounts_development_business] {}:] [leave_group] Received response 6 from 192.168.8.101:9092
[[nkapsi_accounts_development_business] {}:] Disconnecting broker 0
[[nkapsi_accounts_development_business] {}:] Closing socket to 192.168.8.101:9092
Client fetch loop error: 767: unexpected token at '{
activities: [],
_id: 5f0dd8df919fec6c2294fd28,
delivrables: [],
__v: 0
}'
Я не понимаю в чем проблема. Это просто сообщение json, отправленное из приложения NodeJS. Вот это точно
{
"email": "contact@nkapsi.cm",
"name": "nkapsi",
"website: "nkapsi.cm",
"phone": "666555444"
}
Также есть это сообщение, которое отображается много раз
[[nkapsi_accounts_development_business] {business: 0}:] Joining group `nkapsi_accounts_development_business`
[[nkapsi_accounts_development_business] {business: 0}:] Handling fetcher command: configure
[[nkapsi_accounts_development_business] {business: 0}:] [join_group] Opening connection to 192.168.8.101:9092 with client id nkapsi_accounts_development...
[[nkapsi_accounts_development_business] {business: 0}:] Fetching batches
[[nkapsi_accounts_development_business] {business: 0}:] There are no partitions to fetch from, sleeping for 1s
[[nkapsi_accounts_development_business] {business: 0}:] [join_group] Sending join_group API request 1 to 192.168.8.101:9092
[[nkapsi_accounts_development_business] {business: 0}:] [join_group] Waiting for response 1 from 192.168.8.101:9092
[[nkapsi_accounts_development_business] {business: 0}:] [join_group] Received response 1 from 192.168.8.101:9092
[[nkapsi_accounts_development_business] {business: 0}:] Failed to join group; resetting member id and retrying in 1s...
[[nkapsi_accounts_development_business] {business: 0}:] Fetching batches
[[nkapsi_accounts_development_business] {business: 0}:] There are no partitions to fetch from, sleeping for 1s
[[nkapsi_accounts_development_business] {business: 0}:] Joining group `nkapsi_accounts_development_business`
[[nkapsi_accounts_development_business] {business: 0}:] [join_group] Sending join_group API request 2 to 192.168.8.101:9092
[[nkapsi_accounts_development_business] {business: 0}:] [join_group] Waiting for response 2 from 192.168.8.101:9092
[[nkapsi_accounts_development_business] {business: 0}:] [join_group] Received response 2 from 192.168.8.101:9092
[[nkapsi_accounts_development_business] {business: 0}:] Joined group `nkapsi_accounts_development_business` with member id `nkapsi_accounts_development-81a33adb-ff6d-4066-8a7f-b54fd83aeee7`
[[nkapsi_accounts_development_business] {business: 0}:] Chosen as leader of group `nkapsi_accounts_development_business`
[[nkapsi_accounts_development_business] {business: 0}:] [sync_group] Sending sync_group API request 3 to 192.168.8.101:9092
[[nkapsi_accounts_development_business] {business: 0}:] [sync_group] Waiting for response 3 from 192.168.8.101:9092
[[nkapsi_accounts_development_business] {business: 0}:] [sync_group] Received response 3 from 192.168.8.101:9092
[[nkapsi_accounts_development_business] {business: 0}:] Partitions assigned for `business`: 0
Я этого не понимаю.
Проверяю установленный Kafka на 192.168.8.101:9092и это работает. Я могу создать сообщение из приложения NodeJS или от производителя консоли и использовать их с помощью console-consumer.
Пожалуйста, в чем может быть проблема? Полный файл журнала доступен здесь https://gist.github.com/maelfosso/3be57de6bede85f67b36b0143fd6ae58, а POC Ruby on Rails доступен здесь https://github.com/maelfosso/authentication-kafka
Благодарность