984 B
984 B
-
Issues with rabbit ?
- flap when rolling out agent / deploying new agent version
- even crash on big regions
- network flap / rabbit partition
- pause-minority
- reset cluster was ... the solution
- flap when rolling out agent / deploying new agent version
-
What's going on with rabbit ?
- reproduce workload with rabbit perftest
- oslo.metrics
- rabbitmq exporter / grafana dashboards
- smokeping between nodes
=> we identified issues were mostly related to neutron - rabbit flap flood resources to agents
-
How ? RPC implementation in Openstack: aka oslo.messaging
- pub/sub
- RPC server: setup endpoints / queues / listeners
- topic, fanout mechanism
- publish: rpc provided methods
- call
- cast
- cast / fanout=true
- RPC server: setup endpoints / queues / listeners
- notifications: kafka
- pub/sub
-
Journey to get stable
- Infra
- split rabbit-neutron / rabbit-*
- scale some clusters to 5 node
- Upgrade to 3.10+
-
openstack
- Infra