-
Notifications
You must be signed in to change notification settings - Fork 434
Avro support #2468
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Avro support #2468
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,87 @@ | ||
| /* | ||
| * Copyright 2013-present the original author or authors. | ||
| * | ||
| * Licensed under the Apache License, Version 2.0 (the "License"); | ||
| * you may not use this file except in compliance with the License. | ||
| * You may obtain a copy of the License at | ||
| * | ||
| * https://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, software | ||
| * distributed under the License is distributed on an "AS IS" BASIS, | ||
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| * See the License for the specific language governing permissions and | ||
| * limitations under the License. | ||
| */ | ||
|
|
||
| package org.springframework.cloud.contract.stubrunner | ||
|
|
||
| import org.apache.avro.generic.GenericRecord | ||
| import spock.lang.Specification | ||
|
|
||
| import org.springframework.cloud.contract.verifier.messaging.avro.KafkaAvroMessageVerifierSender | ||
| import org.springframework.kafka.core.KafkaTemplate | ||
|
|
||
| class StubRunnerExecutorAvroSpec extends Specification { | ||
|
|
||
| private KafkaTemplate<String, Object> kafkaTemplate = Mock() | ||
| private KafkaAvroMessageVerifierSender sender = new KafkaAvroMessageVerifierSender(kafkaTemplate) | ||
|
|
||
| def 'should send Avro-serialized GenericRecord to Kafka for Avro contracts (bug #2404)'() { | ||
| given: | ||
| def tmpContractDir = saveTmpContract(""" | ||
| label: book_returned | ||
| input: | ||
| triggeredBy: publishBookReturned() | ||
| outputMessage: | ||
| sentTo: book.returned | ||
| headers: | ||
| X-Correlation-Id: abc-123-def | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. What headers are being sent in case of avro messages? Is there any content type? |
||
| body: | ||
| isbn: "978-1234567890" | ||
| title: "Contract Testing for Dummies" | ||
| metadata: | ||
| kafka: | ||
| avro: | ||
| schema: > | ||
| { | ||
| "type": "record", | ||
| "name": "Book", | ||
| "fields": [ | ||
| {"name": "isbn", "type": "string"}, | ||
| {"name": "title", "type": "string"} | ||
| ] | ||
| } | ||
| """) | ||
| StubRunnerExecutor executor = new StubRunnerExecutor(new AvailablePortScanner(18000, 18999), sender, []) | ||
| executor.runStubs( | ||
| new StubRunnerOptionsBuilder().build(), | ||
| new StubRepository(tmpContractDir, [], new StubRunnerOptionsBuilder().build(), null), | ||
| new StubConfiguration('avro', 'avro', 'avro', '')) | ||
| when: | ||
| executor.trigger('book_returned') | ||
| then: | ||
| 1 * kafkaTemplate.send({ | ||
| it.topic() == "book.returned" && | ||
| it.value() instanceof GenericRecord && | ||
| it.value()["schema"] != null && | ||
| it.value()["isbn"] == "978-1234567890" && | ||
| it.value()["title"] == "Contract Testing for Dummies" && | ||
| header(it, "X-Correlation-Id") == "abc-123-def" | ||
| }) | ||
| cleanup: | ||
| executor.shutdown() | ||
| tmpContractDir.deleteDir() | ||
| } | ||
|
|
||
| private File saveTmpContract(String contractYaml) { | ||
| File contractDir = File.createTempDir() | ||
| new File(contractDir, "book_returned.yml").text = contractYaml | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Add a return |
||
| contractDir | ||
| } | ||
|
|
||
| private String header(it, String key) { | ||
| new String(it.headers().lastHeader(key).value()) | ||
| } | ||
|
|
||
| } | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| label: book_returned | ||
| input: | ||
| triggeredBy: publishBookReturned() | ||
| outputMessage: | ||
| sentTo: book.returned | ||
| headers: | ||
| X-Correlation-Id: abc-123-def | ||
| body: | ||
| isbn: "978-1234567890" | ||
| title: "Contract Testing for Dummies" | ||
| metadata: | ||
| kafka: | ||
| avro: | ||
| schema: > | ||
| { | ||
| "type": "record", | ||
| "name": "Book", | ||
| "fields": [ | ||
| {"name": "isbn", "type": "string"}, | ||
| {"name": "title", "type": "string"} | ||
| ] | ||
| } |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,53 @@ | ||
| /* | ||
| * Copyright 2013-present the original author or authors. | ||
| * | ||
| * Licensed under the Apache License, Version 2.0 (the "License"); | ||
| * you may not use this file except in compliance with the License. | ||
| * You may obtain a copy of the License at | ||
| * | ||
| * https://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, software | ||
| * distributed under the License is distributed on an "AS IS" BASIS, | ||
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| * See the License for the specific language governing permissions and | ||
| * limitations under the License. | ||
| */ | ||
|
|
||
| package org.springframework.cloud.contract.verifier.messaging.avro; | ||
|
|
||
| /** | ||
| * Avro serialization metadata for a Kafka contract message. | ||
| * | ||
| * <p> | ||
| * Example contract YAML: <pre> | ||
| * metadata: | ||
| * kafka: | ||
| * avro: | ||
| * schema: classpath:avro/Book.avsc | ||
| * </pre> | ||
| * | ||
| * <p> | ||
| * The Schema Registry URL is configured globally via | ||
| * {@code spring.cloud.contract.avro.schema-registry-url}. | ||
| * | ||
| * @author Emanuel Trandafir | ||
| * @since 4.2.0 | ||
| */ | ||
| public class AvroMetadata { | ||
|
|
||
| /** | ||
| * Classpath or filesystem path to the Avro schema file ({@code .avsc}), e.g. | ||
| * {@code classpath:avro/Book.avsc}. May also be an inline JSON schema string. | ||
| */ | ||
| private String schema; | ||
|
|
||
| public String getSchema() { | ||
| return this.schema; | ||
| } | ||
|
|
||
| public void setSchema(String schema) { | ||
| this.schema = schema; | ||
| } | ||
|
|
||
| } |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,85 @@ | ||
| /* | ||
| * Copyright 2013-present the original author or authors. | ||
| * | ||
| * Licensed under the Apache License, Version 2.0 (the "License"); | ||
| * you may not use this file except in compliance with the License. | ||
| * You may obtain a copy of the License at | ||
| * | ||
| * https://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, software | ||
| * distributed under the License is distributed on an "AS IS" BASIS, | ||
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| * See the License for the specific language governing permissions and | ||
| * limitations under the License. | ||
| */ | ||
|
|
||
| package org.springframework.cloud.contract.verifier.messaging.avro; | ||
|
|
||
| import java.util.HashMap; | ||
| import java.util.Map; | ||
|
|
||
| import com.fasterxml.jackson.annotation.JsonIgnoreProperties; | ||
| import io.confluent.kafka.serializers.KafkaAvroSerializer; | ||
| import org.apache.avro.specific.SpecificRecordBase; | ||
| import org.apache.kafka.clients.producer.ProducerConfig; | ||
| import org.apache.kafka.common.serialization.StringSerializer; | ||
| import tools.jackson.databind.json.JsonMapper; | ||
|
|
||
| import org.springframework.beans.factory.ObjectProvider; | ||
| import org.springframework.beans.factory.annotation.Qualifier; | ||
| import org.springframework.beans.factory.annotation.Value; | ||
| import org.springframework.boot.autoconfigure.condition.ConditionalOnClass; | ||
| import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean; | ||
| import org.springframework.cloud.contract.verifier.messaging.internal.ContractVerifierObjectMapper; | ||
| import org.springframework.context.annotation.Bean; | ||
| import org.springframework.context.annotation.Configuration; | ||
| import org.springframework.kafka.core.DefaultKafkaProducerFactory; | ||
| import org.springframework.kafka.core.KafkaTemplate; | ||
|
|
||
| /** | ||
| * Auto-configuration for Avro support in Spring Cloud Contract. Activates when | ||
| * {@code org.apache.avro.specific.SpecificRecordBase} is on the classpath. | ||
| * | ||
| * @author Emanuel Trandafir | ||
| * @since 4.2.0 | ||
| */ | ||
| @Configuration(proxyBeanMethods = false) | ||
| @ConditionalOnClass(name = "org.apache.avro.specific.SpecificRecordBase") | ||
| public class KafkaAvroContractVerifierConfiguration { | ||
|
|
||
| @Bean | ||
| @ConditionalOnMissingBean | ||
| ContractVerifierObjectMapper avroContractVerifierObjectMapper(ObjectProvider<JsonMapper> jsonMapper) { | ||
| JsonMapper mapper = jsonMapper.getIfAvailable(JsonMapper::new) | ||
| .rebuild() | ||
| .addMixIn(SpecificRecordBase.class, IgnoreAvroMixin.class) | ||
| .build(); | ||
| return new ContractVerifierObjectMapper(mapper); | ||
| } | ||
|
|
||
| @Bean | ||
| @ConditionalOnMissingBean(name = "avroKafkaTemplate") | ||
| KafkaTemplate<String, Object> avroKafkaTemplate(@Value("${spring.kafka.bootstrap-servers}") String bootstrapServers, | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Why can't we reuse the users production template? |
||
| @Value("${spring.cloud.contract.avro.schema-registry-url}") String schemaRegistryUrl) { | ||
| Map<String, Object> props = new HashMap<>(); | ||
| props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers); | ||
| props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class); | ||
| props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, KafkaAvroSerializer.class); | ||
| props.put("schema.registry.url", schemaRegistryUrl); | ||
| return new KafkaTemplate<>(new DefaultKafkaProducerFactory<>(props)); | ||
| } | ||
|
|
||
| @Bean | ||
| @ConditionalOnMissingBean | ||
| KafkaAvroMessageVerifierSender kafkaAvroMessageVerifierSender( | ||
| @Qualifier("avroKafkaTemplate") KafkaTemplate<String, Object> avroKafkaTemplate) { | ||
| return new KafkaAvroMessageVerifierSender(avroKafkaTemplate); | ||
| } | ||
|
|
||
| @JsonIgnoreProperties({ "schema", "specificData", "classSchema", "conversion" }) | ||
| interface IgnoreAvroMixin { | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Can you provide a javadoc why we need this? |
||
|
|
||
| } | ||
|
|
||
| } | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we make this section kafka / avro agnostic? If we just set asBytes() it should work right? Another option is to verify the contentType in which case if it's not json but explicitly sth else then we just pass through? Wdyt?
I also sense that we could have some extension points here. Like check through spi (we already do it in other parts of scc) some interfaces to verify if this contract has a special way of treating payload. We could have some AvroContractPayloadProceesor that would activate when the metadata has avro and it would provide a function to how convert the payload. That way this core logic stays clean of avro but we can inject the behavior. There would have to be some priority / ordering like we do in other spis here in scc.
Wdyt?