Newest 'avro' Questions - Stack Overflow

Questions tagged [avro]

Apache Avro is a data serialization framework primarily used in Apache Hadoop.

0
votes
0answers
6 views

How to directly write/read existing (value) class from avro?

I made a avro schema for existing classes (see /a/56685173/1206998) then managed to write but not read them using: static <T> T writeAndReadAvro(Schema schema, T t){...
0
votes
0answers
19 views

Apache Avro Maven Plugin generate UUID field instead of String

Given the following Avro Schema: { "namespace": "ro.dspr.coreentities", "type": "record", "name": "Organization", "fields": [ { "name": "id", "type": "string", ...
0
votes
1answer
19 views

Recursive schema with avro (SchemaBuilder)

Is it possible to make an avro schema which is recursive, like Schema schema = SchemaBuilder .record("RecursiveItem") .namespace("com.example") .fields() .name("subItem") .type("...
0
votes
0answers
12 views

Avro schema and C implementation

I'm trying to serialize data using the C implementation. The (simplified) schema is like this: [{ "type" : "record", "name" : "Global", "fields" : [ { "name" : "source", "type" : "string" ...
2
votes
0answers
24 views

AvroRuntimeException: Not a union: {“type”:“long”,“logicalType”:“timestamp-millis”}

I am trying to save the data from Spark dataframe to HDFS using Avro schema stored in the schema registry. However, I get an error while writing the data: Caused by: org.apache.avro....
1
vote
1answer
25 views

How to read binary avro fileData, with Source in akka?

I'm trying to read an avro file with Source from akka Streams. Source in akka streams read data like FileIO.FromPath(File), which will read and separate the lines based on (\n) character, where as ...
0
votes
0answers
26 views

BigQuery exports NUMERIC data type as binary data type in AVRO

I am exporting the data from BigQuery table which has column named prop12 defined as NUMERIC data type. Please note that destination format is AVRO and can't be changed. bq extract --...
0
votes
0answers
16 views

Overwriting a task in a subproject

When executing a gradle task for my subproject, declared in my root project, I want to overwrite the input folder for the task. I am working with microservices created with spring-boot. The ...
1
vote
0answers
22 views

IllegalAccessError: tried to access method org.apache.avro.specific.SpecificData.<init>()

Using Avro for serializing data to byte[] and deserializing data. https://cwiki.apache.org/confluence/display/AVRO/FAQ#FAQ-HowcanIserializedirectlyto/fromabytearray? shows sample usage. ...
0
votes
0answers
28 views

KafkaStreamsStateStore not working when the store value is an Avro SpecificRecord

I have a Spring Cloud Kafka Streams application that uses a StateStore in the Processor API, when using a transformer to perform a deduplication. The state store key-value are of the following types: ...
0
votes
0answers
44 views

Unable to deserialise kafkA STREAM to pojo

Getting exception while reading from kafka topic: Caused by: org.apache.kafka.common.errors.SerializationException: Error deserializing Avro message for id 1 Caused by: org.apache.kafka.common....
0
votes
0answers
18 views

Caused by: org.apache.avro.AvroRuntimeException: Unknown datum type org.joda.time.DateTime:

The following field is causing my Kafka Stream application to fail when it tries to produce message. { "name" : "TS", "type" : { "type" : "long", "logicalType" : "timestamp-millis"...
0
votes
0answers
24 views

Avro IDL for Rest Api input validation in python

We've been using Avro IDL to define message sets used on our Kafka back end and are quite happy with it. We've also been interested in tying to validate JSON to a REST api on a Python Flask app with ...
1
vote
1answer
18 views

How to save spark dataframe to parquet without using INT96 format for timestamp columns?

I have a spark dataframe that I want to save as parquet then load it using the parquet-avro library. There is a timestamp column in my dataframe that is converted to a INT96 timestamp column in ...
0
votes
1answer
32 views

Apache avro generates only public attributes

I am building source code of Kaa project and they used Apache avro for generating source code. I used avro library as instruction from here but I got only "@Deprecated" and "public" attribute, I ...
1
vote
2answers
39 views

Avro - java.io.IOException: Not a data file

I am using https://github.com/allegro/json-avro-converter to convert my json message into an avro file. After calling the convertToAvro method I get a byte array: byte[] byteArrayJson. Then I am using ...
0
votes
0answers
21 views

Deserialize avro to generic record without schema

Is it possible to deserialize a byte array/buffer to generic record without having any schema available, beside what's encoded in message? I'm writing a component that takes incoming encoded message ...
1
vote
1answer
25 views

Spark 2.4.1 can not read Avro file from HDFS

I have a simple code block to write then read dataframe as Avro format. As the Avro lib already built in Spark 2.4.x, The Avro files writing went succeed and files are generated in HDFS. However ...
2
votes
0answers
35 views

ClassCastException while using ThriftData for serializing Thrift data in Avro format

We are trying to find a way to serialize Thrift schema into Avro format. The flow is as follows: We have a project 'A' which uses Thrift. The output result object is a thrift based object. We ...
0
votes
1answer
30 views

proper guide for java kafka stream with avro schema registry

I'm seeking for proper tutorial/guide for java kafka-stream with schema-registry. I have google and could't find proper tutorial. I really appriciate if anybody can help me to find out atleast proper ...
1
vote
1answer
38 views

Could not initialize class io.confluent.kafka.schemaregistry.client.rest.RestService

I am trying to setup a kafka producer with KafkaAvroSerialzer for value. And I am facing this error wheneve rit is trying to created the Producer. I am using all the jars provided in confluent 5.2.1 ...
-1
votes
0answers
41 views

Unable to convert RDD[Java Class] to Dataframe in spark scala

I have avro message and .avsc file. I have generated the java class from .avsc file. Now I want to convert the avro(json) message into data frame. I read the message. Successfully decoded the message ...
0
votes
1answer
17 views

Is there another/similiar method for sparks.read.format.load outisde of databricks?

I am trying to load an avro file into a sparks dataframe so I can convert it to a pandas and eventually a dictionary. The method I want to use: df = spark.read.format("avro").load(avro_file_in_memory)...
0
votes
1answer
27 views

How to write a camel router with kafka avro serializer in kafka consumer

How to write kafka avro serializer and desializer (i.e using io.confluent.kafka.serializers.KafkaAvroDeserializer) from("kafka:localhost:9092?topic=customer_payment&groupId=group1&...
0
votes
1answer
30 views

Convert unix timestamp to avro and store it in BigQuery

Avro schema: { "name": "Entity", "type": "record", "namespace": "com.foobar.entity", "fields": [ { "name": "attribute", "type": "string" }, { "name": "value", ...
0
votes
1answer
15 views

If the avro schema is stored with the data, why does the java avro api need me to supply a schema file?

Microsoft Azure decides, in some cases, to dump data in avro format. The data in question is simply json records, from my perspective. So, I just want my json data back from the avro file. I am ...
0
votes
0answers
22 views

Class not found - Avro with Camel

I am using avro schema from: https://avro.apache.org/docs/current/gettingstartedjava.html#Defining+a+schema I have used the avro-tools-1.8.1.jar to create the User class from the schema. The schema ...
0
votes
0answers
22 views

Apache-Nifi not handling microsecond when using ConvertRecord processor to convert csv to avro

We are using ConvertRecord processor to convert CSV to AVRO format and facing issues with date having microseconds. Below is the observation and formats that we have used : yyyy-MM-dd-HH.mm.ss.SSS ...
0
votes
1answer
35 views

Kafka JDBC sink no handling null values

I am trying to insert data with the Kafka JDBC Sink connector, but it is returning me this exception. org.apache.kafka.connect.errors.DataException: Invalid null value for required INT64 field The ...
0
votes
1answer
40 views

Caused by: org.apache.avro.AvroRuntimeException: Malformed data. Length is negative: -53

Trying to use Flink to read a Kafka stream of "avro" serialized data, like this: tableEnv.connect(new Kafka() .version("0.11") .topic(source.getTopic()) ...
2
votes
0answers
36 views

How to serialize timestamp-millis logicalType to avro file using java

I am having a scenario where i want to create avro file having timestamp column which will look like 2016-11-16 06:43:19.77 I have used avro-1.8.2.jar for writing avro file. This works perfect for ...
0
votes
0answers
16 views

Generate classes with decimal datatype with avro-maven-plugin 1.9.0

I have an Avro schema with some fields defined as decimal logical type. From that schema I generate classes using avro-maven-plugin (using version 1.9.0). I would like to avoid generating ByteBuffer ...
1
vote
1answer
67 views

How to create a table from avro schema (.avsc)?

I have an avro schema file and I need to create a table in Databricks through pyspark. I don't need to load the data, just want to create the table. The easy way is to load the JSON string and take ...
0
votes
2answers
39 views

How to use kafka schema management and Avro for breaking changes

kafka schema management with avro give us flexibility to backward compatibility but how do we handle breaking-changes in the scheme? Assume Producer A publish messages M to Consumer C assume message ...
0
votes
1answer
41 views

python trouble de-serializing avro in memory

Currently, I am using requests to grab an avro file from a database and storing the data in requests.text. the file is separated by the schema and data. How do I merge the schema and data in memory ...
0
votes
1answer
48 views

Bigquery to Avro

I currently run a SQL Query to extract data from a Public BigQuery dataset into a Table, from there I can easily use the Export function to generate a Avro file and save it into GCS. How to generate ...
0
votes
1answer
43 views

avro 1.8.2 date type and writing as parquet

Am using avro 1.8.2 and a simple avro idl record like so record FooRecord { string fooString; int fooInt; union {null, date} fooDate = null; } fails to be written to parquet. ...
0
votes
0answers
13 views

Avro Decoders/Encoders for http4s

I am trying to get started with http4s, and i am looking for avro encoders which encode/decode entities to/from avro just like jsonEncoders: import org.http4s.circe._ import io.circe.generic.auto._ ...
0
votes
1answer
27 views

Using the KafkaAvroDeserializer with Alpakka

I have a SchemaRegistry and a KafkaBroker from which I pull data with Avro v1.8.1. For deserialization I've been using Confluent's KafkaAvroDeserializer. Now I've meant to refactor my code in order to ...
0
votes
0answers
65 views

avro schema with json encoding - how to determine schema back from serialized data

I want to use apache avro schema's for data serialization and deserialization. I want to use it with json encoding. I want to put several of this serialized objects using different schemas to the ...
0
votes
0answers
26 views

Avro ReflectData override record definition to logical type

Am using ReflectData to generate a schema from a Java class. One of the fields is private LocalDate localDate; and the reflectData generates equivalent code as a record { "name":"...
-2
votes
1answer
32 views

Partition column disappears in result set dataframe Spark

I tried to split Spark data frame by the timestamp column update_database_time and write it into HDFS with defined Avro schema. However, after calling the repartition method I get this exception: ...
0
votes
0answers
16 views

Is there a way to diff two Avro GenericRecords in Java to obtain only the fields and values that changed?

I would like to write Java code to diff two GenericRecords saved in memory, both with the same schema. I've been unable to find any libraries or any Apache Avro Javadoc on how to do this. My desire ...
1
vote
1answer
55 views

java.lang.Instantiation Exception while deserializing a byte stream into a Scala case class object

I am trying to deserialize an avro byte stream into a scala case class object. Basically, i had a kafka stream with avro encoded data flowing and now there is an addition to the schema and i am trying ...
0
votes
2answers
51 views

Time stamp vs avro schema

I have Vertica scheduler that consumes avro data from Kafka. one of the Vertica's columns is TIMESTAMP, currently,I tried defined the Avro schema like the following example: { "name":"startDate"...
0
votes
1answer
25 views

java.lang.NoClassDefFoundError: org/apache/avro/LogicalType while reading Parquet

I'm trying to read a parquet file, with this simple code: ParquetReader<GenericRecord> reader = AvroParquetReader.<GenericRecord>builder(path).build()); GenericRecord record = reader....
0
votes
0answers
28 views

How to convert a JSON message received by Azure EventHub into an Avro file on Azure Data Lake Storage Gen2?

I'm new to the Azure platform and currently learning about ways of doing things. My use case looks common but I couldn't find any information on the web that helped me. Workflow: A JSON event is ...
1
vote
1answer
30 views

AVRO, convert record to array

My AVRO schema has a fileObject record but I need to change this to be an array of fileObject. How can I do this? { "name": "file", "type": ["null", { ...
0
votes
2answers
56 views

Parsing failing on python for nested avro structure

I have an avro structure which is expecting array structure. I have created the avro structure but the parsing my data into that avro schema is failing Avro schema { "namespace": "com", "type": "...
0
votes
0answers
22 views

Drools KiePackage Fails to deserialize for large number of rules

Deserialization of large KiePackage fails with java.io.OptionalDataException for a large number of rules. I am trying to run Drools rules engine in combination with Apache Beam running on Spark. One ...