I need to create output to 4 different files in my Apache Beam pipeline. I’ll use a very simplified case to protect the underlying work, but the overall structure here needs to be preserved. What I would like to happen is to take an unbounded collection from a Kafka topic and publish it to 4 different files. The unbounded collection will be windowed by having a minimum gap duration between bursts of data by 30 minutes. For each windowed set of data, I’d like one file to contain all entries greater than 100, we’ll call that file greater.dat, one file to contain all entries less than or equal to 100, we’ll call that file less.dat and then a summary file that for each greater.summ and less.sum that contain the number of entries in each. For the following stream of data, this is what i’d expect

I need to create output to 4 different files in my Apache Beam pipeline. I’ll use a very simplified case to protect the underlying work, but the overall structure here needs to be preserved. What I would like to happen is to take an unbounded collection from a Kafka topic and publish it to 4 different files. The unbounded collection will be windowed by having a minimum gap duration between bursts of data by 30 minutes. For each windowed set of data, I’d like one file to contain all entries greater than 100, we’ll call that file greater.dat, one file to contain all entries less than or equal to 100, we’ll call that file less.dat and then a summary file that for each greater.summ and less.sum that contain the number of entries in each. For the following stream of data, this is what i’d expect

I am trying to write a data pipeline in Java using Apache Beam version 2.56.0.

We received following error in Creating Tests for Apache Beam Pipeline with DirectTestRunner.
Cloning and editing the original object will be required to fix the issue. IllegalMutationException from Beam PTransform

We received following error in Creating Tests for Apache Beam Pipeline with DirectTestRunner.
Cloning and editing the original object will be required to fix the issue. IllegalMutationException from Beam PTransform

I have an apache beam dataflow in java which is running fine. but observed below error in error log for a particular day and time like on 2024-05-20 09:55:40.231000 UTC to 09:56:31.206000 UTC. all the message that it processed during the mentioned time were not processed and below error came.

Im trying to create an empty PCollection of a custom Object called Incident

After doing some transformations I get a PCollection<FinalResult> that looks like this:

I have an apache beam pipeline that reads a CSV file and also makes a query to a bigquery table, I need to count how many registers are in each PCollection and based on that create a FinalResult object that has both counts, have not been able to figure out how to combine both result into one object.

In Java Apache Beam , does the @ProcessElement method required to be void? Or can it return an int, string or class?