Relative Content

Tag Archive for apache-nifi

How to process the data as Chunks using Ni-Fi

Team, My Source dataset is Postgres where I fetch data using Ni-Fi and load the data into another database using Ni-Fi. I am having the huge data in source table and I need to process them as small chunks. I am facing issue while processing data as chunks and needs someone’s assistance. Please let me know how we can proceed.

Apache Nifi Reading from CSV and querying to postgresql

I’m struggling a lot with this simple workflow and I can’t make it work for me.
I have a csv file where I have two columns. ID and UUID. This is my document.
ID,UUID
0000317416,78f67ee6-293a-4373-8ed2-bdcbb21b3b2a
0004321903,1ce2327e-44b9-4611-b158-47af342ea8ed

DistributedMapCache doesn’t work properly on multiple nodes

I have Nifi set up on 6 nodes, I want to store one variable in DistributedMapCache, so I instert it with PutDistributedMapCache and then read it with FetchDistributedMapCache. However it works very unpredictably, most of the time fetch processor is unable to find that cache entry. I suppose it’s a problem with multiple nodes but according to documentation of DistributedMapCacheClientService “This can be used in order to share a Map between nodes in a NiFi cluster”. Another thing is that I start my flow with GenerateFlowFile that executes on primary node only and all my flowfiles are on node 1 so I’m not sure how is it supposed to work with DMC. I tested how it works when executed on all nodes and got weird results, 3/6 cache entries still could not be fetched.