Category : softwareengineering

I am developing a project where I give a transcript of a video to an LLM (I’m simply calling OpenAI’s APIs). The LLM generates a wide range of data such as summaries, named entities (NER), etc. I want to store all the data in a database. I’m considering using a relational database and perhaps I’ll need to pre-process the LLM-generated response to store in the tables (OR should I store the data as a JSON format?). Another option is a NoSQL database (eg. MongoDB) where I can dump the response as a document in collections. Can someone please guide me? What is the best practice for the production enviro..

Read more

Myself Working as a machine learning engineer+MLOps, very proficient with python, and some experience with C++, Rust, also did some Scala programming (mostly for big data and spark). Situation Joining a new company (hedge fund) as a data engineer, although the data team mostly uses Python, the trading and market connectivity team uses a java ..

Read more