Vector storage is broken. Maybe it's better to say it's too disconnected from the actual needs of developers. I can say this with a good amount of experience building a project that used vector storage and search. But I can also say it from talking to hundreds of developers who have shared their problems with RAG (retrieval augmented generation) prompting and general vector storage and search solutions.

The problems

When a developer wants to use a vector storage solution, they generally have to take the following steps:

1. Upload source file
2. Vectorize source file
3. Store vector with metadata pointing to the source file
4. Query vector store with specific query text
5. Use metadata from result to fetch source file
6. Use the source file

As you can see, there is a massive disconnect between the source file and the vector storage and search. The file, arguably, is the most important part of any RAG solution. Yet, to get to that source file, you have to jump through multiple network requests.

The solution

Pinata has always solved problems from the file side of the equation. Vector storage and search is no different. Our vector storage solution reduces the steps outlined above to just:

1. Upload and vectorize source file in one request
2. Query vector storage and retrieve source file in one request

6 steps reduced to 2 feels pretty good. But what specifically did we build?

This simple code will not only upload your file, but it will add the file to a group dedicated to your vector index. All files related to the vector searches you're app will be conducting should use the same group ID.

Then, when you want to load the top match for a query using vector search, all you need to do is:

This code will conduct a vector search against the index associated with your group ID using the search term provided. If the returnFile flag is true this request will return the file for your app to use however it needs.

If you would prefer to receive a list of top matches, omitting the returnFile flag or setting it to false will give you a JSON response with the top matches and their content identifiers.

How can you get access?

This feature is in early private beta testing. We need testers who are willing to actively test this and provide feedback while working closely with our team to make this file-centric vector storage solution the best option on the market. Fill out the form below to be considered.